Meta(
    title: "Ion shell for scripting",
    summary: Some("Testing Redox OS\'s Ion shell as a scripting language"),
    published: None,
    tags: [
        "rust",
        "linux",
        "redox",
        "shell",
        "programming",
    ],
)
---
I'm a bit of a fan of hobby operating systems, and have found over time that some useful bits often come out of them. For instance, Sortix is a Unix-like hobby OS that was under very heavy development a few years back, and I've been using their own cleaned up Zlib as a drop in replacement for what every other OS uses. The code is smaller, cleaner, and easier to understand, while somehow maintaining full compatability.

Redox OS is a hobbyist OS written in Rust with a microkernel architecture and a full graphical environment. There are a few nice pieces that show promise. OrbTK is a gui toolkit which managed to go further than a number of other similar projects, and the Ion shell also looks really nice, both as an interactive shell as an alternative to traditional POSIX sh. It's also been benchmarked and shown to be blazingly fast even in comparison with Dash.

Now, before getting into the full article let's be clear that Ion is definitely not POSIX compatible. It aims for a more consistent language design with some nice additional features. If you were to build a minimal system with Ion it would make sense to also include a small POSIX compatible shell just to run POSIX shell scripts. Indeed this is exactly what Redox does, as they have ported Dash to run on Redox and it's included in the default installation.

I've played around with Ion for quite a while now as an interactive shell, but you can only get so much of a feel for a shell's power by using it to launch Vim and run day to day commands. That said, my initial impressions were that Ion has taken some of the best features from Zsh and Fish, such as Zsh's autocd and Fish's automatic integration of history into the normal completion system. A quick glance over the manual [1] shows that the language has been pretty well thought out, too.

=> https://doc.redox-os.org/ion-manual/ [1] The Ion manual

### Some highlighted features
* Ion can optionally use typed function parameters. With or without typing, function parameters must be declared in the function signature (Bourne shell just accepts positional parameters)
* Both HashMaps and BTreeMaps (dictionaries, if you're coming from Python)
* Much better array support than Bourne type shells
* Spaces between your variables/values and the '=' sign (I always forget this when going back to Bourne shell after using a "real" programming language)
* Proper scopes for variables. A variable declared inside a block is not visible outside that block.
* Namespaces
* More consistent control flow syntax. All blocks are terminated with 'end'.
* Ranges and slicing syntax. Ranges can go in reverse (nice) and can go in steps (again, nice).
* Match statements including match guards.
* String and array methods

Looking at control flow, Bourne type shells inherited from an Algol variant the concept of reversing the keyword to terminate a control flow concept. Thus you get if..fi and case..esac. But there was a problem with that idea. The 'od' or Octal Dump utility existed from a very early time in Unix development, which would have lead to a naming collision had 'do' been reversed. So we ended up with 'for x in y ; do ; done' and 'while x ; do ; done'. Ion brings some nice consistency to language constructs here. All functions, conditionals, matches and loops are terminated with 'end'. In addition, the '[]' test construct is replaced with the 'test' and 'exists' keywords. The result is a very clean and consistent syntax. It's quite a bit like Fish to be honest, but has enough new ideas to be a cool new thing of itself and not just a copycat.

I'd be remiss if I didn't go into a bit more detail on the last point in the above list - string and array methods. The concept is pretty straightforward. Since a shell by it's nature spends so much of it's time parsing and evaluating textual input, the user shouldn't have to immediately reach for `awk`, `sed`, `cut`, `tr` or the like when manipulating text. To that note, Ion has the notion of string and array methods, which have a somewhat different syntax than regular function calls.
```
$method(args) # string methods
@method(args) # array methods
```
That should be easy enough to figure out and familiar to anyone with any programming experience, but do note the `$` and `@` sigils. So what methods are available? Let's start with the string methods.
* basename
* extension
* filename
* join
* find
* len
* len_bytes
* parent
* repeat
* replace
* replacen
* regex_replace
* reverse
* to_lowercase
* to_uppercase
* escape
* unescape
* or
Most of those are pretty self explanatory. Note that for `len` you get the normal version which counts the number of graphemes and a version which counts bytes, and there are some variations on the `replace` method. The `or` method allows gicing a fallback value if a variable has not been set, which is important because Ion will error for unset variables unlike most shells which just return an empty string. How about array methods then?
* lines
* split
* split_at
* bytes
* chars
* graphemes
* reverse
* subst
Some things to note here. The `bytes` method takes a string and splits it into an array of bytes. If you print the resulting array you will get the actual 8-bit integers representing those characters. The `graphemes` and `chars` methods split into an array of chars and graphemes. You can split on a specific pattern or on an index.

Taken all together, that covers the vast majority of any text manipulation you might need to do all without resorting to any external commands. For short scripts there may not be any tangible benefit, but for complex scripts with a lot of complex control flow the ability to slice and dice text without forking off new processes should give Ion a pretty good performance edge, assuming the underlying code is of good quality.

Not that there aren't a few things which might trip you up. Declaring a list of arguments as a single string variable 'args' and then passing them to some command, as in 'cmd $args' will give those arguments to the command as if it were a single, contiguous quoted string. You can instead either declare an array variable, or else use the 'split' method to split the string into an array on whitespace characters. Also, since Ion uses the '@' sign as a sigil denoting an array variable, url's which contain a username (john@example.com) must be quoted. This will trip you up for sure the first time you create a git repo and give it a push url.

I also missed being able to extrapolate variables using `eval`. Consider the following sh snippet.
```
foo_version=42
bar_version=69

myfunc() {
    local package=$1
    local version=$(eval echo \$${package}_version)
    echo version
}
myfunc foo # prints 42
myfunc bar # prints 69
```
At any rate I didn't find an equivalent way to expand variables in Ion. But, you could replicate the functionality using a match statement or a hashmap, so probably not a loss. In fact, the hashmap is probably the cleanest way to handle this of the three methods.
```
let versions:hmap[int] = [ foo=42 bar=69 ]
echo @versions[foo]
echo @versions[bar]
```
Yeah, looking at that, it's definitely a much cleaner way to accomplish the same task.

### Completions - a mixed bag
This only matters for interactive use. History is automatically used as a source for completions (as mentioned above) as well as executables in your $PATH and file and directory paths. What's missing is a mechanism for creating your own completions for certain commands. This is a big omission so it's worth mentioning. There was talk on the bug tracker of a plan but not much real follow through. Maybe a feature will materialize, and maybe not. The repository saw 9 commits in April, 20 in March, so it's definitely a living project.

## Ion's killer feature
Ion has one feature that is more interesting to me than everything I've mentioned up to this point. The entire shell itself can be used as a library. In other words, you can embed Ion into another program written in Rust and have an embedded scripting language, much the same way that is often done with Lua. I can potentially see a lot of different uses for this. I'm imagining a built in scripting language for a Gemini server, which can then run your custom scripts without having to fork a shell from the parent process. It's definitely something that would be cool to play around with.

## A small real world script
Since Gcc, Binutils and Musl all saw releases recently, I thought it might be a good idea to update the cross toolchains that I keep on my two laptops. I also figured this would be a good task to automate, which is the motivation behind this post actually. The finished script is 246 lines including comments and some logging to the console. I tried to make good use of the features Ion provides that a Bourne compatible shell doesn't give you, at least where it makes sense.
```
#!/usr/local/bin/ion
#set -e

# ======================================================= #
# Change the following to suit
let jobs = 4
let arch = aarch64
let install_prefix = ${HOME}/cross
# If you're installation prefix is somewhere that your user
# has write permissions, leave this empty.
let SUDO = ""
# Don't change below this line
# ======================================================= #
```
The top of the script just sets up some user defined variables. I ran into some issues with debugging when using `set -e` so it's commented out in favor of manually calling `exit` when certain commands fail. This highlights one of the other issues with the shell - error messages are not very useful at this stage.
```
# Path definitions
let basedir = $PWD
let tgt = ${arch}-unknown-linux-musl
let sysroot = ${install_prefix}/${tgt}
let prefix = ${install_prefix}/${tgt}/toolchain
# This gets re-used several times
let gnu = "https://ftp.gnu.org/gnu"

# Package versions
let versions:hmap[str] = [ binutils=2.40 gcc=13.1.0 gmp=6.2.1 mpc=1.3.1 
    mpfr=4.2.0 linux=6.3.1 musl=1.2.4 ]

# We build gcc twice, and after setting this to '2' we get a different set of
# options
let pass = 1
```
This next bit just sets up some more global values, including a hashmap just like the example above. Next we'll look at a few function definitions.
```
fn get_opts package:str
    match $package
        case binutils
            echo --target=${tgt} --prefix=${prefix} --with-sysroot=${sysroot} \
                --disable-nls --enable-gprofng=no
        case gcc if eq $pass 1
            echo --target=${tgt} --prefix=${prefix} --with-sysroot=${sysroot} \
                --with-newlib --without-headers --disable-nls --disable-shared \
                --disable-multilib --disable-decimal-float --disable-threads \
                --disable-libatomic --disable-libgomp --disable-libquadmath \
                --disable-libssp --disable-libvtv --disable-libstdcxx \
                --enable-languages=c,c++
        case musl; echo --target=${tgt} --prefix=/usr --libdir=/lib
        case gcc if eq $pass 2
            echo --target=${tgt} --prefix=${prefix} --with-sysroot=${sysroot} \
                enable-languages=c,c++ --disable-bootstrap --disable-multilib \
                --disable-libssp --disable-libsanitizer
    end
end
```
We're going to have a single function to build packages since the commands that will be run are going to be so similar (configure, make, make install). So I set up this function which just takes the package name as a parameter and returns the options to be passed to ./configure. Note that for gcc we use match guards to return a different set of options depending on which pass we're on.
```# Two of our packages come as gzipped tarballs, the rest are xz compressed
fn get_ext package:str
    match $package
        case [ binutils gcc gmp mpfr linux ]; echo tar.xz
        case [ mpc musl ]; echo tar.gz
    end
end

fn get_build_dir package:str
    echo ${basedir}/${package}-@versions[$package]
end

fn get_dist package:str
    let ext = $(get_ext $package)
    echo ${package}-@versions[$package].${ext}
end

fn get_url package:str
    let dist = $(get_dist $package)
    match $package
        case [ binutils gmp mpc mpfr ]; echo ${gnu}/${package}/${dist}
        case gcc; echo ${gnu}/gcc/gcc-@versions[gcc]/${dist}
        case musl; echo http://musl.libc.org/releases/${dist}
        case linux; echo https://cdn.kernel.org/pub/linux/kernel/v6.x/${dist}
    end
end
```
These four small helper functions help define where our build directory will be for each package as well as where to get the sources from. Notice that when using a match statement, you can provide an array as one of the cases and if any array member matches that branch will run.
```
fn extract package:str
    let srcdir = ${package}-@versions[$package]
    if not exists -f ${srcdir}/.dirstamp
        let dist = $(get_dist $package)
        if not exists -f $dist
            download_source $package
        end
        printf "${c::green}Extracting %s\n${c::reset}" $package
        tar -xf $dist || exit 1
    end
    if test $package = gcc
        for p in [ gmp mpc mpfr ]
            extract $p
            let version = @versions[$p]
            ln -sv ../${p}-${version} gcc-@versions[gcc]/${p} || exit 1
        end
    end
    touch ${srcdir}/.dirstamp || exit 1
end
```
Skipping ahead a bit, in this function you can see the syntax which replaces the Bourne `[ conditional ]` testing for conditionals syntax. You get two different keywords here, `exists` and `test`, with modifiers such as `not`. It should be pretty self explanatory. You can also see my `command || exit 1` hack for fallible commands, which is actually the same syntax as Bourne shell. Also note the last line in the function, where I'm creating a file called `.dirstamp`. This file only gets created if the sources are successfully downloaded and extracted, so you can't wind up with an incomplete source directory. That's an old trick from the handwritten Makefile days.

One last item of note here. The `printf "${c::green}Extracting..${c::reset}"` line in the above function prints that line in green. You can specify colors by name or by number, as well as other text modifiers like bold or underlined text. That's a lot easier to remember than all of the normal ansi escape sequences, too.

I'm going to skip a lot of the other functions, as it would be largely the same sort of code as what I've already shown.

## Accepting arguments
In a Bourne compatible shell, positional arguments are generally given as $1 for the first argument, $2 for the second and so on, with $0 being the name of the script. In Ion, those variables mean nothing. Instead, the global `@args` is an array containing the command line arguments, with the script name taking the `0` position. This is a departure, but I don't dislike it. It's similar to Rust's `std::env::args()`. This does, however, preclude the use of something like `getopt` for parsing command line arguments. However I tend to think that any program requiring more than extremely basic command line parsing would be better implemented in a compiled programming language. Shell scripting is for basic automation tasks. I'm not a fan of complicated shell scripts such as `libtool` or GNU's `autotools` suite. By the time a shell script becomes that large you have lost all of the benefit of writing it as a shell script, namely, it is no longer possible for an outsider to look at your speghetti code and easily tell what the script is doing.

Anyway, I settled on a simple stratagy for defining the entry point for this script and accepting some options. I'm not using flags such as `-x` or long opts as in `--option`, but just simple subcommands.
```
if test $len(@args) -eq 2
    match @args[1]
        case clean
            clean
            exit
        case distclean
            clean
            distclean
            exit
        case uninstall
            uninstall
        case build
            build
            exit
        case [ usage help ]
            usage
            exit
        case _
            usage
            exit 1
    end
else
    usage
    exit 1
end
```
The `match` statement just calls the appropriate functions and exits after. It would definitely be possible to write a command line parser in the Ion language itself, but for the reasons I already outlined I'm not going to be doing so.

## The verdict
