Add two gemlog posts

Makefile

@@ -6,7 +6,7 @@ build:
 	zond build
 
 upload: build
-	scp -r public/* hitchhiker-linux.org:/srv/gemini
+	scp -r public/* nathan@hitchhiker-linux.org:/srv/gemini
 
 publish:
 	echo "$(submit_url)" | eval openssl s_client -connect warmedal.se:1965 -crlf \

content/gemlog/a_spartan_to_gemini_proxy_server.gmi

@@ -0,0 +1,52 @@
+Meta(
+    title: "A Spartan to Gemini proxy server",
+    summary: Some("I\'m writing a Spartan to Gemini proxy server, but it has me thinking"),
+    published: Some(Time(
+        year: 2022,
+        month: 11,
+        day: 30,
+        hour: 1,
+        minute: 16,
+        second: 47,
+    )),
+    tags: [
+        "spartan",
+        "gemini",
+        "software",
+        "rust",
+    ],
+)
+---
+For a few days now I've been hacking in my spare time on another idea of mine. My own capsule has been happily running on Agate since it's inception and I've never had an issue with it. When I decided to bihost it on Spartan I wrote the Agis server and have been quite happy with it.
+=> https://github.com/mbrubeck/agate Agate Gemini server
+=> https://codeberg.org/jeang3nie/agis Agis Spartan server
+
+I'm always looking for ways to run things more efficiently, and for fun new projects. Now, Agate is a great high performance server, but Agis uses significantly less memory at idle than Agate. The biggest difference I believe is that Agate is using the Tokio async runtime. Tokio is great. It's really high performance, and you can build monstrously high capacity web services with it. But that's just it. It's overkill. Instead of using async code everywhere and pulling in everything that Tokio requires, when I built Agis I did so with a simple threadpool model and four worker threads handling requests. This is configurable of course, but let's be honest and assume that for most of us four simultaneous requests is wishful thinking on Gemini. But let's stick with four worker threads vs the whole async runtime and check out the momory usage.
+```
+% pgrep agate
+645
+% pgrep agis 
+14423
+% sudo pmap 645 | tail -n 1; sudo pmap 14423 | tail -n 1
+total kB         1156800    6644    9756    4332       0
+total kB          410912    2556    1358     364       0
+```
+So yeah, we're looking at less than half the memory usage at idle. It definitely won't scale well compared with the async version but my opinion is that it really doesn't have to. There is plenty of performance with four worker threads and there's the added bonus of simpler code.
+
+## The project
+My first pass project is to create a simple reverse proxy server that just takes in gemini requests and forwards them to a Spartan server, then forwards the response back through Gemini. It's not fully cooked yet, but it's close. The library portion of the code knows how to handle each connection. It just needs the actual binary which will only have to get the config, start up the worker threads and begin listening for connections to forward to them.
+
+Writing it has so far been pretty easy, especially considering how much code I already have available for reuse. I had already begun stripping the client code for various smolweb protocols out of GemView and putting them into a modular library crate called Bucky. Bucky is the first time I've used cargo workspaces, and this tweak makes it so that each protocol lives in it's own crate, with one crate of common code that they share. So the proxy server is able to just use the bucky-spartan crate to talk Spartan. So basically, I had to write a Gemini server, without the server having to care about how to get the data being requested. That's the job of the Spartan server. So the proxy server itself can be pretty dumb.
+
+=> https://codeberg.org/jeang3nie/bucky Bucky smolweb client library
+
+I think I'm going to finish it at least to the point where it's able to do what it was written for, but it's giving me other ideas in the meantime.
+
+## Other idea #1
+So I was thinking to myself, that while this is likely going to significantly reduce the memory usage of the overall stack, I could reclaim even more memory if I just had a single server which can speak both protocols to begin with. Agis would be a great start if I went this route. It already has virtual hosting, CGI and ScriptAlias support. For my use case I'm running the same static content on both Spartan and Gemini, which is even easier to handle than any type of dynamic pages.
+
+## Other idea #2
+The proxy could also be expanded to proxy Gopher content over Gemini. Since Bucky already has a Gopher client, this would mostly involve writing a converter that takes in a Gopher map and spits out Gemtext, and then putting it all together. I think that sort of program might have a wider audience than my original proxy idea, which was mainly for my own benefit.
+
+## In the meantime
+Right now those other ideas are pretty much just speculation. The proxy server might be ready in a couple more hours work, but I'm not sure when I'll get back to it. The code will be public and freely licensed once it's working, but until now it's sitting in a private repo on my own Gitea instance.

content/gemlog/server_moved.gmi

@@ -0,0 +1,67 @@
+Meta(
+    title: "Server moved",
+    summary: Some("When an update fails, I bail"),
+    published: Some(Time(
+        year: 2022,
+        month: 12,
+        day: 17,
+        hour: 15,
+        minute: 31,
+        second: 24,
+    )),
+    tags: [
+        "Void",
+        "Linux",
+        "sysadmin",
+    ],
+)
+---
+Around the time that the Raspberry Pi 4 was released I took what was for me a pretty major plunge and got rid of all of my x86 desktop computers in favor of using the little Arm sbc's as home servers, while keeping a couple of x86 laptops for development and other day to day use. Currently there are three of them in a stack on my TV stand next to the router. The one running this capsule also run Apache and Gitea. Up until this week it had been running quite happily on OpenSuse Tumbleweed. I'm a fan of rolling release distros, even though the upkeep can be a little bit more work. I have been using mostly Arch for the past ten years, but Suse had proven to be stable for me for quite a long run. That run came to an end when the board failed to boot after an update.
+
+Now generally speaking I can recover most installations when things get borked, but there has been a trend in Linux the past decade to not only make the boot process "prettier" (by hiding most boot messages) but also to make the most out of a single kernel configuration by providing a minimal kernel combined with an initrd that loads the modules required to bring up a given system. I'm actually not a fan of the initrd approach, and in this case it was making recovery more complicated. Without boot messages it was impossible to know what was going on, and there was no usb keyboard available to change parameters.
+
+At any rate, after an initial unsuccessful attempt at recovery I decided to go ahead and change to another distro which I have been curious about for a fairly long time, Void. I've had, on my main laptop, a VM with Void installed for quite some time and it's been rock solid. It also gives me a chance to test against another C library as I'm using the Musl distribution. It's been encouraging enough that I installed onto real hardware last week as a dual boot. So I decided to take the plunge and move all of the services over.
+
+Installing Void onto an SD card was an all manual process. I chose to use the minimal tarball rather than "burning" an image. I don't know why more distros don't provide this sort of option. Well, I do actually, because everyone thinks that you need a nice point and click installer (hint: you don't). Basically, to install from a tarball you create your partitions, mount them, and extract the tar file onto the mounted partitions. You can then set things up if desired by chrooting into the new system and installing packages, editing configuration files and adding users.
+
+How do you chroot into an aarch64 system from an x86_64 laptop, you might ask? Qemu user mode emulation to the rescue. Basically, you need `qemu-user-<arch>` compiled statically, where `<arch>` is the architecture of the machine that everything is going to run on. In this case, the Qemu binaries are compiled to run on x86_64 and emulate aarch64. You need to register those binaries using binfmt_misc, and then the static binary is copied into /usr/bin of the target filesystem. You can then chroot into your Arm filesystem and do whatever is needed.
+ 
+=> https://wiki.archlinux.org/title/QEMU#Chrooting_into_arm/arm64_environment_from_x86_64
+
+In my case, there were packages for most of the software I need in Void's repositories. I installed Apache, Gitea, Rust, Cargo, Gcc and Neovim from the repositories. I needed the development tools because the servers that I couldn't get from the repositories had to be compiled from source. This is a nice trick actually, as you don't need a cross compiler. So before I ever put the SD card into the slot on the board it already had all of the software that was needed and what I hoped were working config files.
+
+## First boot
+Void uses Runit for init and process supervision. When the supervisor is started it takes a directory as an argument and supervises all of the services defined within that directory. Each service is in a subdirectory, which has at minimum an executable file named "run". This file is usually a shell script which starts the service, without sending it to the background. Should the service go down the supervisor will automatically bring it back up. To mark a system to not be started you just create a file in that subdirectory named "down". Nice and simple. All of the services that were to be run on this board were marked as "down" on first boot, to be tested and brought up one at a time so that I could avoid chaos, with the exception of opensshd which is needed to be able to log in from my laptop.
+
+Speaking of ssh, I made a mistake setting up pubkey authentication for my user. It's one that I've actually made before unfortunately, and hopefully I'll remember in the future. Anyway, in order for pubkey authentication to work the permissions should be 0700 for the .ssh directory and 0644 for the authorized_keys file. Anything more permissive than that and the daemon considers your keys to be insecure and will not accept pubkey based authentication.
+
+Anyway, other than ssh only allowing password auth for my user (which is since switched off in favor of pubkey only) everything came up just fine on first boot. I was also able to bring up my services for Gemini, Spartan and Finger one at a time without incident, although I later discovered a bug in Agis (my Spartan server). Attempting to bring up Gitea revealed some incorrect permissions, which was an easy fix in the end. The fun started with Apache.
+
+It seems that the `mpm_event` threading model which Apache has switched to as default is not supported with Musl libc. This is not documented anywhere, and the vanilla configuration shipped with Void has it as default. So in order to get the server to even attempt to run required switching that to `mpm_prefork`. After that ssl was a blocker until I tracked down the correct modules to enable and simplified my vhost configs. During this stage I learned that Certbot is now taking an aggressive stance towards requiring ssl everywhere, to the point that it now edits your vhost configs to redirect http requests using mod_rewrite. I absolutely fucking hate this. It should be my choice as the server admin whether I want to offer pages via regular http as well as https, period. I simply do not want any software changing my hand written configuration files. Probably time for a new Acme client at this stage.
+
+## Working with Runit
+Runit is crazy simple. As such, it does not have dependency handling for your services built in. So if you have a service that depends on another service, you have to hack that into your `run` file yourself. If you have some shell scripting experience this usually isn't hard though. Let's take a simple network server as an example. The files being served are on a separate disk mounted on /srv. So our service depends on /srv being mounted and the network being up. Since the supervisor will automatically try to start any service that exits, after a short delay, we can just check if the disk is mounted and ping the network, and exit if either of those conditions aren't met. I also redirect that output to /dev/null so as not to clutter up the console, but that's personal taste and you might want to see all of your error messages.
+```
+#!/bin/sh
+# Start up our hypothetical server
+
+# Check if /srv is mounted
+mountpoint /srv 2>&1 >/dev/null || exit 1
+
+# Ping the network interface
+ping -c 1 192.168.1.42 2>&1 >/dev/null || exit 1
+
+# Use `exec` here so that the service will have the same pid as this script
+exec myserver --nodaemon
+```
+All of the services that were installed via the package manager come with a service directory that's ready to go, but I had to roll my own for Agate, Agis and Toe (Gemini, Spartan and Finger respectively). I also modified a few of the service directories Void provided to hack in this sort of primitive dependency ordering. It's important if you do this to do so in a copy of the provided service directory so that it doesn't get nuked by the package manager at the next update, so I copied /etc/sv/apache to /etc/sv/apache_local and used the apache_local to make my changes. Those service directories then just get symlinked into /var/service.
+
+In order to verify that things are working before the system attempts to bring up your custom service repeatedly, you create a `down` file in the service directory before symlinking it as I mentioned above.
+```
+touch /etc/sv/myserver/down
+ln -sv /etc/sv/myserver /var/service
+# Start it once and verify it's working
+sv once myservice
+# If everything checks out, remove the `down` file
+rm -v /var/service/myserver/down
+```