From d5be32d3048f9a33300c895a7770e1fa3d892ec4 Mon Sep 17 00:00:00 2001 From: Nathan Fisher Date: Sun, 21 Feb 2021 17:34:28 -0500 Subject: [PATCH] Ported a few more old blog posts Fixed links in templates to point to gitea --- .../news/building-packages-using-pkgsrc.md | 24 +++++++++++++++++ content/news/git-migration-to-gitea.md | 27 +++++++++++++++++++ ...sions-build-enhancements-and-the-kernel.md | 19 +++++++++++++ templates/base.html | 2 +- templates/categories/single.html | 13 +++++---- templates/index.html | 4 ++- templates/news.html | 4 ++- templates/tags/single.html | 4 ++- 8 files changed, 88 insertions(+), 9 deletions(-) create mode 100644 content/news/building-packages-using-pkgsrc.md create mode 100644 content/news/git-migration-to-gitea.md create mode 100644 content/news/testing-some-reversions-build-enhancements-and-the-kernel.md diff --git a/content/news/building-packages-using-pkgsrc.md b/content/news/building-packages-using-pkgsrc.md new file mode 100644 index 0000000..532c88e --- /dev/null +++ b/content/news/building-packages-using-pkgsrc.md @@ -0,0 +1,24 @@ ++++ +title = "Building packages using pkgsrc" +date = 2020-07-13 +[taxonomies] +tags = ["NonGNU","Package","Pkgsrc"] ++++ +HitchHiker was conceived to be as Unix-like as possible while running a Linux kernel and quite a bit of GNU userland. To that end a lot of Unix conventions are followed in the build tree, such as driving the entire system build using make. The original system of ten years ago had a ports tree and it's own shell based package tools, and I strongly considered bringing that back. However, I have an aversion to the constant reinvention of the wheel that plagues OSS development and do not desire to put another package manager out there if there is something that can be adapted to our needs. + +Having used pkgsrc previously (both natively on NetBSD and on Linux) I knew that it works, but not without some hacking if you're using it elsewhere than NetBSD. It's simply a huge project to maintain a packaging system that works across different platforms. However, I felt that my knowledge might have increased to the point of being ready to give it another go. + +The initial bootstrap brought up a few problems immediately. It turns out that dash is unable to bootstrap pkgsrc due to an incompatible echo builtin, and zsh, while not being officially unsupported, has issues with the way patching is done in pkgsrc due to the difference in the way zsh handles a no-match situation when using file globbing (most shells return an empty string, while zsh returns an error). My solution is something that I was already toying with - replace dash with mksh. + +As shells go, mksh is in a nice niche. It's small yet has a lot of features, and it doesn't require external libraries to build (even dash requires libedit if you want line-editing capability). If built as the lksh variant it is very POSIX compliant and makes a quite suitable /bin/sh for running shell scripts. It's probably not as powerful as bash or zsh, but that's not quite the point. It's also crazy fast. Some tests put it ahead of dash in speed, while some put dash ahead. The jury is out on that so far as I can tell, but it's definitely quick enough to see a notable increase in speed when running large shell scripts or running many scripts in parallel. + +I managed to bootstrap pkgsrc with no issues against lksh (installed as /bin/sh) and have had a lot of success building packages. There have been a few errors, and the process has helped me to find a few bugs, which I will enumerate below. + +* The ulimit builtin in mksh does not accept the -m option, breaking the build for libcups (and possibly other packages). There is already a solution, as the current sources for mksh accept ulimit -m on MacOSX for compatibility. I just extended that particular ifdef structure to include Linux as well and rebuilt the shell. My patch has been submitted upstream. +* The glibc build looks for bash specifically and sed's the shebang to point to wherever it finds bash. As there is a symlink pointing from /usr/bin/bash to busybox ash in the temporary toolchain, for compatibility reasons, that's how ldd was getting installed. This was only discovered when building certain packages, as the interpreter did not actually exist after removing the temporary tools. I've created another patch that changes this behavior to use /bin/sh, with no problems in usage so far. +* At some point (I'm not up for researching this) the function definitions for major() and minor() were moved from to in Linux. This caused a failure in the build of the Intel video driver shipped with pkgsrc and required patching 3 .c files to include the correct header on Linux. My patches have again been sent upstream to the package maintainer. +* There were numerous build failures when linking against our version of curses (NetBSD curses ported to Linux by Sabotage). Considering the number of packages that would need to be patched in order to retain this package I have reverted HitchHiker back to the official ncurses. + +The actual build directory for mksh has the source directly imported, bypasses the official build system and uses it's own Makefile to build both mksh and lksh in place in /bin, with intermediate object files placed in /obj/mksh and /obj/lksh. Quite a bit like our build of the ee text editor. The dash shell and libedit still exist in the tree for now, but are not built by default. + +It is not yet decided whether pkgsrc will be the permanent source for third party packages on HitchHiker, or whether a ports tree will be developed to replace it. There are pros and cons that must be weighed. The chief complaints are that there are a fairly significant number of packages that pkgsrc duplicates which are already present in the base system, and that there are still build failures on Linux that must be fixed. However, for the time being using pkgsrc will both provide a quick way to get a large number of packages built for HItchHiker and also help to contribute back to the pkgsrc project and hopefully make the framework function better on Linux in general. diff --git a/content/news/git-migration-to-gitea.md b/content/news/git-migration-to-gitea.md new file mode 100644 index 0000000..d525bc8 --- /dev/null +++ b/content/news/git-migration-to-gitea.md @@ -0,0 +1,27 @@ ++++ +title = "Git Migration to Gitea" +date = 2021-02-20 +[taxonomies] +tags = ["Golang","Git","Gitea","Web Development"] ++++ +As part of the continuing web site redesign I have migrated the git repo from +Gitweb to Gitea. + +Gitweb is a perl CGI script included with the git source distribution. It is very +basic, but does allow viewing repositories and provides url's for cloning. There +isn't anything particularly wrong with gitweb, but it's not exactly pretty and +requires running a CGI on the server. + +Gitea is written in Go and is a complete solution in a single binary. If anything, +gitea is a bit overkill for our purpose, but it looks great, provides an easy to +use interface for creating and managing repositories, and is quite easy on +resource utilization. As it runs it's own server, it also enabled me to put it +on another machine and just use a reverse proxy from the public facing server. +Those two machines (both Raspberry Pi's) are now dedicated to a single function +and running quite happily. It has also made my server configuration quite a bit +more simple. + +Of note, gitea is now the only dynamic content provider on hitchhiker-linux.org. +Everything else is static pages. I have taken the time to begin writing a handbook +and posting the source to gitea, which can be built and perused offline if +desired, which will be useful to new users upon first installing the system. diff --git a/content/news/testing-some-reversions-build-enhancements-and-the-kernel.md b/content/news/testing-some-reversions-build-enhancements-and-the-kernel.md new file mode 100644 index 0000000..a831d52 --- /dev/null +++ b/content/news/testing-some-reversions-build-enhancements-and-the-kernel.md @@ -0,0 +1,19 @@ ++++ +title = "Testing, some reversions, build enhancements, and the kernel" +date = 2020-07-03 +[taxonomies] +tags = ["Libraries"] ++++ +A lot has happened behind the scenes here. Not surprisingly, much of it has involved bug squashing and testing. + +First, the wild libc tangent. I spent around a week taking a much closer look at musl libc than I ever have previously and through much trial and error managed to compile the complete HitchHiker base system. I'll devote a full post to this at a later date, but to briefly sum up where things stand, there is enough still standing in the way to give me pause about moving to musl at the present time. HitchHiker has a number of goals in mind, and one of the main goals is the ability to relatively easily bootstrap the system from source while running a different Linux distribution. The build process currently involves three passes each of building gcc and binutils and two passes at Glibc. Going with musl would introduce quite a bit of additional complexity, as the only way I got it to fully work, coming from a Glibc based distro, was to build the full temporary toolchain, build the cross compiler and linker using the temporary toolchain (and linking to the temporary glibc in /toolchain), and then bootstrapping a native musl compiler from within a chroot. + +Additionally, during testing I uncovered some unexpected behavior in the compiled system. I hesitate to blame musl, as it is entirely likely that the offending packages are expecting Glibc and thus perpetuating some "gnuism's" that break compatibility with other libc implementations. As an example, the passwd utility from the shadow package does not wait for the user to input the new password, but skips right past and spits out a "password unchanged" message. Changing password worked using usermod and passing the password as an argument on the command line. Most likely this could be relatively quickly fixed, but it gives me pause to think that there could be any number of other future issues of this sort when replacing our libc. + +So long story short, we're forging ahead with Glibc for the present. I did some work to implement some configurability in the build system and created a mechanism whereby the installed locales can be chosen at the time the system is built. The default still compiles and installs all locales supported by Glibc. If one copies the file locales.mk.all to locales.mk and uncomments the lines for the desired locales, then only those locales will be compiled and installed. This saves some space and some fairly significant compile time. There is about a 9 minute difference between installing only en_US.UTF-8 and en_US.ISO-8859 and installing all supported locales on my laptop. + +Another bug became apparent only after rebuilding world with a fresh rebuild of the temporary toolchain. Originally we had replaced the full gettext suite with gettext-tiny, courtesy of Sabotage Linux. Apparently, vim had been using msgfmt from the temporary toolchain, which was an older build and the full binary from gettext. When the only binary available was the gettext-tiny version, it segfaulted during the vim build and compilation stopped. The only options were to hack the vim build system, disable nls in vim, or reinstate GNU gettext. At this juncture we're using gettext-tiny in the temporary toolchain and GNU gettext in the completed system, as I feel native language support should not be sacrificed in favor of a smaller system. HitchHiker should be fully functional whether one is based in an English speaking country or Cambodia. + +The next bug which was uncovered only reared it's head when trying to compile the kernel. We were using libelf-compat, again from Sabotage, which is a cutdown older version of elfutils with a cleaned up build system. During the kernel build there were cryptic errors about the size of symbold being larger than the size of the files with certain modules. Replacing libelf from libelf-compat with that from the current version negates the issue. + +On to the still incomplete kernel build system. Some work has been done here, but it's not yet fully functional. Complicating matters, we're going to be using the most current mainline kernel on x86_64 and the RPI foundation's kernel on RPI. Default configurations have been generated, and the plan is to use those if the user does not intervene but allow for customization in two places. The first place will be an ability to pass the build system a path to a different configuration that can be maintained by the user. The second will be a variable that can be passed that will drop the user into the Kbuild menuconfig target before building the kernel, in order to make any necessary tweaks. diff --git a/templates/base.html b/templates/base.html index cd36a71..1c11228 100644 --- a/templates/base.html +++ b/templates/base.html @@ -96,7 +96,7 @@ Download
  • - Source + Source
  • diff --git a/templates/categories/single.html b/templates/categories/single.html index 580b03c..ba987e0 100644 --- a/templates/categories/single.html +++ b/templates/categories/single.html @@ -4,15 +4,18 @@

    Posts tagged "{{ term.name }}"

      {% for page in term.pages %} -
    • +
      + + {{ page.title }} + {% if page.date %} - {{ page.date }} -- +
      {{ page.date }}
      {% endif %} - {{ page.title }} {% if page.summary %} -
      {{ page.summary | markdown(inline=true) | safe }}... +

      {{ page.summary | markdown(inline=true) | safe }}...

      {% endif %} -
      • + Read More -> + {% endfor %}
    diff --git a/templates/index.html b/templates/index.html index ff3ec2a..86bdf73 100644 --- a/templates/index.html +++ b/templates/index.html @@ -20,7 +20,9 @@ check out the Documentation. {{ page.title }} -
    {{ page.date }}
    + {% if page.date %} +
    {{ page.date }}
    + {% endif %} {% if page.summary %}

    {{ page.summary | markdown(inline=true) | safe }}...

    {% endif %} diff --git a/templates/news.html b/templates/news.html index bd36230..9328288 100644 --- a/templates/news.html +++ b/templates/news.html @@ -10,7 +10,9 @@ {{ page.title }} -
    {{ page.date }}
    + {% if page.date %} +
    {{ page.date }}
    + {% endif %} {% if page.summary %}

    {{ page.summary | markdown(inline=true) | safe }}...

    {% endif %} diff --git a/templates/tags/single.html b/templates/tags/single.html index 7b9ffc5..ba987e0 100644 --- a/templates/tags/single.html +++ b/templates/tags/single.html @@ -8,7 +8,9 @@ {{ page.title }} -
    {{ page.date }}
    + {% if page.date %} +
    {{ page.date }}
    + {% endif %} {% if page.summary %}

    {{ page.summary | markdown(inline=true) | safe }}...

    {% endif %}