Extended installation and pkgsrc chapters

src/SUMMARY.md

@@ -14,6 +14,7 @@
             - [Qemu](installation/binary/riscv/qemu.md)
             - [BeagleV](installation/binary/riscv/beagle-v.md)
     - [Installing from Source](installation/source/README.md)
+        - [The Build Process](installation/source/build-process.md)
         - [Downloading HHL Source](installation/source/downloading.md)
         - [Preparing to Build](installation/source/preparing.md)
         - [Building HitchHiker](installation/source/building-hitchhiker.md)
@@ -22,6 +23,11 @@
     - [Setting up Networking](configuration/networking.md)
 - [Managing Processes](s6/README.md)
 - [Installing Third Party Software](pkgsrc/README.md)
+    - [Getting Pkgsrc](pkgsrc/getting-pkgsrc.md)
+    - [Bootstrapping Pkgsrc](pkgsrc/bootstrapping.md)
+    - [Installing Binary Packages](pkgsrc/installing-binary.md)
+    - [Building Source Packages](pkgsrc/building-source.md)
+    - [Extending Pkgsrc](pkgsrc/extending.md)
 - [Updating HitchHiker Linux](updating/README.md)
     - [From Binary Sets](updating/binary.md)
     - [From Source](updating/source.md)

src/installation/binary/downloading.md

@@ -1 +1 @@
-# downloading
+# Downloading Hitch Hiker Linux binary tarballs

src/installation/source/build-process.md

@@ -0,0 +1,37 @@
+# The Build Process
+This section describes the steps that the HitchHiker build tree takes when
+building itself from source. Note that this is not a list of the commands that
+must be run by the user, those are actually quite simple. Rather, this page is
+here because it is both useful and reasshuring to understand what is being done
+when one does run the commands.
+
+It is expected that certain programs are available on the *host* system, ie the
+system that HitchHiker is being built on. In the case that certain programs do
+not exist, those programs are compiled in the 'bootstrap' portion of the build.
+HitchHiker puposely keeps this part light and prefers to use the native tools of
+the host.
+
+In the second half of the bootstrap phase, a sysroot compiler and linker is built
+and installed under ```/toolchain```. This toolchain may or may not be a cross
+toolchain, but in either case is of the same version of gcc and binutils as are
+to be used in the finished system and are built in such a way that they search
+for headers and libraries under ```/src/build``` rather than ```/```.
+
+In the case that we are compiling for an architecture that is different than the
+one which is being used to build the system, this toolchain is what is known as
+a *cross* toolchain. What this means is that the compiler and linker run on our
+*build* machine (the machine performing the build) but produce machine code that
+runs on our *host* machine (the machine the finished system will in fact run on).
+
+The actual build of the sysroot compiler takes place in several steps:
+1. The **binutils** package, which contains our linker and assembler.
+2. A temporary and minimally capable **gcc** compiler is built, which is only
+capable of building our C library.
+3. The *linux* kernel headers are installed into our *sysroot*.
+4. The temporary gcc and binutils built in step 1 are used to build *glibc*, which
+is installed into our *sysroot*.
+5. The final *sysroot* gcc compiler is built, which is a fully capable C and C++
+compiler for our final system.
+
+We then use the toolchain and bootstrap tools that we have built to compile the
+rest of the final system and install it into our *sysroot*.

src/installation/source/downloading.md

@@ -1 +1,22 @@
-# downloading
+# Downloading the Hitch Hiker Linux source
+The Hitch Hiker Linux source code is available as an xz compressed tarball from
+the hitchhiker-linux.org website, or can alternatively be cloned from the main
+git repository.
+
+The current stable source tarball can be fetched using *wget*:
+```Bash
+wget -c https://hitchhiker-linux.org/pub/stable/src/src.tar.xz
+tar -xJf src.tar.xz
+```
+Alternatively, should one wish to build the most recent development version of
+HitchHiker, the source can be checked out via *git* over https.
+```Bash
+git clone https://git.hitchhiker-linux.org/hitchhiker/src.git
+```
+>**_NOTE:_**
+>
+> Currently, the build tree expects to be located in the top level directory
+> **/src**, and it is therefore necessary to either ```cd /``` prior to running
+> the previous commands, or else to *bind mount* the actual source directory
+> onto /src prior to building. This requirement may be changed in the future in
+> order to increase flexibility.

src/installation/source/preparing.md

@@ -1 +1,34 @@
 # Preparing to Build
+If Hitch Hiker is being built on a Linux system other than itself, the following
+software is expected to be present:
+* Binutils - version 2.25 or greater
+* Bison - version 2.7 or greater
+* a functional **bzip2** and **bunzip2** command
+* a functional **diff** command
+* a functional **find** command
+* Gcc - version 6.2 or greater
+* a functional **grep** command
+* a functional **gzip** and **gunzip** command
+* Linux kernel version 3.2 or greater
+* M4 - version 1.4.10 or greater
+* GNU make
+* a functional **patch** command
+* GNU sed is required to build the Linux kernel
+* a functional **tar** command
+* Texinfo - version 4.7 or greater
+* a functional **xz** and **unxz** command
+Change directory to /src and create your **config.mk** file by copying
+**config.mk.sample** and editing it appropriately. The comments in the sample
+explain what each variable does.
+```Bash
+cd /src
+cp config.mk.sample
+vi config.mk
+```
+As a final step before building, it is neccessary to prepare the environment in
+which we are going to build Hitch Hiker. There is a script provided for this
+purpose which cleans up some environment variables and sets one **MAKEFLAG** in
+order to let **make** find our include files.
+```Bash
+source scripts/setenv.sh
+```

src/pkgsrc/README.md

@@ -1 +1,19 @@
 # Installing Third Party Software
+Hitch Hiker uses the [pkgsrc](https://www.pkgsrc.org/) framework for installing
+extra software on top of the base system. Contrary to what is common in the world
+of Linux, there is a clear distinction between the base system and all third
+party packages. This, in turn, allows for more developer focus on the base system
+and less time packaging software. This is the model followed by all of the various
+BSD systems, where the base system and packages are considered to be separate
+projects.
+
+### What is Pkgsrc
+Pkgsrc is a cross platform framework for packaging and installing third party
+software which is developed and maintained by the NetBSD project. Currently it
+works on all BSD systems, Minix, Darwin,MacOS X, and Linux to name a few, across
+multiple architectures.
+### Further Reading
+In addition to this chapter you are highly encouraged to read
+[The pkgsrc guide](http://www.netbsd.org/docs/pkgsrc/index.html) and keep it
+handy as reference, as a full reference to pkgsrc is beyond the scope of this
+chapter.

src/pkgsrc/bootstrapping.md

@@ -0,0 +1,33 @@
+# Bootstrapping Pkgsrc
+Before installing packages from source or from binary, it is necessary to
+bootstrap pkgsrc. This is neccessary in order to build the tools that pkgsrc
+requires which are not a part of the Hitch Hiker base system.
+```Bash
+cd /usr/pkgsrc/bootstrap
+./bootstrap
+```
+Once this is done it is necessary to either reload the shell startup files or
+log out and back in again, in order to add /usr/pkg/bin and /usr/pkg/sbin to our
+$PATH.
+```Bash
+exec zsh --login
+```
+As a final step, it is neccessary to add /usr/pkg/lib to the default linker path.
+This path is configured in **/etc/ld.so.conf** and can be extended with files
+placed in **/etc/ld.so.conf.d**. Finally, after configuring the linker paths, we
+run **ldconfig** to update the linker's cache of shared libraries.
+```Bash
+install -d /etc/ld.so.conf.d
+echo '/usr/pkg/lib' > /etc/ld.so.conf.d/pkgsrc.conf
+ldconfig
+```
+
+>**_NOTE:_**
+>
+> Ordinarily the preceding command would be run as the root user. It is, however,
+> possible to install packages as an unprivileged user by first making the package
+> prefix directory writable by that user. In the case of a normal installation
+> this would entail creating /usr/pkg and transferring ownership of it using the
+> **chown** command *before* bootstrapping pkgsrc. It is also possible to
+> bootstrap pkgsrc in one's own home directory. However, as this will break
+> compatability with any provided binary packages it is therefore discouraged.

src/pkgsrc/building-source.md

@@ -0,0 +1,28 @@
+# Building Source Packages
+The first step in building a package is to locate the directory within the pkgsrc
+tree that contains that package, and then entering that directory. There is a
+provided search utility called **pkglocate** which can be used to do a keyword
+search. However, in practice pkglocate is not a very effective search tool.
+
+A much better search tool can be found at ```pkgtools/pkgfind```. To install it:
+```Bash
+cd /usr/pkgsrc/pkgtools/pkgfind
+bmake install clean clean-depends
+```
+Now running ```pkgfind <searchterm>``` should return a much better list of search
+results.
+> **_NOTE:_**
+>
+> It is good practice to always run the ```clean``` and ```clean-depends```
+> targets after running ```bmake install```, to avoid leaving old work directories
+> all over the tree
+As demonstrated by our installation of ```pkgfind```, the general procedure to
+build a package using pkgsrc is to:
+1. Locate the directory containing the package we wish to install.
+2. ```cd``` into that directory
+3. Run ```bmake install``` in that directory.
+> **_NOTE:_**
+>
+> We run ```bmake``` as opposed to just ```make``` because pkgsrc uses the NetBSD
+> version of the make utility, which supports somewhat different ```Makefile```
+> syntax than GNU make.

src/pkgsrc/extending.md

@@ -0,0 +1,28 @@
+# Extending Pkgsrc
+While pkgsrc contains an extensive collection of software out of the box, there
+are times when there are packages that have not yet been packaged or have build
+failures on Hitch Hiker. For those feeling somewhat adventurous, here are two
+ways to extend the available packages, providing software not in the default
+distribution or alternative packages with Hitch Hiker specific bug fixes.
+## pkgsrc-wip
+The [pkgsrc-wip project](https://pkgsrc.org/wip/) is a collection of work in
+progress packages which for one reason or another have not yet been accepted into
+the main distribution. Much of the time these packages are perfectly functional,
+but might not work on every OS that pkgsrc supports. The pkgsrc-wip repository
+is managed via git, and due to the main pkgsrc framework being managed under CVS
+it is safe to clone pkgsrc-wip right into pkgsrc.
+```Bash
+cd /usr/pkgsrc
+git clone https://pkgsrc.org/wip/ wip
+```
+This will put all of pkgsrc-wip right inside pkgsrc at **/usr/pkgsrc/wip**. Once
+there, using wip is exactly the same as using the rest of pkgsrc.
+## pkgsrc-hhl
+The pkgsrc-hhl project is and extension to pkgsrc containing packages that are
+intended specifically for Hitch Hiker. These packages may add functionality not
+available in pkgsrc, fix build issues, or integrate packages from pkgsrc better
+with Hitch Hiker. Installation and usage of pkgsrc-hhl is similar to pkgsrc-wip.
+```Bash
+cd /usr/pkgsrc
+git clone https://git.hitchhiker-linux.org/hitchhiker/pkgsrc-hhl.git hhl
+```

src/pkgsrc/getting-pkgsrc.md

@@ -0,0 +1,8 @@
+# Getting Pkgsrc
+The current stable pkgsrc can be downloaded from
+[this link](http://cdn.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.bz2). It should
+then be extracted to /usr/pkgsrc.
+```Bash
+wget -c http://cdn.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.bz2
+tar -xjf pkgsrc.tar.bz2 -C /usr
+```

src/pkgsrc/installing-binary.md

@@ -0,0 +1,67 @@
+# Installing Binary Packages
+It is entirely possible to skip this step if one wishes to build all software
+from source. However, should one wish to work with binary packages provided by
+Hitch Hiker, it is recommended that the first package built be the
+[pkgin](https://pkgin.net/) package manager.
+```Bash
+zaphod@magrathea% cd /usr/pkgsrc/pkgtools/pkgin
+zaphod@magrathea% bmake install clean clean-depends
+```
+To configure pkgin, edit the file ```/usr/pkg/etc/pkgin/repositories.conf```
+to point towards the package repository appropriate for your architecture and
+version of Hitch Hiker. The [hitchhiker-linux.org](https://hitchiker-linux.org)
+server is structures as follows:
+```
+pub
+└── 2021q2              # Release version
+    ├── src             # Source tree
+    ├── aarch64         # Architecture
+    │   ├── packages
+    │   │   └── All     # Package repository
+    │   ├── release     # Release tarballs
+```
+Therefore, if the Hitch Hiker release was 2021q2 and the architecture of your
+machine was riscv64, the repository can be added with the following command:
+```Bash
+zaphod@magrathea% echo 'http://hitchhiker-linux.org/pub/2021q2/riscv64/packages/All' \
+  >> /usr/pkg/etc/pkgin/repositories.conf
+```
+Once pkgin is installed and configured it functions similarly to any good package
+manager one might have previously used under Linux, such as apt or pacman. To
+see the list of possible operations just run the ```pkgin``` command without
+arguments:
+```Bash
+zaphod@magrathea% pkgin
+Usage: pkgin [-cdfhlnPtvVy] command [package ...]
+
+Commands and shortcuts:
+list                (ls  ) - List installed local packages
+avail               (av  ) - List all available remote packages
+search              (se  ) - Search for a remote package
+install             (in  ) - Install or upgrade packages
+update              (up  ) - Refresh local and remote package lists
+upgrade             (ug  ) - Upgrade all packages
+full-upgrade        (fug ) - Upgrade all packages (deprecated)
+remove              (rm  ) - Remove packages and any dependent packages
+keep                (ke  ) - Mark packages that should be kept
+unkeep              (uk  ) - Mark packages that can be autoremoved
+export              (ex  ) - Display PKGPATH for all keep packages
+import              (im  ) - Import keep package list from file
+show-keep           (sk  ) - Display keep packages
+show-no-keep        (snk ) - Display autoremovable packages
+autoremove          (ar  ) - Remove orphaned dependencies
+clean               (cl  ) - Remove downloaded package files
+show-deps           (sd  ) - List remote package direct dependencies
+show-full-deps      (sfd ) - List remote package full dependencies
+show-rev-deps       (srd ) - List local package reverse dependencies
+provides            (prov) - Show which shared libraries a package provides
+requires            (req ) - Show which shared libraries a package requires
+show-category       (sc  ) - List all packages belonging to a category
+show-pkg-category   (spc ) - Show categories a package belongs to
+show-all-categories (sac ) - List all known categories
+pkg-content         (pc  ) - Show remote package content
+pkg-descr           (pd  ) - Show remote package long-description
+pkg-build-defs      (pbd ) - Show remote package build definitions
+stats               (st  ) - Show local and remote package statistics
+zaphod@magrathea%
+```