Nevertheless its stability and performance can’t be denied, and so here I am, taking to this text-based medium once again to jot down some notes, for the benefits of my future self and anyone else who may stumble upon this page.

Splitting access

In my homelab I host a single multi-node instance of K8S that hosts services available both, exclusively on my lan, as well as from the Internet. To achieve this I create separate ingress controllers, one for each of the use cases. This allows me to forward traffic from the internet to one of them, on its dedicated IP address, but not the other, hosted behind a different load balancer IP on my local net.

Be default the Traefik K8S helm chart will create a service, which needs to be set to type LoadBalancer. As mentioned in the previous blog entry, this will trigger MetalLB to assign our new ingress instance a dedicated virtual IP address to our new service. Let’s say we did that and called this instance of the Traefik helm chart traefik-private. All examples bellow will use the official Traefik helm chart.

~/Code/k3s/traefik-private main ❯ tree .
.
├── Chart.lock
├── charts
│   └── traefik-34.1.0.tgz
├── Chart.yaml
├── templates
│   └── dashboard.yaml
└── values.yaml

3 directories, 5 files

Since we will be creating a second ingress controller alongside it, we need to take care of a few configuration steps in the values.yaml. We need to make this ingress unique, so that we can assign different applications in our cluster to one or the other ingress controller. We can do this by naming each ingress class and setting one of them to be the default one. In my case, my private ingress will be default.

traefik:
  {..}
  # Make service type LoadBalancer to give it an IP address on our local network
  service:
    enabled: true
    single: true
    type: LoadBalancer
  # Configure the ingress class kubernetes object itself
  ingressClass:
    enabled: true
    isDefaultClass: true
    name: "traefik-private"

  # Also configure the kubernetesIngress provider, otherwise Traefik won't consume the relevant ingress entries
  providers:
    kubernetesIngress:
      enabled: true
      ingressClass: "traefik-private"

Now we need to make a copy of this chart, and rename it to something like traefik-public, modify the values.yaml slightly, then install it on our cluster as well, and boom we have two ingresses, each with its own IP address.

traefik:
  {..}
  # Configure the ingress class kubernetes object itself
  ingressClass:
    enabled: true
    isDefaultClass: false
    name: "traefik-public"

  # Also configure the kubernetesIngress provider, otherwise Traefik won't consume the relevant ingress entries
  providers:
    kubernetesIngress:
      enabled: true
      ingressClass: "traefik-public"

Voila, this is what we’ve got so far in graphical form. Two targets that we have the freedom to configure routing to and from as we please.

Getting in

So now to the whole reason why we bother splitting these ingresses on a network level. When I am out and about, I still want to have access to the services that my homelab provides. This usually is done with the use of some sort of a VPN. The basic (or actually not so basic) outline of how to achieve Wireguard-based VPN access to your infra can be, yet again, found in one of my previous posts. This works very well for the vast majority of services. What it does not work for, are services which need a public presence. For example, a Matrix Synapse server, a Mastodon server, or even a Nextcloud server in case you want to be able to share links to files with people like you can do with popular commercial cloud-based alternatives. The solutions is definitely not to just expose everything, so let’s be smart about it.

This rather complicated graph (don’t bite, it was late at night when I drew it) hopefully allows you to digest the next few paragraphs a little easier. In order to allow public access to services running inside your homelab, you need to find a way to give them public IP addresses. Since we are not opening ports or doing any nonsense that would require your ISP to give you a publicly routable IP, we are going to take advantage of the infra we’ve already got plus a little extra.

In this setup we have 3 important entities. The subnet router within your home network, the Wireguard server in the cloud somewhere and a reverse proxy server, also in the cloud, where the last 2 have public IPs. Logically, you could combine the last two, but I like to keep them separate, both to make the setup cleaner and to separate the two concerns for security reasons.

The Wireguard network in question is a star and spokes design, meaning we have a central server that every other node connects to, and via this node they can therefore reach every other node. The subnet router and reverse proxy are, although special types of nodes, still just nodes. Addressable directly on the 10.10.0.0/24 subnet as per the diagram. As a VPN user I also get assigned an address on this subnet, and therefore can access all of the spokes once connected. How does my roadwarrior machine know how to find the homelab subnet, where all of my services reside, i.e. 10.0.0.0/24? Two steps. The Wireguard machine advertises that the 10.0.0.0/24 subnet is available via the subnet router node, in this case 10.10.0.5. This is configured using IPTables. The subnet router in turn is configured to forward all traffic coming from the Wireguard host (10.10.0.1) targeted at the subnet 10.0.0.0/24 via its interface on said subnet, so 10.0.0.7. As a result of all this, my roadwarrior node can access both ingresses (10.0.0.5 and 10.0.0.6), and any other hosts on this subnet, including the local DNS server I host, which will translate the URLs to their local addresses.

The reverse proxy node is more specialised. It is designed for a single function, to forward only select requests coming from the internet on its port 443 to the public ingress IP address only. We can do this by deploying HAproxy on it. Following is an example of a simple HAproxy config file, matching what the diagram above presented:

# Configure the frontend, so how we receive traffic on the reverse proxy
frontend traefik_public
        # This needs to be your actual IP address you're trying to bind, the Xs are only for this example...
        bind X.X.X.6:443
        default_backend    traefik_public_server
        # Enable send X-Forwarded-For header
        option             forwardfor

# Now configure backend, so where do we send the requests
backend traefik_public_server                                                                                                 
        balance            roundrobin
        # We select the single IP as a backend, corresponding with our traefik public instance in the homelab
        server             backend01 10.10.0.6:443 check

The config file is usually located at /etc/haproxy/haproxy.conf. To find out more you can look up the HAProxy website and the official configuration documentation.

This configuration is simple because we are telling HAproxy to forward all of the HTTPS traffic it receives down to our homelab traefik instance designated as the “public” one. It can do that because, just like my roadwarrior laptop, it itself is also a node on the wireguard network, and it also knows how to reach 10.10.0.0/24 subnet.

Finally to the point. We can advertise our public services using a public DNS provider, and tell it to point to the HAproxy, which will happily forward those inwards via the Wireguard net. Therefore, as per this example. we can set test1.example.com to point to X.X.X.6. Likewise, since we don’t want people to connect to test2.example.com, so we just won’t set that in public DNS right? So, that is correct, however anyone can host their own DNS server or just override their local DNS config on their machine. This means that, because HAproxy in with this configuration forwards all HTTPS traffic indiscriminately to the downstream server, if we only had one ingress in the homelab for all services, we would be exposing them one way or another. So, by splitting them, we have network segregation and an easily configurable infrastructure, where changing the service from public to private and vice versa is as simple as tagging its Kubernetes ingress object with the correct ingress label, then publishing or deleting the public DNS entry. Now, that’s neat.

I wanna see it!

Something I always struggled with in the past was configuring the Traefik dashboard correctly. By correctly I mean so that it actually works. This is double annoying, because we have two ingress controllers, so on average I fail twice per cluster on this task. This is where I normally engage you with a semi-whity story about how I embarked on a journey of discovery and what hardship I overcame to get to where we are not. I ain’t doing that now, I just fixed my dashboards, and I ain’t breaking them again just to take screenshots for you. Deal with it.

So, what do I want? I want access to dashboards from both the ingress controllers AND I don’t want the whole freaking internet to have access to either of them. That’s not so much to ask, is it? The defaults in the Traefik chart enable the dashboards using their fancy CRD ingressRoute. All I will say is, NO. Ok, not really. So we will still need to enable the dashboard ingressRoute item but we will not configure anything but the most basic stuff for it, and we will use a normal honest to God ingress object to actually expose this dashboard.

traefik:
  {...}
  ingressRoute:
    dashboard:
      enabled: true
      annotations: {}
      labels: {}
      matchRule: Host(`dashboard-public.example.com`)
      services:
        - name: api@internal
          kind: TraefikService
      entryPoints: ["traefik"]
      middlewares: []
      tls: {}

We will have to do this in both the charts for both ingress controllers, obviously changing the match rule to match the hostname you’ve chosen. You will notice that we are selecting the “traefik” entrypoint, this is on purpose. We need to enable this entrypoint now, like this:

traefik:
  {...}
  ports:
    traefik:
      port: 8080
      expose:
        default: true
      exposedPort: 8080
      protocol: TCP

This will make sure that the ingress controllers will not respond to our dashboard-public.example.com or whatever on the normal ports that they serve traffic to real users, like 443 or 80. This is important for the public ingress especially, as that would mean people on the internet could mess with your dashboard if they could figure out the hostname you gave it, and we don’t want that. What do we want? We want the private ingress controller to serve both dashboards, so that I can view it when I am authenticated on my VPN damn it!

I started digging and found this: https://github.com/traefik/traefik-helm-chart/issues/143. Traefik devs basically said, ain’t nobody got time for this BS, just make a generic service and ingress entries and begone demon! So off I went.

Since we already set the dashboards to be available on the traefik port 8080, and both Traefik ingresses will have created a service each for this, I have foregone that part, and instead went for just an extra ingress object per controller.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-public-dashboard
  annotations:
    # This here annotation is the key btw:
    kubernetes.io/ingress.class: traefik-private
    traefik.ingress.kubernetes.io/router.entrypoints: websecure,web
spec:
  tls:
    - hosts:
      - dashboard-public.example.com
      secretName: wildcard-cert
  rules:
    - host: dashboard-public.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: traefik-public
                port:
                  number: 8080

Both ingress object will look the same, you just need to change the name, host rule and target service name. Importantly the annotations need to stay the same. This will mean that only the ingress controller that I called traefik-private (in this case available via 10.0.0.5) will configure entries for both dashboards, and put them on its websecure and web entrypoints, which are the standard port 443 and 80 bois. Here you go Hilmar, I know you struggled with this, now you know how to do it and I can always refer back to this page also. Phew…

~ ❯ k get nodes                                                 8s ○ k3s
NAME                STATUS   ROLES                  AGE     VERSION
k8s01               Ready    control-plane,master   2y51d   v1.27.2+k3s1
k8s03               Ready    worker                 2y50d   v1.27.2+k3s1
k8s05               Ready    storage                2y50d   v1.27.2+k3s1
k8s11               Ready    control-plane,master   594d    v1.27.2+k3s1
k8s13               Ready    storage                242d    v1.27.2+k3s1
k8s14               Ready    worker                 14d     v1.27.2+k3s1
k8s21               Ready    control-plane,master   46d     v1.27.2+k3s1
k8s22               Ready    worker                 45d     v1.27.2+k3s1
k8s23               Ready    storage                45d     v1.27.2+k3s1

Still, I am running k3s version 1.27.2. Which is a few major versions behind, latest one currently being v1.31.4.

Memory

Now, because the last time I set up a new cluster was over two years ago, I don’t remember exactly what steps I needed to take to configure certain aspect of the cluster. Things like ingress and load balancers you only really configure once at the beginning, then you don’t need to touch them pretty much ever again. In my current cluster I am using MetalLB as the L2 load balancer, then Traefik as the ingress controller. Both of these components have also moved on since the last time I set them up, and despite my best efforts I have also fallen behind a few versions on those. Well, and here we are. At least this time I am taking notes, so the next time I do it I can at least refer to this page. This is a gift from me to my future self.

MetalLB

I won’t copy-paste the entire values.yaml for metallb. I am using the official chart in its vanilla state. I basically am not changing any configuration whatsoever. Basically keeping all default values as found here.

As I am writing this, I am using version 0.14.9 of the metallb chart.

apiVersion: v2
name: metallb
description: Metallb chart, configuring bare metal load balancer
type: application
version: 0.1.0
dependencies:
  - name: metallb
    version: 0.14.9
    repository: https://metallb.github.io/metallb

It seems to me that unlike in older versions of metallb, the actual configuration for the load balancer gets installed via a couple of custom resources, which are not generated via changes to the values.yaml. Therefore I defined them as following to mach my use case:

I created addresspool.yaml which defines a default pool of IP addresses that I want to use for my LoadBalancer-type K8S service entries. Since I am using a non-standard IP block, I added each address separate as a list item, masked at /32.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - 192.168.0.10/32
  - 192.168.0.11/32
  - 192.168.0.12/32
  - 192.168.0.13/32
  - 192.168.0.14/32

At first I though that was it, but it turns out that MetalLB will not configure the speaker pods to advertise my service IPs if I don’t explicitly tell it to. For that I created another custom resource of kind L2Advertisement. It enables the virtual IP address advertisement feature for each address pool linked. In this case we are linking the default address pool, as defined in the previous step.

apiVersion: metallb.io/v1beta1
kind: L2Advertisement 
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
  - default

Traefik

In order of messing around and figuring out what to do next, at this point I started to configure Traefik and its dashboard, to make sure that the LB services are working correctly and MetalLB IPs are getting advertised. But knowing what I know now, I wanted to make sure the dashboard is exposed over https and has a proper certificate. So, if you’re following step-by-step, this is the time to insert a TLS cert secret to use. I am using a wildcard cert, which I call wildcard-cert like so:

kubectl create secret tls wildcard-cert --cert ./fullchain.pem --key ./privkey.pem -n default

I won’t tell you how to obtain your certs, that’s its own topic for another time…

Just as with MetalLB I am using the official Traefik helm chart, this time version 33.2.1.

apiVersion: v2
name: traefik-private
description: Traefik chart, configuring private ingress controller
type: application
version: 0.1.0
dependencies:
  - name: traefik
    version: 33.2.1
    repository: https://traefik.github.io/charts

Again, basing my chart on the default configs. I only made a couple of minor changes to the values.yaml. (The full file can be seen here.)

I configured this Traefik instance to be the default ingress. Calling it private as this one will be resolvable only on the local network. I usually create a second ingress dedicated to external traffic for services exposed to the internet.

traefik:
  {...}
  ingressClass:
    enabled: true
    isDefaultClass: true
    name: "private"

I modified the dashboard ingress route block. I added the “websecure” entrypoint both to test it, and to have a TLS-protected endpoint where I can check the dashboard. The “web” endpoint is left there too, though I am yet to set up the https redirect middleware. I am linking the TLS cert that I described in the previous step.

traefik:
  {...}
  ingressRoute:
    dashboard:
      enabled: true
      annotations: {}
      labels: {}
      matchRule: Host(`treafik-private-dashboard.domain.com`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
      services:
        - name: api@internal
          kind: TraefikService
      entryPoints: ["websecure","web"]
      middlewares: []
      tls:
        secretName: wildcard-cert
        domains:
        - main: treafik-private-dashboard.domain.com

Lastly I enable the dashboard using additional args block.

traefik:
  {...}
  additionalArguments:
    - "--api.dashboard=true"
    - "--log.level=INFO"
    - "--api.insecure=false"

This entry isn’t about a project, but about trying to find a solution to a seemingly small issue, then going down a rabbit whole and coming out of it with some new knowledge, ready to be slapped on this here website. It required a lot of searching around and learning new things, but we got there in the end.

Just install Linux they said, it will be fine they said…

A dear friend of mine has gone onto the path of becoming a data scientist and a programmer of sorts, pursuing many a Udemy course in search of Python coding prowess. I volunteered to be a partner in crime for his coding adventure and tried to support him throughout his learning process as best I could, but there was a recurring theme to our regular coding get togethers: we spent half our time getting his computer to work instead of coding. I won’t lie, this was exaggerated by two facts: he was running Windows 10, and I know very little about Windows at this point. Bootstrapping WSL2 on his machine helped only partly, as that tool has its own problems (for example, not allowing to utilize VPN connections within the VM) which hamstrung some of our collaboration attempts.

Change

It was time for change. I encouraged my friend to give Linux a try, not as a the only OS on his main computer, but on an extra hand-me-down device he had laying around spare. Neither of us were certain that moving to Linux was going to be smooth and painless, so this small step had the best chance of success with little to no sacrifices.

We settled on Debian. Mainly for the reason that I am running either Debian or its derivatives on the daily, so it would decrease friction in our collaboration. It is also pleasantly stable and free of corpo-influences, that may make it unusable down the road. It was a joy when I received a message that my partner in Python-crime was going through the installation wizard one day, though he reached out because he had found the process less than smooth. Although pretty much all the default settings in the installer should give him a usable OS, he missclicked either here or there, and ended up with a dud of an install. This reminded me… That there is a way to preseed the installer with all the config necessary, and that I should be able to do if for him, so that he can skip the boring cumbersome part of this ordeal and go straight to the tasty, juicy coding bits instead!

You’re a wizard!

If you are familiar with the way VMs are typically deployed in Cloud environments, you will no doubt know the term cloud-init. It is a mechanism that allows to pass scripting and instructions into a virtual machine as it boots in order to automatically provision it. You may even be one of us dinosaurs, who goes way back, and remembers the days before the great on-prem exodus, where pxe-boot ruled the land of auto-provisioning. Admins would prepare special OS bootstrap images, expose them to a machine via DHCP options 66 and 67, and voila, glorious automation!

As you may imagine, neither of these options were available to me, though I still wanted to help my friend and offer him a pleasant start of his Linux adventure. I remembered somewhere that there is a way to pre-configure the Debian installer with all the options, in order to make the process essentially hands off and foolproof. So I started digging…

Documentation, sweet documentation…

A few minutes of searching yielded promising results. I found that, indeed, there is a preseeding mechanism for the DebianInstaller. I first landed here: https://wiki.debian.org/DebianInstaller/Preseed, then went on to click a bunch of links on this landing page. Appendinx B is an invaluable resource going in-depth on all the different options available. It however didn’t provide me with an immediate overview of what a minimal functioning preseed.cfg file should look like.

At first I tried the brute force method, to get a feel for what is possible. I have read that one can use an existing installation of Debian to automatically generate a preseed file, which essentially will collate all the options a user selects manually in the installation wizard for reuse in subsequent installations. I obtained this information from an excellent blog entry on linuxconfig.org. Credit where credit is due, most of the following content on this page comes from said blog entry, so hats off to the Egidio Docile for writing the piece! Nevertheless, I will follow with my findings, in part to create a note for my future self to remind me what I’ve done.

First try

I decided to create a new VM on my trusty framework laptop, going through the Debian installer manually, selecting the options appropriate to my friend’s needs. This, as expected, went smoothly. In the now running VM I opened the terminal, installed the necessary utility and generated myself a preseed file:

sudo apt-get update && sudo apt-get install debconf-utils
sudo debconf-get-selections --installer > preseed.cfg

Then SCP’ed it back onto my framework. At this point I realised, I am not sure how to deliver this config into the installer. I was convinced that the best way forward would be to embed it in the ISO itself. This way there should be no need for my friend to interact with the installer at all, I hoped… Well, it didn’t quite work.

It is not straightforward it seems to modify an ISO. Searching on Kagi and using their “quick answer” AI generator, I played with mounting the Debian netinstall ISO, copying its contents into a new directory, plus the preseed.cfg file, then packaging the directory again into a bootable ISO using the genisoimage utility. I was discouraged after a few attempts to boot a VM from my new ISO, and decided that even if I can make it work, the long turnaround time for baking my own ISOs with various preseed.cfg file versions makes it a bad choice for experimentation.

Thankfully there are multiple other mechanisms that can be used to deliver the preseed file to the installer, and the easiest one is to let it be downloaded during the course of an “automatic installation”. This has the downside of expecting that my friend will have to enter the URL on their machine to tell the installer where to get the file in question, but it let me iterate quickly and learn how to properly write such a file. As a way to share my preseed.cfg I uploaded it to my Nextcloud instance and created a share link for it. This can then be entered into the installer. On the other end of this, I mounted my Nextcloud storage locally on my framework, so that I could edit the file directly in real time, then boot up a fresh DebianInstaller in a VM and get it to download the current working version, rinse and repeat.

At this point it became apparent that the automatically generated preseed file would not work, and as some online guides alluded to, such generated files are extremely verbose and working with them can be tricky. Well, nothing was lost, I moved on to plan B, which is to actually learn how to write a functioning preseed file!

Plan B with a capital B

Now that I had a way to quickly test preseed files in a VM I decided to try to actually understand what goes into one. Again, I used linuxconfig.org blog post as a baseline, and just collating their snippets produced positive practical results for me:

d-i debian-installer/language string en
d-i debian-installer/country string US
d-i keyboard-configuration/xkb-keymap select us
d-i netcfg/get_hostname string machine
d-i netcfg/get_domain string
d-i passwd/root-login boolean false
d-i passwd/user-fullname string Name Surname
d-i passwd/username string username
d-i passwd/user-password-crypted password $6$1osx5k3fldnaKAC/$a0G7hmSFFXUvC3M.vzSPhSqJ1KOh5Y5K5hcn7tBzMpcyJBloLGCFynmHnGx065F5QrDpuSaxlQwmAQrMTxlCl.
d-i time/zone string US/East
d-i partman-partitioning/choose_label string gpt
d-i partman-auto/disk string /dev/vda
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/method string lvm
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish 
d-i partman/confirm boolean true 
d-i partman/confirm_nooverwrite boolean true
d-i apt-setup/use_mirror boolean true
d-i mirror/country string US
d-i mirror/http/mirror select deb.debian.org
d-i mirror/http/proxy string
d-i apt-setup/cdrom/set-first boolean false
popularity-contest popularity-contest/participate boolean true
tasksel tasksel/first multiselect gnome-desktop,standard,ssh
d-i pkgsel/include string vim git
d-i finish-install/reboot_in_progress note
d-i cdrom-detect/eject boolean false

As per the guide, I used the mkpasswd utility to pre-assign the user a password of “password”. Yes, I will tell my friend to change it to whatever he wants. It’s either that or become a part of a botnet I guess…

sudo apt install whois
mkpasswd -m sha512crypt

This file takes care of most of the options in the Debain Bookworm installer. Most. It failed for me during the Grub config step, where I was forced to select the boot partition manually. The guide does not mention Grub setting anywhere, so I went back to the official docs to see what’s what and get some damn answers.

Voila!

Here is the info we needed: https://www.debian.org/releases/stable/i386/apbs04.en.html#preseed-bootloader

# Grub is the boot loader (for x86).

# This is fairly safe to set, it makes grub install automatically to the UEFI
# partition/boot record if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true

# This one makes grub-installer install to the UEFI partition/boot record, if
# it also finds some other OS, which is less safe as it might not be able to
# boot that other OS.
d-i grub-installer/with_other_os boolean true

# Due notably to potential USB sticks, the location of the primary drive can
# not be determined safely in general, so this needs to be specified:
#d-i grub-installer/bootdev  string /dev/sda
# To install to the primary device (assuming it is not a USB stick):
#d-i grub-installer/bootdev  string default

# Alternatively, if you want to install to a location other than the UEFI
# parition/boot record, uncomment and edit these lines:
#d-i grub-installer/only_debian boolean false
#d-i grub-installer/with_other_os boolean false
#d-i grub-installer/bootdev  string (hd0,1)
# To install grub to multiple disks:
#d-i grub-installer/bootdev  string (hd0,1) (hd1,1) (hd2,1)

# Optional password for grub, either in clear text
#d-i grub-installer/password password r00tme
#d-i grub-installer/password-again password r00tme
# or encrypted using an MD5 hash, see grub-md5-crypt(8).
#d-i grub-installer/password-crypted password [MD5 hash]

# Use the following option to add additional boot parameters for the
# installed system (if supported by the bootloader installer).
# Note: options passed to the installer will be added automatically.
#d-i debian-installer/add-kernel-opts string nousb

An MD5 hash for a password for grub can be generated using grub-md5-crypt, or using the command from the example in Section B.4.5, “Account setup”. 

As per the above block I was clearly missing the d-i grub-installer/only_debian boolean true and d-i grub-installer/bootdev string default lines.

The final config file looks something like this:

d-i debian-installer/language string en
d-i debian-installer/country string US
d-i keyboard-configuration/xkb-keymap select us
d-i netcfg/get_hostname string machine
d-i netcfg/get_domain string
d-i passwd/root-login boolean false
d-i passwd/user-fullname string Name Surname
d-i passwd/username string username
d-i passwd/user-password-crypted password $6$1osx5k3fldnaKAC/$a0G7hmSFFXUvC3M.vzSPhSqJ1KOh5Y5K5hcn7tBzMpcyJBloLGCFynmHnGx065F5QrDpuSaxlQwmAQrMTxlCl.
d-i time/zone string US/East
d-i partman-partitioning/choose_label string gpt
# WARN: The volume name may be different depending where you're installing!
d-i partman-auto/disk string /dev/vda
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/method string lvm
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish 
d-i partman/confirm boolean true 
d-i partman/confirm_nooverwrite boolean true
d-i apt-setup/use_mirror boolean true
d-i mirror/country string US
d-i mirror/http/mirror select deb.debian.org
d-i mirror/http/proxy string
d-i apt-setup/cdrom/set-first boolean false
popularity-contest popularity-contest/participate boolean true
# Missing bootloader config
d-i grub-installer/only_debian boolean true
d-i grub-installer/bootdev  string default
tasksel tasksel/first multiselect gnome-desktop,standard,ssh
# Extra packages to install
d-i pkgsel/include string vim git zsh htop curl git ripgrep fd-find bat byobu kitty-terminfo tldr
d-i finish-install/reboot_in_progress note
d-i cdrom-detect/eject boolean false

I must say, I am quite happy with how this turned out. It is certainly going to help me bootstrap VMs and machines faster going forward. There are many other options that you can configure here, the most notable is probably the network config, although the pitfall of the public URL based delivery method for the config file means that the installer needs Internet access before the configuration can be applied, so do bear that in mind.

Lastly, something to remember is that the d-i partman-auto/disk string /dev/vda line may differ from one install to another. From what I gather, the default disk path in a KVM install of debian will always be /dev/vda, while bare metal would be /dev/sda (for a SCSI or SATA device). Again, for detailed info on this refer to the official docs: https://www.debian.org/releases/stable/i386/apbs04.en.html#preseed-partman

Oh, and as a bonus, I also found this sample config helpful to put it all in perspective: https://www.debian.org/releases/bullseye/example-preseed.txt.

Happy times

As of the time of writing this, we are yet to attempt this automated install process remotely on my friend’s machine. I must say, it is quite exciting that we have the technology to automate and simplify things at a distance, and I am looking forward to see how we will fail at it tomorrow! Though I would be hesitant to bet that any Python coding will be happening on that new machine in the next day or so, knowing our luck. And with that, till next time, dear reader.

This one is mostly for those of my friends who are interested in my adventures. And also for me to look back on and remember the journey. The following will be mostly un-redacted transcriptions of my voice notes which I recorded at the beginning of each day of the journey, talking about the previous day. It should paint a good story, and allow me to anchor the images and voice notes a little.

Day 1

▶️ Day 1 recording

Hello, it’s the morning of the 18th of July, day after the flight. I woke up after about six hours of sleep, it’s about 8 a.m.

Right now the Sun is very bright outside so I think it’s time to wake up and start going against the jet lag

So I was advised by Lufthansa to be really early at the airport because it is still the tail end of Euro 2024 taking place here in Germany so expected high volume of passengers. Got to wake up at 6, head to the train station at 7, made it to the airport well in time, just after 8, luggage drop-off really easy, used the fancy terminal first time ever, everything is pretty automated so that’s nice.

Actually I still remember back in the day flying with with Ryanair between Poland and UK and having to check luggage was an absolute pain and waiting in queues for hours and here was just pretty instant, you go to the counter if you’ve checked in before then you just scan your pass and it prints you a label for your luggage, you put luggage on a tape and off you go. It’s actually pretty sweet.

Security checks without any problems passport checks didn’t happen in Berlin obviously because we’re flying to Frankfurt first. Had some food, second breakfast basically, then went to my flight gate when it was announced shortly after. Really I didn’t even have to wait much even though I was way early.

I took a little action cam with me to see if I can maybe record some footage and make a little compilation for people. A lot of people asked me for photos and some sort of narrative or diary around this trip so we’re gonna give it a shot, kind of inspired by Pete’s diaries. Diaries from his trips are always great.

So we got on the first plane to Frankfurt easy mode, only an hour hop, maybe even less than that. Got off in the, apparently, dreaded Frankfurt Airport and I have to say that was probably the simplest change. I didn’t have to walk far, there were no issues at the gates, no problems with passports, didn’t have to walk much basically means that you end up going to the airplane by bus but really I still feel like the the fact you don’t have to walk for miles and miles makes it a good change.

The flight to Vancouver was more colorful I would say. I ended up being seated on the aisle side on the right side of the middle row. Had fantastic trip companions, two ancient arab folks of some kind, an old grandpa and grandma who decided that they basically need the whole row to themselves, the whole four seats in the middle and they didn’t want to give space to other passengers that booked that space. They just wouldn’t even talk to the crew, they didn’t care and they didn’t really speak English. The crew gave up and gave passengers other seats, though I did end up sitting on my seat in their row. In the end they basically took three spaces out of the four and the grandma that was sitting next to me was… uh well it’s not manspreading but it works the same I guess… I felt quite cramped with her elbowing me for the whole eight goddamn hours…

The worst thing though probably was that she was just incessantly coughing without covering her mouth or anything and demanding non-stop attention from the crew, asking for drinks and food all the time, which I have to say the crew took on with a smile on their faces. They were very professional and nice about it, I would probably have gone crazy if I had to serve these people.

I’m not sure what it was. My instinct tells me there was probably some sort of a big family gathering or a wedding because there was a bigger group of these either Arab or Hindu folks on the plane because when we were asked to board at the Frankfurt Airport, boarding is supposed to commence in groups and every passenger gets a boarding group. First families with kids and disabled people board, then group one and group two, then group three which is supposed to make the process orderly. But this mass of people which I presume most of do not speak neither English nor German, at that point simply just stood in the way and waited at the gate completely disregarding the crew and basically blocked access to anyone with kids or disabilities or whatever group the staff were calling out. The staff eventually just gave up and told people to board as they are.

In contrast once we landed in Canada it was a completely opposite vibe. There were barely any people at the gate. It was a small propeller plane to take us on the last hop from Vancouver to Cranbrook. There was one older lady at the gate managing everything, she announced that the plane is very full and that she would like everybody to cooperate so that we don’t delay. But like, everyone just formed an orderly queue and slowly meandered onto the plane. It was very chill and calm.

Passport control and immigration ingress in Vancouver was really smooth. So before you come in as a Polish person, you need to go and basically link your passport to their ETA system, that’s what they call it I think. You pay $7 and then with your biometric passport and that ETA, you can come and go into Canada for as long as the passport is valid or for the next five years, whichever lapses first. Then you have to fill out the immigration form, which can also be done electronically. Lufthansa, when you check in for the flight, will tell you that you can go and install an app called Arrive Can, which I did.

I filled it out a couple of days before my arrival and it actually worked. So when I arrived at the airport in Vancouver after landing, before you go and see an immigration officer or whatever they’re called, at the little window, you go to a little computer stand and scan your passport. And if you’ve done the electronic immigration form, it’ll just make you accept what you’ve pre-filled and print your little piece of paper and that’s it, saving your time. I was actually surprised that it works because a lot of the time these government app things will just be trash. They’ll tell you, yeah, you have to do this and that. And then in the end, it doesn’t even matter. But no, this one worked.

Following that, I was actually grilled. Had a big quiz of questions to answer to the immigration dude. It was a younger dude. So I don’t think there was anything malicious or serious about this whole questioning. He was just checking me out. He basically asked me, like, nine or ten questions about where I’m from, why I’m flying, where I’m flying to, how long I’m staying. But the fun part is, like, a bunch of the questions he repeated seemingly to catch me out on being inconsistent, which actually, you know, it can happen after you’ve been on your legs and on the plane for, like, 12 or 14 hours at that point, being quite tired. But, yeah, no. It was all good.

Went through, found my gate, and, yeah, the rest was just chill. The last hop was was super easy and very chill on the little propeller plane was again an hour and a little bit later I landed. The Cranbrook airport is just fantastic, it’s basically one big room, you leave the the landing strip you go into a building and basically that’s it, that’s everybody standing there waiting for people, the luggage reclaimed tapes are there, and then you just walk out the other end and you go straight onto the car park. It reminds me of some of the super small airports in in the UK um or maybe like, smaller ones in Poland. Although even in Poland they usually have more than one room. I think midlands airport was a little bit like this although they had two rooms. They had the pre-passport-check and the luggage reclaim and then the room outside where people were waiting… But yeah it was very very cute. Brent found me no problem, obviously, didn’t have to search much considering the the location of my arrival, and then we went to town to do some shopping because I told brent to not bother buying too much without me, as I might as well go with him and check out what there is. This was actually kind of fun, he showed me his co-working space where he actually does his normal day job activities and it was pretty nice. We picked up a parcel and then went to a supermarket, which was kind of funny for me. It’s kind of like UK supermarkets. We tried to find some soft drinks… I was looking for something sweet but non-caffeinated just to help me cruise along for the rest of the day and address the jet lag a little bit so decided i want to find sprite. Yeah, it is turned out that in that supermarket sprite comes either in these massive two liter bottles or in six packs which you cannot unwrap. There’s no 1L, 1.5L, 0.7L or even a single 0.5L bottle at the drinks aisle. We did find a single small bottle in the fridge by the by the checkouts though!

So this is like peak North America right there if I’ve ever seen it, obviously you can forget about any reusable glass containers or anything like this, and all the small fruit is packed in plastic as well. They actually had a lot of organic fruit, which was surprising, however all of them were packed in massive bags, so if you want an organic orange: here’s 10 organic oranges, you want an organic apple: here’s a pack of 15 and so on. This feels really counterintuitive after coming from Germany where I can basically buy organic things as single items to my to my heart’s content and just take exactly what I need and not more. Again, this does also happen in the UK, but at least from what I remember when shopping there you generally buy bigger quantities to get a better price. So yeah, it’s throw me off a little bit.

So after doing shopping and getting some ingredients for a barbecue that Brent’s brother and sister-in-law were apparently preparing for us along with some other friends that were around, we headed back to indulge.

The first thing that I noticed arriving at the cabin and something that completely I did not expect and also threw me is that this place is full of hummingbirds. I’ve never seen hummingbirds, they’re so cool, and Aaron, Brent’s brother, puts out hummingbird feeders which are like little plastic containers with sugar-water, so the hummingbirds can go and sip a little bit at a time. They just hang them around the property and you can watch them just hover and buzz around. They’re super cute and tiny. Super nice and I completely did not expect this.

So then we greeted Brent’s family and the friends that were there, and all of their pets, of which there are many, there’s a dog and three cats. Well, I greeted the dog, the cats can go and do whatever they want, they’re not really interested. We sat down and went cooking and doing things around. Everyone was super open and friendly and life was just nice and chill and easy. Which is definitely something that’s much appreciated after being on your feet at that point for like 18 or 20 hours or something… In the end, we cooked some burgers, we ate some burgers. Had some chats, apparently I introduced halloumi to them, like they’ve never had halloumi cheese. Which is kind of crazy to think, as it’s super popular in Berlin and even in Poland. But yeah, I threw some halloumi on the grill and everybody liked it. So, plus one for me, I guess. Ended up talking to everybody. It was until about, well, 11 or midnight local time, at which point I was on my feet already for 26 hours. And then crashed in my own little cabin here, which is also super sweet. Just a single room with a mini kitchen annex and its own bathroom. And it’s just fabulous.

And that was basically day one.

Day 2

▶️ Day 2 recording

Great, day number two complete. It was bit of a lazier day, it’s probably okay to help with jetlag. Still following Garmin jetlag advisor and its suggestions seem to be working fine. Had an easy morning while Brent was finishing up some work, we chatted on and off a bit, checked out the bikes then had lunch together. After lunch I waited a bit then went for a run. I felt pretty good but when I started running, after about five minutes, my heart rate just spiked and wouldn’t come down unless I walked. Garmin scheduled me 55 minutes of running, but in the end I did just 30 because it was incredibly hot and hard to breathe due to local forest fires. Smoke in the air probably contributed to my heart rate spikes so it’s probably not only jetlag. But I mean, I did the run and that’s what matters. The area isn’t nice to run in. The views are nice, running through a forest along the river, but the floor surface was really harsh. It was either sandy or or full of pebbles.

The edge of the river and the riverbed itself, where it is dried out and you can cross sections of it, it’s just pebbles big and small, pretty rough to run on. Running through the forest itself there are actually no paths here. It is a pretty wild area, not really tamed in any way so you’re just finding your way through some bushed and forested over areas without any paths, trials or roads or anything.

So yeah, actually the reason I turned around where I did was not because I felt tired or i felt like i don’t want to keep running, but I just couldn’t really find an agreeable path to keep going forward in the direction I started. Maybe next time I’ll go a different direction, ask brent and we will devise a better route that will let me go a little bit longer. I know he did some 10k stuff and he wasn’t going in small loops so there’s definitely a way to go somewhere further I’m just not familiar enough with the area.

After I came back we finally got going on the bikes. The KTM that I got lent by Vance is basically brand new. I rode it up and down the hill here just to see what it feels like and it’s great. It’s super light, super easy to move, the handlebars are very straight and wide, so very easy to control the turns. I didn’t go far and it was a gravel road so I didn’t go fast either, just checking that it runs and starts and stops. The chain needs cleaning, it looks like it hasn’t seen any action in a while, there are spots of rust on it. We’re gonna go and get some proper chain cleaner today in town to clean both bikes chains and we already have chain lube to then lube them afterwards. So that’s that.

Brent’s bike… That’s a whole other story… So! Brent’s idea was that we have to replace the one broken indicator light. The light itself is fine, it’s just the the rubber gasket, like the rubber housing on which it is attached to the rest of the bodywork is broken, cracked, so in a pinch we could just tape it together really. But we do have a second identical bike that is a non-starter and that should have the light we need that we can just transplant. I tried to remove the the broken one initially, but it was a non-trivial/non-obvious and I gave up until we take a closer look together because I think instead of just removing the light itself which is what I tried you have to follow the the power lead of it into whatever junction box it’s plugged in somewhere behind the headlight behind the plastics, which is hard to reach. So we’ll do that later. The other thing that Brent said we should do is remove and clean the carburetors so we started disassembling the bike to get to the carburetors. We need to remove the tank and the tank had to be drained anyway. So yeah, a few minutes later we had the tank off, it was pretty easy.

Then we realized you can’t easily remove the carburetors without removing some other bits and bobs, so we started disassembly. We opened the air box amd we found that something was trying to nest in there and ruined the air filter. We hoovered up the remains of the air filter, which was just like bits of fluff, from the entry cavity of the air box. Then removed the ruined filter and added a new filter to our shopping list. Then we went to drain the tank. So the tank has basically one or two year old fuel in there at this point so it definitely had to be drained. Surprise surprise, that also didn’t quite work. The fuel tap with a little handle that lets you, you know, open the tank and put it into reserve, wasn’t draining. We ended up just removing the tap altogether and draining the tank, which was easy enough.

And then Brent decided we should probably, while we’re at, it take the little tap apart and see see what’s inside. See why the fuel is not flowing basically. Brent took it completely apart, cleaned out. All the little components, when they were laid out separately, had flow on their own. So we put it back together and it doesn’t flow again. So I’m not really sure what it is, Brent’s not really sure what it is. So our best bet is going to be, once again just, steal the tap from the donor bike and hope that will drain. At least we have parts.

I think that’s our plan and we should eventually get there. Next stop tomorrow, which is now today as I am recording, we should receive a new front tire. The front tire needs to be replaced and the back tire has to be swapped from the old bike to this bike so we’re gonna have to drive into town with all the bits and pieces, or maybe even the whole bike on a trailer and drop it off at the garage. Have them swap the tires, balance the wheels and then we’ll also ask them about the fork seals because the fork seals on this bike also look pretty tragic. And I mean, they will work in a pinch but it’s definitely not great if they start leaking, or even worse, if they already have leaked all there was to leak in the past and there’s not much oil left.

So yeah, there’s definitely a lot to do on this bike, and more than Brent led me to believe while we chatted remotely. But I was fully expecting that, so that’s okay. So yeah, in summary, we had a pretty nice lads’ day out in the sun fixing some some bikes and progress has been made. I think we’ll get there in the end. We’ll probably be able to ride bikes together sometime next week, and if not, then I can still ride my KTM already so everything is good.

That that was pretty much the day! So yeah, see you tomorrow!

Day 3

▶️ Day 3 recording

So another day passed, this one’s probably gonna be shorter as far as the log goes, because the day seemed kind of short. Brent ended up working way too late into the night the day before so yesterday I woke up at 8am, but he woke up close to 11. We basically spent the whole day working on the bike, but we did make good progress so that’s awesome. We managed to put his bike on the lift which took a bit of finagling, but we made it. We removed the front wheel, then cleaned it. We removed the back wheel on his to-be-ridden motorbike then removed the back wheel off of the donor bike. We decided we’re not gonna be swapping the tires on the two back wheels, but rather just use the back wheel from the second motorbike. Instead of just taking them both to the shop as they were, Brent has transplanted the disc brake rotor and the sprocket from the good bike wheel to the donor bike wheel, because this means that we won’t end up with only whole wheel when the shop is done with them. This way the tires stay on for both wheels, and we just needs the good wheel to be centered.

We checked the petcock again on the tank and it’s still not working, so we might have to see if we also steal that from the other bike. We’ll figure it out today. We asked the shop about the forks and they said it’s cheaper if we bring just the forks to them, instead of the whole bike, but they should be able to to reseal it and re-oil it as necessary. They said they should be all done hopefully by Tuesday. We didn’t make it to the shop yesterday, we just missed the shop closure with the delivery of the parts, but that’s okay because they wouldn’t have done anything till now anyway.

Next thing to do is to remove the forks, then we can bring everything back to the shop. The work on the bike stretched way too long yesterday but we kind of wanted to hit the goal and, you know, finish what we set out to finish on that day, and then actually hopefully make it to the shop. We didn’t make it to the shop but that’s okay because at least we completed the planned technical work. Bad thing is, I had breakfast at 8am and then we went to town at like, 5:30pm. We were in town just as the shop was closing and we only got to get dinner at about 8pm, so I didn’t eat for 12 hours and it really fucked me up. I was really hungry for most of the day but I didn’t want to eat any snacks or anything small because I always feel like it’s screws with me. Snacks just feel like they add calories to the pot but don’t satiate me, so I’d rather wait and have a real meal later. And the meal was really fantastic. We had really awesome chips with cheesy sauce and maple bacon, it was really really good. As a main I had a fried chicken burger sandwich, some more chips and a salad which was also very nice, although the cheesy chips were definitely the the highlight.

In the evening when we came home we watched some budget bike battle to hype ourselves up for the next day. Yeah short and sweet this time.

Day 4

▶️ Day 4 recording

So after a short day we had a long day…

Saturday, even though I woke up late… I we went to bed a little bit on the late side on Friday after showering and so on, I think it was after one when I fell asleep and then I woke up at seven and was fully awake, which sucked but I managed to fall asleep. Took me like, almost an hour to fall asleep, and I slept again till like, 10:40 or something, and at the time I woke up Brent was already going to town to deal with stuff. He got the tires in the car, his brother started working on things in the meantime, I had breakfast, got dressed etc etc… Left to work on cleaning some stuff out and about and I said hey what is there to do should we do something… He gave me the keys to the ATV so that I can get down to the bottom garage and back up. I said let’s check out the other bikes air filter and see if we can steal it for the bike we’re rebuilding because the second bike, the one that’s not rideable right now, is not at the main garage but just down the hill, on the edge of the property.

It’s a bit of a hike to go up and down so… I got to ride the quad for the first time and it was pretty fun. I ended up going up and down three or four times because I needed to take the seat off, find the keys, then to remove the air filter you actually have to take the tank off, to take the tank off you have to take some of the fairings off, so yeah it’s a bit of a doozy.

I made it, managed to take the bike apart, just to find that the the air box is full of trash. Some sort of small animal made itself a house and the filter itself was not damaged but it was extremely greasy and not usable. So I went back up, brought all the tools etc, and I started making the plans with Aaron for what’s gonna happen during the day. I said that I’d like to make pizza for everybody because they have this nice gas barbecue which actually goes up to over 400 degrees and they have a pizza stone to put inside the grill. So I thought I’ll use my newly acquired superpower of fancy pizza dough making and we’ll make some pizza. So we called up Brent, gave him extra bits and bobs to buy, told him to buy a cheap laser thermometer so we can measure the stone temperature, and then get pizza ingredients and toppings which was great. He did it, we got the dough going by the time he was back. Actually we ended up getting flour from a shop just around the corner that was Brent-compatible, so we had two batches, one with just normal white flour and one with some sort of whole grain, ancient wheat flour, though still smooth enough for pizza, so that Brent can have it. And the dough turns out great, basically the rest of the afternoon we kept working on the bike together, the three of us, and managed to start it up. I also say Aaron and Brent were doing most of the work needed to start it up, they changed oil and cleaned out the carbs, did a bunch of things… We put the spare wheel back on just so that we can get access to the oil release nut at the bottom of the engine. Drained it, refilled it, added coolant, topped up brake fluid, and then I took the fairing off on the front and went again back down to the second bike to see if I can borrow a light indicator because one of the front light indicators on the bike we were fixing was kind of… Well, it was ripped. The the rubber housing that it lives in was ripped so it was soggy and it’s just not usable. I managed to remove one of the rear indicators, as for reason the bottom motorcycle doesn’t have front indicators at all, they’re somewhere but not on the bike. The the front and back indicators looked identical but when I finally managed to free the one from the bike downstairs, which wasn’t that easy because the the locking bolt was quite rusty etc, and a little hard to get to. But hey, got it out, brought it back up, and it turns out that it is ever so slightly different. Like, the back indicators and the front indicators, the housing for the actual light is the same but the part that attaches to the bike is slightly different… So yeah, wasted a bit of time, but hey, whatever…

Still, good fun riding the ATV up and down!

While doing all this, every half an hour I was flipping pizza dough to great success! In the end I managed to tape the old, kinda dangly, broken rubber indicator and I’m pretty sure it’s gonna survive our trip. Then I fitted them into the the fairing, no problem, nice and stiff in there. And that’s basically where we left it off so… Today I have to basically put the front fairing back on, connect all the electric connectors back in and then I have to take the back fairing off to… To kind of stiffen up the the back indicator lights, because they were loosened for transport and I need to replace their fitting plates with screws so that they stay in place. So yeah, that’s gonna be my job for today… And I think most of the stuff we still need doing is basically non-urgent because we’re waiting for the wheels, which will be ready either tomorrow or Tuesday, latest Tuesday…

Yeah, so to finish off the day we baked all the pizzas. I formed the dough into nice pizza shapes, Brent ended up putting toppings on the pizza and we actually saved half the dough and put in the fridge for for later because it was actually quite a lot. When I last made it, I made one batch, and we had four small pizzas, it was so great it disappeared, so this time I though we should make eight! But actually even with just 4 pizzas this time there was still some left because it was just so much… Still, everybody loved the pizza! It makes me happy because it’s so nice that Aaron and Kaylin let me stay here in their cabin, this whole area is so great, they let me use their equipment, have the food, so I thought it would be very nice to at least treat them to a little bit of nice food. So yeah, we’re probably gonna cook something more for them over the next few days, just to leave them with some nice memories after I’m gone.

Maybe they’ll invite me again.

Day 5

▶️ Day 5 recording

Another day another entry. Just wish I could remember what day number this is, because they’re all starting to blend together…

So last night was Sunday, that was mostly a day focus on the podcast. So I ended up agreeing to come on the podcast because I thought, why not, I probably won’t get a chance to do it again in a long time, so might as well. It also helps that the topics were something I could contribute on, so we were talking about GrapheneOS and Garmin watches. I actually just listened to the finished product because I felt like I was just rumbling, I was just going on about shit a bit too long, as I have the tendency to do… But it actually turned out pretty good! Drew, the editor, did a great job and I sound pretty cohesive and actually quite nice! But that basically took up a whole morning and half the afternoon… I got up, Brand messaged me say “yo, come around”, went over to his hut and we chatted a bit, started preparing for the podcast and had some fried eggs just because the podcast is like… Four hours? Four hours of sitting in the podcast studio, and that’s basically also at about the time that I get hungry… So, thought, let’s get fed, let’s get some drinks prepared… We had to move Brent’s furniture in the studio to make space for a second speaker, in a way that we don’t interfere with each other. Had a little problem with the sound, with technical stuff. We first tried to hook up two audio interfaces to one computer and do both audio streams from our side over to the JB studio from that, but there were some problems so we ended up just setting up the framework laptop and using that for me and then Brent was using his own laptop, and that works decently. Thankfully I had my own mute button so I could just mute myself most of the time and then unmute whenever I want to, and that worked decently. That’s pretty much it for the podcast stuff.

I wanted to that, this was pretty fun, pretty different and pretty goddamn swampy! The recording room was basically a sauna, I was completely wet, like, every inch of my body. So we decided to take a shower and and start making progress on things that still need to be made progress on for the bike ride to happen. But of course, we got derailed and instead of working on the bike or doing anything else, all four of us, Brent and his family, we just went to the to the river for a swim, which was actually quite nice and kind of crazy. Going into the river it felt extremely cold, so much so that I had problems breathing with my face under, which, I don’t know, I normally associate with water like below like 14 or 12 degrees… I thought it was like 10 or something… That’s kind of the impression I had, but I got warmed up, I managed to swim a couple of hundred meters, it was actually nice in the end I think you just needed to get used to it. I was so super surprised, the water was apparently 18 degrees! That’s what Garmin said afterwards, and I was like what the f… Crazy… So yeah I think it’s partly because I’ve stopped doing cold water swims. And I’m not used to it anymore. And partly because it’s a river, right? So the water is flowing, so you there’s an ever onward flow of new fresh 18c water, so it cools you down more maybe… Yeah it was kind of interesting because I’ve never swam in a river that actually has any visible flow, so swimming in Berlin for example. In Spree it’s basically just a super lazy river, you can’t even see it move, whereas here… We went to an eddy, this one corner behind the bend of the river, where there was no flow, and so you could go out into the river, have the river carry you a little bit, just on the edge of the river current and then you could just hop back into the eddy and then come back up. It was quite fun, I even made Brent actually do a little lap there and back so, great success!

Addendum: also interestingly last night was the first night I actually had proper sleep. I fell asleep easily, I was actually really tired at 11, I kind of fell asleep in Brent’s chair and then moved over to my cabin as we were watching some budget bike battle Italia. Yeah, first time I went to bed and I slept through the night all the way till 8:30 Thankfully I closed all the curtains this time, didn’t forget anything, which helps. But yeah, even my Garmin watch says that I’m finally up to speed now, which is kind of crazy that you really feel it like it makes sense, the HRV has dropped, the sleep quality metric has gone up drastically and… And I had uninterrupted sleep so, bravo Garmin as well… I guess and bravo me for returning to a normal human state.

Day 6

▶️ Day 6 recording

So yesterday, Monday, the first day we didn’t actually really do anything, any maintenance on the motorbikes. We got up, made a list of things we still need to buy and then realized time is running short if we actually want radios. And then after some short discussion, decided to pull the trigger on a couple of Sena radios that Amazon would actually still deliver to us in time. When we checked, next day delivery was available for the next 17 minutes as we were buying it in the morning. So, well done for checking it on time and actually getting it. Yeah, they should be here tonight.

So that’s the date. I’m recording the day after. So that’s going to be great! I think Brent is pretty excited about it. We’re actually probably going to leave the radios here for now after this trip so that Brent and Aaron can use them when they ride bikes, which he hopes they will, together. And then Brent can bring the radios back with him to Berlin in September when he comes around and then we can ride together again and also chat. So that’s really exciting. This is it.

And then the rest of the shopping that we needed to do was mostly in town. So we went to town. We also took our running gear because there are some nice trails just outside of town. We went for a nice 6K run. And the trails were actually really nice. Brent made me run through some fucking bushes to take a shortcut because he wants to get on the trail NOW. And now my shoes are just full of fucking sticky, spiky… Spiky shit, whatever this plant is. I don’t know. But my leg hair was just full of it. I was just like completely covered in this shit. But the rest of the trail was very nice. I took some photos and even a couple of videos. So there will be a backdrop for this later. And then we went and showered at this co-working place, which was great. People there are really nice. The shower knob for some reason was removable in my shower. And also the shower only has one knob. And it only goes left and right. So the pressure of the shower was also linked with the temperature somehow. So it goes from really cold and slow to cold and fast flowing. To then eventually gradually warmer and then slower again. I don’t know. But it was fine. It was a good shower. And then we went to have some food. He took me to this really old fire station kind of building that was turned into a restaurant. I think it was called the Fire Hall. And it was very nice. And the food was nice. And the people were nice and smiling. And everything was pretty tasty. I decided to go and try something that’s apparently Canadian or like local-ish. So I got a poutine with pulled pork. And it was really good. It was also really big, like everything here. So yeah, I decided not to have a starter or any sides. It’s just the main is just too filling. And they had alcohol-free beer. Which is actually great. That it’s just universally kind of available in most places now. Because I don’t have to look at the menu. I can just ask for alcohol-free beer. And I generally get something decent. Because people do surprise me. And they say, oh, would you like a drink? And I’m like, fuck, I haven’t looked. But then I also had the lemonade. Homemade lemonade. And it was super nice. But the thing was, there was quite a lot of sugar in there probably. Still.

Then we went shopping. And Brent took me on a crazy ride around town. So at this point, I came to town on my motorbike to give it a bit of a shakedown. Fuel it up. Test drive it. Put air in the tires. It needed some extra pressure. Yeah. And it was pretty crazy. So he was like, oh, let’s go to this shop. Obviously, we have no radio. We have no really good way to communicate. So I’m just following him. And he goes to one place. He’s turning right here, turning left here. And then all of a sudden, he’s about to turn left. And then he just changes indications and turns right instead. And I’m like, fuck. I feel like a dick because I’ve been indicating left for a while. And then I’m just going right. And this happened multiple times until we stopped. And I was like, what the fuck happened, Brent? I kind of thought I already knew. And he confirmed it. It’s basically almost all the places we’ve been to. Almost all the places we wanted to go to happened to be closed on Monday. So he did drive past, check it out. And then like, actually, it’s closed. So let’s keep going to the next one. Still, we managed to get a couple of essentials, which we really needed. We got a sunscreen. And we got some cream for me. I seem to have sore spots under my armpits. I think it’s just from sweating. Because it’s been really, really hot the whole time. And especially when I’m lying in bed, I feel like I’m just lying in puddles of sweat under my armpits. And I think that’s just starting to become itchy and sometimes painful. So yeah, I got a nice cream for that. We got the sunscreen. He got food for his cats. And then we got some apples and skyr. Skyr for me for breakfast/supper. And then apples so that I can make racuchy at some point.

Yeah, so things that we still need to get is a different helmet for me. There’s like a used store here somewhere or some sort of like equipment gear exchange place. And we can get quality secondhand things. So yeah, we’re going to try and see if we can find a helmet. Apparently, helmets are differently shaped here a little bit from European ones. So all of them, even if they’re big enough, they’re usually too short for me. Like my chin just sticks out and the bottom of my head in the back just sticks out. The only one that kind of fits or fits comfortably, let’s say, is Brent’s helmet. But then he wouldn’t have one if I use his. So yeah, we need to get a spare. And also I tried on Brent’s hiking boots. And they’re both decent and decently comfortable. But they’re both pretty high and I actually don’t have long enough socks. And I think they will just be rubbing and creating sore spots on the backs of my feet. Well, not feet, the backs of my shins. So I need to get some like nice long hiking socks as well for this. So yeah, so that’s to be done today. And then we obviously need to finish building Brent’s bike, which it should be surprise free. But you never know. To be continued.

Oh, no, wait, shit. And when we came back from shopping, we did planning. Okay, so that’s another one. That’s going to be a long recording. So we got maps. We can like type and draw on shit. We checked forest fires. There are actually a lot of forest fires. It was really dry recently. And then there were dry storms. And a bunch of thunder strikes actually caused pretty big fires. And some on the routes that we are taking, they will hopefully clear out by the time we’re going. So we should be able to go the route we want. But we designed it in a way that we can either go around the mountain range clockwise or counterclockwise. And since we’re going for three nights, the middle night is always the same. And that’s where we booked the hotel. So we should be able to get to the hotel one way or another. And not like waste that booking. And we can basically wing it whichever way we want on the way there. Hopefully it works the way we planned because I think the route is amazing. So yeah, we’re going to go first from here. We’re going to go through Windermere and Invermere up to Banff. Where we’ll probably have lunch of some sort. And then from Banff, we’re going to go to Lake Louise. Take a peek. And then from Lake Louise, we will go westwards. Hopefully if we can find the campsite to Yoho Park, which is just after Golden. (PS. Actually it’s just before Golden). So just west of Golden. We checked the campsite maps for Yoho Park and there were still three spots. But for some reason, the website does not let you book them later than two days in advance. And we just missed the deadline for booking one of those campsite spots. So Brent is going to call this morning and confirm that we can either do that or that we can do some sort of backcountry, i.e. more wild camping. And once we confirm the spot, we will know which kind of direction exactly we’re staying overnight there.

On day two, we will head from to Revelstoke, or as they apparently call it, Revy. And that’s going to be a really crazy, nice mountain pass through like some towns with glaciers on the peaks around them and things like this. Apparently lots of things to see. So that is by kilometer, I think the shortest hop from one day to another. But it might take us, quite a long time to just, you know, take it all in, stop around if we want and take a look and stuff like that. Then in Revelstoke, I decided to book a hotel that’s more in town. That we basically had two options confirmed by Aaron, Brent’s brother. One is more in town and one was just outside of town in a little forest. I decided to book the town location just because the town itself is apparently supposed to be nice. And I wanted to be more in a walking/short hop distance to things. I didn’t want to have to drive into and out of town after driving for fucking hours and hours before that. And then hopefully we can actually, you know, go and interact with some people and get some culture, culture, like local, local culture and just have a bit of a mountain town experience rather than only wilderness. And we’ll see. Then from from there, there’s a bunch of things we can do. I think there is some sort of a roller coaster. There are some hiking trails as there are everywhere. And there’s also a potential for paragliding in Revelstoke, which we will explore. Hasn’t been really settled or fully explored yet. We still have to do it, but I’m just happy to go to the hotel.

Then from Revelstoke, on day three, we will go to Nelson for quite a long distance. That’s going to be quite a long day on the bike, but it’s also supposed to be very pretty and windy and really motorbike, motorbike-awesome road. And it will include the ferry as well. So we need to look at the ferry. So south of Revelstoke to get across a very wide river, lake-ish kind of, you know, river basin thing. The ferry is every hour, apparently. Then we go back eastwards and then south. The route kind of overshoots Nelson. So then we curve back west down to Nelson. And in Nelson, like I said earlier, we have an in-town campsite booked. Nelson is supposed to be more low key. More low key and hipster, but like super, super nice and pretty and awesome place. Again, I think kayaking or canoeing is on the cards there. Swimming, hiking, a bunch of things. And the area is supposed to be really, really nice. So yeah, we’ll go there. We’ll stay overnight on the nice little campsite that still has apparently plenty enough of space. Then from there, we have a few choices on day four. So night three is there.

Day four, we’re making our way back home and that’s going to be Saturday. And I’m flying back on Monday. So I’m giving myself a one day of buffer. Yeah. And day four, we have two choices depending on how much time we spend in the morning in Nelson, et cetera, and how badly we want to get back home. We will either first head a little bit westwards. Oh, I can’t remember the names of the places, but there’s like apparently a really cool paved road that goes through like a side of an overside of a mountain. It’s quite nice and windy. We might take and then loop around back up to Nelson and then from Nelson because that’s west of Nelson. And then we’re going to go back east. From Nelson. Toward home. And there’s a few things we have on the list on the way there as well that we could stop and explore. So, yeah, that’s basically the trip. And we did all of that. We made the list of things to still like check out and figure out.

Yeah. So today is going to be mostly bike rebuild and finishing off a couple of planning bits and bobs. And doing a test pack to make sure that we know that we fit in our luggagey kind of packing bags for the motorbikes and that we have everything. Yeah.

More tomorrow.

Day 7

▶️ Day 7 recording

(Tomasz) So let’s record another entry. Come closer. This one’s gonna be with Brent.

(Brent) Hello!

(Tomasz) So what did we do yesterday morning? We got up. I hit up your cabin. We made some calls, right? We called…

(Brent) Yeah, we called Parks here to see about the campsites. And we found out nothing. First come, first served.

(Tomasz) Basically, yeah. But we might find out more. They said we should talk to the park administration here in Lake Louise or Banff.

(Brent) And then we called the motorcycle shop.

(Tomasz) Shortly after the tires were done.

(Brent) Yeah, like by 10 a.m.

(Tomasz) We checked Amazon. The radios didn’t arrive yet at that time.

(Brent) So we had coffee.

(Tomasz) Then we went to town, basically. We didn’t do anything else, right?

(Brent) No, that’s true.

(Tomasz) Just went to town. We got the wheels. And they kind of didn’t give a shit about my preload.

(Brent) Very much not.

(Tomasz) At the Kawasaki place. They’re like, yeah, that’s very nice. Smiley, of course. And then like nobody, nobody talks to us. There’s just no reaction. Default answer. You reach this part of the phone tree, you will not get no further.

(Brent) We’ll say everything but no.

(Tomasz) Yeah. And then I think we checked… You checked Amazon and then the Senas were delivered!

(Brent) Yeah, which is great.

(Tomasz) So, Senas were delivered. We picked those up and then we were hungry. So we went to grab food and we started tearing into the Sena boxes.

(Brent) So we got food at a place called Hot Shots, which is like a cafe, but they have some great vegan food. The drinks were good.

(Tomasz) It was really good food, way better than McDonald’s, that’s our foreshadowing, yes. We’ll do today tomorrow.

(Brent) Okay and then…

(Tomasz) We went to the outdoors shop, we bought way too many things!

(Brent) Yes we did! A new bottle, some socks…

(Tomasz) I found another one of the the soft flask bottles that I have but half a liter instead of 250ml and it’s even slightly insulated and it’s great, it works super awesome in my sling. It’s just squished into the sling and it doesn’t get scratched by the keys like the metal one.

(Brent) Feels like I need one! What else did we get there?

(Tomasz) Spray! I got a massive can of bug spray. Brent had slight anal pain when he looked at the size… He was like, you’re gonna carry this yourself, and I was like, man I need this bug spray!

(Brent) I was expecting something half this size…

(Tomasz) But whatever…

(Brent) We got a map there too.

(Tomasz) We got a map which we forgot to take, very helpful… And we got my socks. Didn’t try them on but they look good.

(Brent) Water filter?

(Tomasz) We got like, a water pump with a filter because… I have life straws at home but one live straw to share between two people is kind of awkward. If you’re in a pinch you’d rather just like be able to pump from some source and carry the water with you because we don’t know where we’re going to camp and how far the water source might be. If we, you know, wild camp tonight.

(Brent) We’ll take some Lake Louise water…

(Tomasz) We could filter some Lake Louise water just for fun! Like this thousands of thousands years old water with some ancient microbes… So, but then we came home and we just had to fix your bike, finish the bike basically.

(Brent) You worked hard.

(Tomasz) You mostly did the bike, I started helping you, I cleaned my bike, helped you with some stuff, I cleaned your fairings, did the the rest of the indicator lights and then…

(Brent) The droopy boy?

(Tomasz) No, this one I did the day before…

(Brent) Oh okay…

(Tomasz) I just did the back ones.

(Brent) Right, right…

(Tomasz) And then I decided it’s time to make the rest of the pizza because the dough is there and it’s got to be made and we need food so Brent carried on, I made pizza. The first one was a flop, literally a flop, because I prepared it too early and the grill took way too long to heat up, and then the pizza stuck to the board. But it was eatable… At least the dog liked it!

(Brent) Yep, he was happy!

(Tomasz) So what’s the name of the dog again?

(Brent) Fiddick.

(Tomasz) Fiddick surprised us with very high appetite…

(Brent) He’s only done that like three times…

(Tomasz) So clearly my pizza must be good for him to do that.

(Brent) The whole wheat one only!

(Tomasz) Yeah he ate both, one and a half whole wheat ones, and did not touch mine (plain wheat). I was kind of… To be honest getting tired and disheartened by the end of the day.

(Brent) Disheartened by what?

(Tomasz) By the fact that we still had shit to do in the morning. I wanted to be packed and ready, but we weren’t… The only thing we had left by the time we went to bed was basically chain cleaning and tidying up the garage basically, and packing.

(Brent) I would say from my perspective yesterday was very productive, it was the first day I got my bike on the road.

(Tomasz) You rode it right? You rode it a couple of times… And you had a bit of a bbb moment when your oil pressure was getting high?

(Brent) We fixed it, we fixed it… We got a check on that actually.

(Tomasz) Yeah, we’ll check it when we go back to the bike.

(Brent) Yeah, so that felt actually… Like it was actually going to happen! This trip yesterday, the inflection point.

(Tomasz) I have to say, when you started riding, you lost those bolts, and the fuel or whatever was getting into oil, I was like… Okay, it’s over…

(Brent) This is the end…

(Tomasz) But I don’t know,

(Brent) A little excitement?

(Tomasz) It keeps going, it keeps rolling. For now! Okay well, I think that’s it. We’ll do today tomorrow. See you later!

Day 8

▶️ Day 8 recording

(Brent) Oh my god, it’s recording?

(Tomasz) Yes, it’s recording. Day one of the trip!

(Brent) Good morning.

(Tomasz) What happened?

(Brent) Ok, bye!

(Tomasz) We were supposed to wake up at 7 so that we can leave at 9. I woke up at 7.

(Brent) I woke up at 7.30…

(Tomasz) Yeah, but we did not leave at 9. We just had… The packing took ages in the morning. I have to say, I was a little bit pissed, because if I knew everything is going to take so long I would have just woken up at 9. I could have slept 8 hours, because most of the stuff kind of you had to do…

(Brent) I didn’t realize how much was all on me…

(Tomasz) Yeah, I mean, it was not something we could have predicted, but this is the main reason why I was annoyed in the morning. I could have slept, you know… I didn’t have to wake up. We had breakfast, had coffee. We put all of our stuff on the cart. Collected everything, found sleeping bags, tents mattresses… All the water containers, everything filled up. Went to the bike shed, bike garage, got the bikes out, put chain lube on both of them… You put your little chain fairing on. And then I basically loaded my bike up. I don’t know how we would have done it without me having panniers.

(Brent) Yeah, I know…

(Tomasz) There was no way we would carry… If that bike couldn’t carry panniers we would have been like…

(Brent) Suffering.

(Tomasz) Yeah, I don’t know, I mean, what would we have to drop?

(Brent) Less water for sure.

(Tomasz) Yeah. Thankfully we didn’t actually, I think we would have been ok for water, we would have had to just more proactively fill up in like, hotels or restaurants or whatever. But yeah, we wouldn’t have the bladder.

(Brent) Less clothing.

(Tomasz) I would probably have dropped shoes. Maybe even hiking boots.

(Brent) Same.

(Tomasz) And then we would be maybe ok. I would probably have enjoyed it more without the panniers, cause once we started riding… So we left at about 12 in the end, 5 past 12, 10 past 12, something like this… And the bike feels more or less fine, pretty good in, like, normal conditions up to like, 120, 130 km an hour. I did a little bit of a speed test just to get the feel for it, and after 150k it starts wobbling.

(Brent) That’s awful.

(Tomasz) Like, severely wobbling, very uncomfortable. Yeah, so I maxed out at 160 and I was like, no, I’m not doing anymore. Whenever we were cruising and overtaking, overtaking up to like 140, 145 and then going back down feels fine but not any more than that.

(Brent) Yeah, there’s not much reason to go that fast or anything.

(Tomasz) No, not really, it was more of a shakedown just to see how the bike behaves.

(Brent) Wobble down.

(Tomasz) Sometimes when I brake it wobbles as well.

(Brent) That’s the front suspension?

(Tomasz) Yeah, even if I brake from like, I think the front suspension combined with a lot of weight on the , on the panniers and stuff.

(Brent) At least you can brake.

(Tomasz) Yeah, so, how was your experience on the bike?

(Brent) it’s been a while since I’ve been on that thing, but it’s set up the same way as my previous motorcycle, almost identical, so that felt really nostalgic if anything. And, yeah, I forgot how crouched the seating posture is? Or at least maybe I’m less crouched.

(Tomasz) You have adapted the yoga position of a crouching frog on top of that tank.

(Brent) But the bike did alright! I think the brakes are terrible. Like they feel like garbage, so I’ve not really been using them, because I’m saving them for when I actually need them. Brake brake brake brake brake!

(Tomasz) Downshift, downshift!

(Brent) But, otherwise pretty good, like on the road it took a while to warm up a little bit but I think yesterday’s track was perfect for that.

(Tomasz) So where did we go then? In the morning we went up north, so thankfully the pass we wanted to take was no longer closed due to the fire. We have ridden past the spot where we had seen, like, burnt up grass and shrub on the roadside which is interesting for me. I mean for you this is nothing new but for me this is kind of new.

(Brent) Yeah the fire came right to the road.

(Tomasz) So yeah thankfully we could make it all the way to, like, north, on like a better mountain pass road than we would have otherwise… We stopped briefly in Invermere to just take a sneaky, cheeky photo of the lake, Lake Windermere. And then we tried to have some food. Yes, I don’t think what we did could be categorized as food.

(Brent) The poutine queen would have been great…

(Tomasz) Yes, so everything was kind of closed in this one little area where we went except McDonald’s and I was like, ah fuck it. I was actually really hungry then because I had breakfast quarter past seven, right? And it was like, 1.30, 2 or something… Normally when I’m at home I start suffering at 12 if I don’t eat at 12… That’s the problem, it was like after 2. I was like, I have to have something, so we had McDonald’s.

(Brent) McDonald’s in Invermere. It was terrible.

(Tomasz) Like, the pictures, the goddamn pictures on the machine, they looked like food. You see the burger and it’s like, bacon strips inside, nice chunky piece of meat and some other bits and pieces. And then it comes and it’s just like, it’s fucking flat like someone sat on it, there’s like barely any fucking burger inside, and two tiny measly like, thin slices of bacon. I’m like what the fuck is this. The chips, the salty chips, I think they were, they were just chippy salt. It was mostly salt. A little bit of chips. The drink was okay but that’s just because you can’t really screw up sugar.

(Brent) The best thing I think was the egg sandwich.

(Tomasz) Tt was okay.

(Brent) I took the egg and you had the sandwich.

(Tomasz) I had better. I had better morning egg sandwich in McDonald’s in UK. And they say the UK McDonald’s is apparently considerably worse than on the continent in EU, and that was still better than this. It’s like, you come to North America you expect the OG experience, and you just get fucking shafted. Like what is this…

(Brent) This is the OG experience.

(Tomasz) But this sucks! I’m like, starting to think that, actually, consumers… Not just consumer protection but consumers themselves, what they are willing to put up with and buy has really dictated that outfits like McDonald’s and Burger King and all the others, they actually have to put more effort in…

(Brent) There is like McDonald’s tourism, because every McDonald’s is a little regional and they get to do their own thing.

(Tomasz) Yeah, but I have a feeling that on the whole, if you go to a fast food restaurant in Germany you’re gonna get better food than here.

(Brent) Yeah definitely.

(Tomasz) Even though I’ve only been to McDonald’s, but I don’t really want to try the others after this. And oh my god have you seen that like, birthday party thing?

(Brent) Yeah, they couldn’t even sing!

(Tomasz) This is the saddest McDonald’s birthday that child had, it’s just like dad and two kids and these three guys came to sing happy birthday to this poor kid. When you’re fucking traumatized at least for the next month, like holy… I don’t know.

(Brent) Enough about McDonald’s.

(Tomasz) So what happened after?

(Brent) Well then, then we drove like, 10 minutes into, uh, Fairmont hot springs.

(Tomasz) Yeah.

(Brent) And then we went around the roundabout and started on the mountain pass.

(Tomasz) Ah yes.

(Brent) Do you remember that initial like, kilometer? That was the impressive.

(Tomasz) Because all like, you’re driving like through town, and all of the sudden there’s just like… Cliffs around you. Like just rocks, like rock walls on both sides. It was very pretty.

(Brent) It’s almost like a natural arch that you go through, almost.

(Tomasz) Yeah it’s very exciting. Very different, like… The only thing I can compare it to that I’ve been in is probably. um… Driving around the cliffs in Llandudno in Wales. There’s this, like, peninsula, and they carve the road basically into the cliff side and you go around it and you have like cliffs, sheer cliffs and even overhangs and you go like around this peninsula and there’s just like a sheer drop and sea on the outside. So this is kind of a similar feeling.

(Brent) And then there was a mountain goat.

(Tomasz) There were a bunch of mountain goats.

(Brent) Oh we saw one…

(Tomasz) We didn’t stop to milk them.

(Brent) They’re not milking goats.

(Tomasz) Well, I said, you could try… Just make sure it’s a female goat before you start milking.

(Brent) Well it depends what you’re into.

(Tomasz) Yeah, so yeah, and basically that was the funnest part of the day. I would say. I was able… I was still a little bit behind you all the time, I was basically catching you up on the straights, because I was worrying that the wobble will get worse in the corners. I also wasn’t sure, like, the front feels like soft and squidgy, and it’s sometimes… I lean in the corner and I have to lean with my body weight, to lay the bike over and then as you do you just accelerate out of the corner, to straighten out. But I just… It wobbles as well.

(Brent) Oh man that’s terrible.

(Tomasz) In the corners a bit. I think it’s mostly because of the soft front, like it doesn’t, it doesn’t exit cleanly, it exits like this…

(Brent) Oh gosh really?

(Tomasz) So I’m a little bit slower in the corners than you are.

(Brent) Well good.

(Tomasz) I just try to catch you on the straights, just not to get left behind. Thankfully as long as you stay in the power band it accelerates pretty quickly, so it’s not a problem. Yeah and then we basically decided, because we were so late, and I mostly didn’t want to stress about getting here on time and like, finding the campsite, because we did not in the end… I mean we didn’t have a reservation, so I was like, I don’t want to risk it. I don’t want to be like looking in the middle of the night… Or just, not in the middle of the night, but like, by dark looking for a campsite. Like, let’s skip Banff, straight to Lake Louise. So, we stopped there, it was nice. It’s a nice lake you have there! We took some photos.

(Brent) It’s like, the spot you have to go.

(Tomasz) Yeah, we dipped our toes, it was very smoky.

(Brent) Yes. I’ve never seen it like that.

(Tomasz) It was extremely smoky. Like when we are at your cabin, and you see it from a distance, it’s like “yeah yeah okay”… You can even sometimes feel the smoke in the air. But there it was like proper smoky. It was like smoked bacon smoky.

(Brent) The sun was just like, a fireball.

(Tomasz) Yeah, it was like, blazing red. Yeah, and then we missed the entrance to the visitor center, so we went straight to park.

(Brent) Oh no…

(Tomasz) It was like, “yeah no problem, it’s open until like 8 or something, whatever”, I was like “okay fine… probably not…”. But hey, I don’t think we’re gonna… We said we’re gonna stay there for like, half an hour. It was like 4 something, when we entered. We went up to the motorbike car park, there was a nice older lady, greeted us there, it was pretty neat. We went to dip our toes in the cold water, I considered jumping in a kayak, but that shit ain’t cheap.

(Brent) Proper tourism prices.

(Tomasz) $150 was it?

(Brent) $150 for half an hour, $160 for an hour.

(Tomasz) Yeah, I mean, I don’t know if I… If I just had more cash flow… If I just had like… Piles of cash on my account for no reason, I’d just do it, just because. I wanna make sure that this trip can continue, so I decided to skip the kayaks. We have to pay for other things too. Uh, so, we went to have food in a restaurant, the restaurant was like the nice fancy restaurant with the, was it called, patio? But the outdoor area was close to tourists because they were, they were moving guests from establishments in Jasper. Because Jasper was shot down and evacuated due to the fires, but there was a smaller like, downstairs restaurant place at this fancy hotel, that we could go and have food and the food was really nice. And yeah I ordered fish and chips. It was a lot.

(Brent) Yeah it was too much.

(Tomasz) I don’t think I ever ordered a fish and chips, and got like, two full fish fillets. I was like, wow! And it wasn’t even like, outrageous considering like, price wise. Because you do get a lot. And it is a high tourist spot, so it was a premium, but it wasn’t like, unacceptable.

(Brent) Just slightly more than McDonalds.

(Tomasz) Yeah, what was strange, was that when I was paying, and it says “do you want to pay a tip?”, and the smallest percentage that is on the auto screen, was like 18%… I was like, what! I’m not paying 18% tip! Get the fucks… So I went and manually typed in 10%. Because the lady was nice. I wasn’t gonna not pay a tip, but I’m not paying like 20 something dollars in tip. I was like “come on, what is this?”. Yeah, and then we found the visitor center, just in time to not be in time. We go in the door and the guy is like, “oh you guys know we are closing in two minutes”. Great! Thankfully, we basically missed out on nothing. They didn’t do a fucking shit. We went in and they were like “yeah there are campsites, here’s a map of the wrong region, good luck! be on your way out please! you want to pay? oh yeah you can just get out and pay on the machine”. And then the machine was like, “yeah, if you cannot pay, then you can just pay on the next day”. And we were like, why is that? Because the machine doesn’t work! Yes the machine is borked.

(Brent) How many tries did it take you to get the machine to work?

(Tomasz) 3 from me, and probably 3 or 4 from you until it actually took our money.

(Brent) And I think we did it wrong in the end.

(Tomasz) No I don’t think so…

(Brent) No?

(Tomasz) No, because we had 2 vehicles, and we paid for 2 vehicles.

(Brent) No, we paid for 2 people.

(Tomasz) Yeah, that’s fine, you don’t pay for a vehicle… Like that’s the thing, right?

(Brent) But then we got to the campsite here and it says each vehicle needs a permit…

(Tomasz) Well that’s ok, there was no option to have, 2 like, separate… I think it’s the same amount, and I don’t think anybody gives a shit. We did our best. We tried to give them money! It’s not our fault we don’t want to take it. The same about the campsite. We need to still figure out how to pay for this campsite, it’s the next day now. So, but I think there are some other people who also came here like, spontaneous, so we can also ask, if we are in a pinch or whatever. I don’t know we can maybe donate somewhere online to them later. “Sorry, we tried to pay! Couldn’t, so here’s some money!” So we left…Oh yeah, we did a bit of shopping, which was actually pretty nice. They had pretty good amount of stuff.

(Brent) For a small place, good selection.

(Tomasz) And they had a bunch of like, organic and natural things as well. And then we were off to call…

(Brent) Fuel.

(Tomasz) Oh jesus this goddamn fuel pump… Fuel pump just didn’t work, like you put money, and then you press the button, and then you like, press the thing, the handle on the fuel pump, and just, nothing happens.

(Brent) And the guy inside, you couldn’t hear him, but he’s like: “oh yeah he just has to squeeze the handle”. I’m like: “he’s squeezing the handle”.

(Tomasz) I was showing you right!

(Brent) I know! It was so obvious…

(Tomasz) And so, what did he do did he just like cancel my transaction?

(Brent) He just cancelled everything we tried again and then it worked. I don’t understand it.

(Tomasz) And it kind of worked because it like… It had no like, it was not an analog trigger, it was like, the full on or full off, so fucking splashing fuel!

(Brent) Oh God!

(Tomasz) Because you know, if you want to actually fuel the tank you can’t put the, um, the nozzle all the way in, because the tank is shallow and the moment the fuel reaches the nozzle it will stop fueling. So you have to pull it out a little bit. But because it’s just going full bore, just like fsjduf23h32ur!!!!! everywhere… Yeah, terrible. Terrible fuel station. One out of five stars, would avoid. Yeah, but then we fueled and we got here, and it was actually a pretty short hop, but it took a while to go up this mountain pass. So the pass from like, the main road/motorway up to Takakkaw Falls… You liked it a lot. I was less impressed.

(Brent) You said you didn’t like it.

(Tomasz) I would have liked it on this bike if this was empty and the suspension was firm. It would have been amazing. But the suspension was not firm, the bike was loaded and I was just struggling to keep up with you. And I couldn’t even hear you talk because of the mountains were in the way.

(Brent) You’re like 100 meters away and I can’t hear you, because of the mountains in the way.

(Tomasz) Because you’re like, you go around the bend and you’re just gone. And I’m like, I don’t fucking know.

(Brent) I almost dumped it on a switchback.

(Tomasz) I saw it and I took care to go slow, and like, took a nice wide… Well actually I didn’t go so wide, I just like feathered the clutch and did it in the first gear, so it was fine. But yeah it’s definitely, so this guy, this random motorcycle this guy we met at Jasper, said that this and up to… Yeah, and up to Lake what is it called??

(Brent) Lake Emerald?

(Tomasz) Lake Emerald? Is like the best biking roads and I can definitely agree. I just need a good bike.

(Brent) Well maybe we can take some weight off of your motorcycle and put it on…

(Tomasz) No I don’t think it’s gonna help… I think it’s part weight, part suspension and part just aerodynamics, because this bike is way lighter, so… Which means that wind, the way it goes around the panniers and stuff, has also more of an effect. So unless you remove the panniers altogether, I don’t think the handling will change much. This is my issue. Again, it doesn’t make it not fun, it just means I can’t keep up with you. I just have to go a bit slower because it’s not comfortable. But that’s fine. I mean, it’s better than not going. Just imagine going by car.

(Brent) Oh, it would be so lame!

(Tomasz) So yeah, I keep saying this, this is more of a reflection than a complaint.

(Brent) Yeah, better than no bike.

(Tomasz) Yeah, exactly. So yeah, we made it and we walked around. I was just like, I just don’t want to do anything.

(Brent) You were grumpy.

(Tomasz) I want to camp, I want my goddamn tent. I want to know we are done. I don’t want to be like searching for anything anymore. I just want to relax now.

(Brent) So where are we in the end?

(Tomasz) So we are at Takakkaw Falls. I think this is called the Takako Falls Camp.

(Brent) I think so.

(Tomasz) I don’t remember it having a different name. So we are basically in between like two waterfalls. One is like really, really big and one is medium big. Or probably was big earlier in the season and now it’s just a little bit dried out. And it turned out the camping site here that was supposedly full to the brim based on the website is like, half empty, more or less. Okay, not half empty, at least a quarter of spots here are not reserved or not taken.

(Brent) But when we looked, there was one campsite.

(Tomasz) There was one… There was either one or two. I think there was two and we just chose one. That was the one we wanted more. That was number three. But then we came here and there were a few. And I met all the… I heard other people talking that they came here also without reservation and they found spots. So something is wrong with the reservation system or something.

(Brent) Which worked in our favor at least.

(Tomasz) Well, it worked in our favor in the sense that we found a spot. But it also worked against us because we weren’t able to plan accordingly.

(Brent) True.

(Tomasz) Ahead of time and I was stressed anyway, that we would have to look for this. So… But yeah. We folded everything out. Tent is here. We had some snacks. We went for a nice walk. We found a really cool little like, bridge over this like, waterfall river bed. Really nice clean water. We’re probably going to go pump some water.

(Brent) Yeah.

(Tomasz) To refill our sacks. Even just for fun. Just to see how it works. And then we found a nice like very hard flowing river. Over some like rocky area. In general it’s very nice. And the trail is very nice, hard packed, well maintained it seems like. So definitely a spot worth recommending I’d say. So if you have some other friends who like want to explore.

(Brent) Okay.

(Tomasz) Explore the area. Don’t you think?

(Brent) Well I was, I don’t know. I wasn’t sure what you would say. Because it’s the first time you stay in this style of campsite. And I wasn’t sure how you felt about it.

(Tomasz) Well it’s just a campsite really. I mean you sleep on the ground. The thing is the camps are, the tents are spread out quite far apart. In comparison to what I’m used to.

(Brent) Like we can’t really see anybody from here.

(Tomasz) No. Like, you see people walking past on the path. And yeah. If you stand straight and around you can see tents like, somewhere. But not anywhere near you. So. You know. I mean. Campsites are just like this here right? So out of. Like the campsite set up itself aside. The location. And the way to get here. And I guess the hiking opportunities. They seem pretty decent.

(Brent) Yeah I agree.

(Tomasz) So yeah. And then we went to bed. And then we watched some BBB. And then Brent just fell asleep. With like this half crouched position. He was just watching with his eyes closed. And then I decided to also just doze off. Cause I was like I couldn’t. I didn’t feel tired until I laid down and we started watching. And then I was just like off. So good day.

(Brent) Wait! Then there was that crack of lightning. Do you remember that?

(Tomasz) That was in the morning. That was the next day.

(Brent) Oh! Amazing!

(Tomasz) See you later!

(Brent) Bye!

Day 9

▶️ Day 9 recording

(Tomasz) Yeah, there is a fire. So we’re on the ferry.

(Brent) Yeah. Which ferry, Galena ferry?

(Tomasz) So, what did we do yesterday?

(Brent) Oh, right… Wow.

(Tomasz) We woke up and it was a thunder storm that walk me up.

(Brent) Lightning strike. That was crazy.

(Tomasz) Yeah, you didn’t seem to give a shit. I was like…

(Brent) Meh, it’s normal…

(Tomasz) It was yeah, I was… I had my eyes closed and I saw the flash and I heard it almost immediately, I was like, fuck! Super close.

(Brent) It was loud too.

(Tomasz) Yeah, so we packed up without much trouble. It was wet. It was kinda raining. We were chickening out like: “eh, let’s wait a bit more till the rain goes away”, just watching budget bike battle. As you do… We packed up, took all of our gear back. It didn’t rain at the beginning of the bike ride, right?

(Brent) No, no, we did the downhill with all the switchbacks.

(Tomasz) Yeah, so down. Yeah, I tried my riding condom for the first time.

(Brent) Oh, yeah!

(Tomasz) And on the switchbacks it was fine, but when we… When we hit the road later, it was too… Yeah, it was too much like a flag whipping in the wind, just like throwing me around.

(Brent) An inflated condom!

(Tomasz) Yeah, pretty much. Yes, I had to switch into my jacket, thank God I have my normal jacket, because I’d be drenched and cold.

(Brent) Oh, yeah.

(Tomasz) Yeah, we had to wait for a train for a very long time on the crossing.

(Brent) On a trail.

(Tomasz) Yeah, so we try our trail, we try to go to have some food because I was “hangry”.

(Brent) Yes, you were.

(Tomasz) We didn’t have breakfast really. I was like: “I want a warm breakfast and I want something warm to drink”.

(Brent) It was kind of late too.

(Tomasz) It was late, as it always is when we leave.

(Brent) Today is better.

(Tomasz) Yeah, because we had a deadline of 11 for the hotel checkout. But yeah, so we waited for a train like a good 10-15 minutes.

(Brent) It was terrible.

(Tomasz) Then we waited another 5-10 minutes for entrance allowance.

(Brent) Oh, it was more than that.

(Tomasz) Yeah, we waited for a while. Getting to the restaurant in the field because there were so many people. And then we just had a generally horrible ride.

(Brent) Yeah, we did.

(Tomasz) We took a look into Golden. A wonderful little city.

(Brent) Oh, we saw nothing of interest.

(Tomasz) Yes, we will go there exactly zero times in the future.

(Tomasz) Yeah. Let’s do that. Yeah, and then we just suffered in the rain.

(Brent) This is a large highway with lots of wind and lots of rain.

(Tomasz) Yeah, we were just playing the game avoid the truck. Basically.

(Brent) Stay out of harm’s way.

(Tomasz) Yeah, we stopped for a quick coffee at one point. Oh, thank God I had these heated handlebars. It kept me sane. I was like, this is the one thing that kept me sane.

(Brent) You missed a part. We left trail on which motorway?

(Tomasz) Motorway? On which way?

(Brent) I was going to call it a trail. That’s more accurate.

(Tomasz) Yeah, Brent found a shortcut. Like, let’s not wait for the train anymore. There’s a shortcut here. Or like, you know, you can cross the rails without crossing the rail track in a way that, you know, you get stopped. And it was fabulous. It was just a great little green lane. I had lots of fun. My soft suspension came into its own for the first time.

(Brent) Finally, you’re leading at someting.

(Tomasz) Massive potholes.

(Brent) Oh, it’s just muddy too.

(Tomasz) Muddy. The mud was the worst.

(Brent) I do not like the mud.

(Tomasz) Yeah, so potholes and like when even terrain and gravel and even the big rocks were fine. But the mud, that was…

(Brent) ..like a lot, yeah.

(Tomasz) Whipping around. Crazy. Yeah. Then we finally arrived in Revvy and the sun came out again, just as we were finishing. A little bit of sunshine. The hotel was pretty nice. I wouldn’t call it like the best thing ever.

(Brent) No. It was fine.

(Tomasz) But it was comfortable and it was convenient.

(Brent) Hotel.

(Tomasz) Yeah. So we went to town for a quick, quick nibble.

(Brent) And to see what it was about.

(Tomasz) And to see what it was about. Yeah. I was hoping for more people?

(Brent) Yeah. Me too.

(Tomasz) I was hoping for more single ladies.

(Brent) All the single ladies!

(Tomasz) It was mostly either families with kids or a couple of couples. It was a lot of Germans, for some reason. A lot, yeah. Out of that like, 10 people we saw it was like four or five Germans at least. Yeah, it was really empty. I’m really like, surprised. It’s supposed to be high season. Although Revvy is not necessarily like, the destination for high season… I mean, Lake Louise was packed.

(Brent) Yeah yeah yeah…

(Tomasz) So we over-committed, ordered too much food because that’s what you always do when you’re “hangry”. Now we still have pizza in my pizza slot (i.e. laptop sleeve). We went back and we jumped in the hot tub, yeah… Surprise hot tub was good! We met some guys, basically anybody we talked to was just like, cool and chill… You just talk to them and chat about random shit. Yeah, there was a… There was a guy from, where was he from? The local? He said he studied in Crambrook?

(Brent) Well, he’s likem near Edmonton.

(Tomasz) Yeah, so Mike lives near edmonton and then he has a co-worker, James, who’s from South Sudan, he’s just working there as well. Yeah. And then we packed it in and went to bed. We… Oh yeah, I washed my clothes and they just didn’t dry. There was no draft in that room. Even with the both windows open there was no draft really.

(Brent) Even with the clothesline.

(Tomasz) Yeah, my thermal base did the best out of all the clothes I have, but that’s to be expected, that’s what it’s for. Quick day. Many events, not many that you’d want to remember.

(Brent) My bike stopped leaking fuel.

(Tomasz) Oh yes, your bike stopped leaking fuel. I need to be honest, I kind of now I’m really hyped for heated grips, but I don’t… I really don’t want to install them. I don’t want them on my bike, even though I want to be able to use them… I really wish I had a bike that just came with hidden grips.

(Brent) Just get some muffs!

(Tomasz) Yeah, I don’t think… I think i’ll skip muffs and i’ll go straight to muffins!

(Brent) Okay.

Day 10

▶️ Day 10 recording

(Tomasz) Morning.

(Brent) Hello.

(Tomasz) What happened in the morning?

(Brent) What happened yesterday?

(Tomasz) Where were we? Ah, we were in a hotel in the morning.

(Brent) Oh yeah!

(Tomasz) Forgot about that already.

(Brent) Great parking spot. Breakfast.

(Tomasz) Mediocre at best.

(Brent) Yeah.

(Tomasz) We got up and went to get hotel breakfast. I decided I just wanted, since it’s already included in the price, I might as well try. And trying I did. The less said about it, the better. The coffee was okay. It was not as bad as in Deutsche Bahn, but it was not as good as, you know, good coffee. They didn’t have like fried eggs. They had this like egg cheese puck thing that you can put in your bread or in your mouth or whatever. Anything goes really at that point. The sausages were basically just like… I had one just to try. I just wanted to see how bad they were. Because they looked bad and I knew they were going to be bad. I just wanted to see the scale of badness.

(Brent) How bad? Oh my gosh.

(Tomasz) I think even English sausages are better and that says a lot. They’re bad.

(Brent) Okay.

(Tomasz) But whatever. I mean, they didn’t taste bad. They were just not really sausages. It was just like… I don’t know. Like the texture and the consistency wasn’t there.

(Brent) I admit I didn’t even look at them.

(Tomasz) Yeah. You shouldn’t have. You didn’t miss out. I had like a little… What was it called even? This piece of bread I had.

(Brent) English muffin.

(Tomasz) Yeah, English muffin. I don’t know why it’s called an English muffin. I don’t think they exist in England. Because I saw another lady just take it and put this cheese in it. And I was like, that sounds like a decent idea. It was not. It was not a good idea. I put ketchup in there and it was still dry. And what else did I have? Oh yeah. And I had some just potatoes and salsa.

(Brent) The salsa was the best part.

(Tomasz) The salsa was decent and the potatoes were okay with salsa as well.

(Brent) That’s like the least exciting part of our whole day.

(Tomasz) Yeah. You know, the coffee would have probably been so much better if they hadn’t had it. So much better if they had real milk. Not just like these tiny creamer cups. Is it really hard to get milk here?

(Brent) No.

(Tomasz) Okay. So it’s a choice.

(Brent) Yes.

(Tomasz) Okay. Then they’ve made the wrong choice. Okay. So yeah, then we left and we packed, which was pretty, pretty easy. Our stuff hasn’t dried properly yet from the previous day, but I have my packing cubes and I just put all the wet stuff in one packing cube and it was okay. Yeah. Packed, set up bikes, gave back the cards, checked out, we went to fuel and then we were off.

(Brent) Yeah. On that big bridge in Revee, right?

(Tomasz) Yes. So we had to cross the bridge and then turn left. Turn left immediately and then go down to the ferry. So thankfully the weather was good. I was a little bit on the cold side at the beginning, but as the time went, it just got better and warmer and better and warmer.

(Brent) And better.

(Tomasz) And better.

(Brent) Yeah. We finally got off the major highway and onto these little…

(Tomasz) Yeah. Yes. So it was, it was more of a side road or… Yeah. Single lane each way, kind of through the countryside down next to a river until we got to the ferry. At the ferry, we got told to go and line up with all the bikers and it was nice because there was a group of bikers right at the front of the queue, so like I went past the one part, one like group of bikers and didn’t really see a good spot to curve into. But then I saw that there’s actually a big… There’s a wasp here? I saw there was a big, like first free spot right at the front next to the other, the other front bikers. I was like, okay, let’s go down there. That’s a good spot. We waited, I don’t know how much, how long, 10 minutes maybe for the ferry.

(Brent) Yeah, 10, 15, something like that.

(Tomasz) Yeah. So it was all good. We saw some crows.

(Brent) Yeah.

(Tomasz) Being social. Yeah. Nice views. In general, yeah. It was just pleasant, chill and everything was smooth. I actually thought we were going to have to pay for the ferry. Normally things like this you have to pay for and here it’s just like, it’s so Canadian. I want it, I want it in my life. It’s more Canadian than this.

(Brent) Free ferries.

(Tomasz) Free ferries.

(Brent) Yes, what was.. How was the ferry?

(Tomasz) Fast. Pretty fast, it was doing like 16k an hour.

(Brent) Oh, I didn’t even notice.

(Tomasz) Yeah I checked, I told you.

(Brent) I didn’t listen.

(Tomasz) You don’t care when I say stuff. It’s fine. I say things all the time, why would one thing matter more than another.

(Brent) Exactly.

(Tomasz) Um, yeah, I left my my watch recording, I was just curious, so yeah 16k an hour. Good ferry. Good stuff. We took some videos and photos, walked around and then we, uh we managed to fashion a gopro mount for the nopro.

(Brent) The nopro…

(Tomasz) And I said let’s pull over at the other end at some point so that we see if it stays on the back. Um. And you were like: “yeah there should be a space to pull over”. There was no really space to pull over at the other end and we were right at the front of the pack so everybody was kind of, quote unquote, chasing us. But I found a turning point and we just like turned off to the left and then stopped, and then we zip tied the fucker, so that it doesn’t fly away.

(Brent) It was a great idea.

(Tomasz) I mean we don’t know how the videos turned out. We didn’t know but we have high hopes and we put the camera on my bike for a bit and on Brent’s bike for a bit and we switched places in the in the line and the roads were great. This was the best road and the best weather at the same time. The road surface was trash a lot of the time so we had to be careful. There were lots of potholes, there were lots of like, long cracks running along the road. With that in mind though the road was empty, almost empty.

(Neighbour) Hey guys, I’m just leaving with the car…

(Brent) Hey!

(Neighbour) I just wanted to know what you guys wanted me to do with your food

(Brent) Oh, you just throw it over here, yeah that’s awesome.

all right, it’s by your bike.

(Brent) Thanks so much, awesome, thank you.

(Tomasz) Thanks a lot!

(Neighbour) Have a good day!

(Tomasz) That’s fine, that’s just part of the recording now.

(Brent) Right, we had to negotiate for our bear bag. Yeah, well, you didn’t want me to put it in free.

(Tomasz) Yeah, we’ll get to it. Yeah, what was I saying…

(Brent) Great roads, potholes…

(Tomasz) Yeah and there was a super cool fedex driver who was just giving us a run for our money.

(Brent) We could not catch that thing.

(Tomasz) But he was actually really good because you know, a bunch of people pulled, he was so fast he scared a bunch of people and they turned off and stopped and let him pass and so they let us pass as well. So yeah, well done fedex guy. If only I knew… I knew uh how to reach you I’d give you a tip.

(Brent) Um, I saw a black bear. Ahead of you and you and you didn’t spot it.

(Tomasz) Yeah I didn’t see it. But maybe I just don’t know what to look for…

(Brent) Big massive black thing on the move.

(Tomasz) On the moose. Just riding a moose… Yeah, no, I didn’t see it, I was like: “where is it, where is it?”, he’s like: “just there at the top of the crest”… Nah, I can’t, there’s nothing there… Yeah so I don’t know, I mean I can’t really say much because it was just hours and hours of amazing roads and just like… And just riding, riding deep in the corners practicing, you know… Planning the corners entry and exit. I managed to get along with the bike better today than yesterday because I kind of started holding my weight more with my, with my lower body and legs and I wasn’t putting as much pressure on the bars, and that made it less twitchy which actually allowed me to take these like… Long deep corners without being scared for my life. So yeah, way better. Another thing I started doing was something I remember my my motorcycle instructor was always telling me, like, look well ahead like, focus your eyes as far as you can, at the end of the road, because that’s where you’re gonna go, and this helps a lot because your body will just fall in tune with where you’re looking, so instead of looking at just a little bit ahead, just to, you know, I don’t know, spot potholes or whatever.

(Brent) Especially in some of those like super long corners.

(Tomasz) Yeah, you look well ahead and and this makes it way smoother. Yeah, so I had a lot of fun. I hope the videos come out well. Yeah and then we just arrived in Nelson. Which, which was actually, an actual town.

(Brent) Even people in it. So weird.

(Tomasz) There’s people here.

(Brent) Just a bunch of actors I think.

(Tomasz) Yeah I mean, the cyclist can’t be real. What did we do? So we found the campsite no problem, it was booked, it was marked with Brent’s name’o on the tag, because he booked it Thank you Brent for booking the campsite.

(Brent) $33.

(Tomasz) Yeah, $33 well spent, let’s say. Because it’s like, it’s a very central location, which is, I mean I saw it on the map but it’s even better than I expected in terms of walking distances. Because even on the map it might look close but it’s actually far because everything is far here. We expected we’d have to drive around town, but we didn’t. We just walked, so… After we folded out our wet stuff over the bikes because it was hot, everything dried, the tent instant dried after we pulled it out which was awesome. Um, we went to do some kayaking because there was still time, we arrived actually early enough for once to do some activities yeah! So we got a kayak for an hour and we went up from the marina in town to to the bridge that we rode over on the way into town and then back which was really cool. I took some nice photos, yeah. Brent popped his kayaking cherry.

(Brent) Yeah I did.

(Tomasz) I can’t believe you didn’t kayak…

(Brent) I know, I’ve canoed like, countless times.

(Tomasz) But it was similar though, no? Isn’t the paddle very similar or the same?

(Brent) No, it’s one-sided.

(Tomasz) Ah it’s one-sided!

(Brent) Very long and wooden.

(Tomasz) So it’s more like rowing… Actually no, not really, because you have an anchor for rowing right?

(Brent) It’s somewhere in between the two.

(Tomasz) Fine, yeah. Yeah I kept banging my paddle on the side which I assume you’re not supposed to do if you want to go fast because it just wastes energy. Like it doesn’t really hurt the paddle or the boat and just like um… But I also don’t know proper technique because no one ever taught me, I was just always messing around on these. I was like, renting one and doing a bit of a roundabout, like we did yesterday. And the guy said, uh it will take us half an hour one way to the bridge. I mean, we took breaks and it took us 25 minutes. So we did well and we came back. Yeah, we wasted a little bit of time at the beginning partly because the guy was kind of busy after he signed us up and partly because we were messing around or whatever. It’s not like we were trying to beat any records, we were trying to just do something else for once other than just motorcycling and camping.

(Brent) Using your upper body.

(Tomasz) Yeah. I’m not so super sore but I feel it.

(Brent) Okay. It’s a lot of twisting.

(Tomasz) I feel like, yeah, I feel i’ve done something, which is which is good to do something in life sometimes.

(Brent) Uh, the helicopter was cool.

(Tomasz) Yeah we saw a, was it fire helicopter? What is it called?

(Brent) Like a firefighting helicopter.

(Tomasz) Firefighting, right, yeah, hauling a massive tank of sorts, on a wire, of water, or a container with water to douse the fires. Oh yeah, when we were riding past this valley, it was full of smoke. Yeah that was like, you couldn’t see anything, you’re just like, as if the clouds were covering the whole valley and you’re just above clouds, and there’s nothing. In one part so that was really crazy, I wonder if there’s any people living there. If there are they probably have to get out right? you can’t breathe this. Or they just wear gas masks all day long. These big elephant ones you know, like in the trenches.

(Brent) Yeah that’s, everybody has some around here.

(Tomasz) It’s definitely that, it’s got to be that. They definitely didn’t bother leaving, just imagine having sex in these gas masks in the smoke, it’s just like being in kitkat.

(Brent) Oh my God, how you went from smoky valley to kitkat, I’m not sure…

(Tomasz) Well, it’s always easy around you.

(Brent) Oh gosh…

(Tomasz) After paddling around the lake you wanted to go swimming but the beach was quite far and my sandals got wet and I couldn’t really walk in them properly. So I was like, no. I was just like no, we’re not going, I want to go back to the camp.

(Brent) So we went to walmart.

(Tomasz) Yeah, so we went to walmart. Walmart was horrible as expected.

(Brent) Kind of like the breakfast.

(Tomasz) Yeah, I mean, it’s not like… It wasn’t like bad bad, but it was just not nice. There was just nothing, nothing like, pleasant about it. I mean the people who worked there were nice enough but uh… The the way like, I have to I have to ask permission to remove a memory card from a rack, someone has to do it for me. Then I pay and I still get stopped on the way out? To see if i paid? It’s like, you fucking know if i paid, there’s a system for it, but you just don’t bother implementing…

(Brent) Yeah…

(Tomasz) Yeah I mean, why do I need to pay immediately if they don’t even remove the security tag. Wouldn’t that be the whole point? Just fucking, I don’t know. It’s a little lame, let’s say lame is probably the best way to phrase it.

(Brent) Lame.

(Tomasz) Yeah, so we actually ran out of memory on the nopro, so that’s why we have to go and get, get a memory card and now we have another 128 gig card so we should be able to just record the shit out of this trip for the rest of the time. Yeah, I don’t know how we’re gonna transfer it, I think we have to transfer all of the memory card contents to your computer locally and then I take them and transfer it to mine as well if we’re gonna be doing any sort of co-editing, so we don’t have to send like 250 gigs of data over the internet.

(Brent) Yeah yeah yeah…

(Tomasz) Ym yeah, I don’t know, we’ll talk about editing this later. These are way too long when we do it together.

(Brent) Sorry. You talk the whole time, I don’t know…

(Tomasz) Yeah because I feel like i’m talking to you so it becomes a story, and I feel like I need to insert jokes. When I do this on my own I’m just like, bam bam bam bam, this is what we did.

(Brent) So it’s more interesting this way?

(Tomasz) It is, but I’m not sure if it’s going to be easier to like, go through when it gets transcribed.

(Brent) All right.

Tomasz Yeah, canoeing, then uh we got back here, you fixed my shoes that you’ve ruined earlier.

(Brent) True.

(Tomasz) So I could wear my running shoes, my feet were very happy, my toes were twinkling, and um, then because we had shoes we decided: “let’s go to town and see what there is to see”, maybe socialize a bit, maybe you know, find a nice place to go and do some… Have some food or drink, um… We were, you tried to go to the toilet and you locked out the whole camp from the toilet.

(Brent) I did.

(Tomasz) Because you decided it’s a great idea to just guess a password.

(Brent) I was one digit off, I don’t know, of four, which is pretty bad.

(Tomasz) How was I supposed to know that, I didn’t know the code at all. For all I could see is you were just tapping random buttons.

(Brent) No, I wanted to be impressive by remembering and I did not remember.

(Tomasz) It was not impressive, but it was okay. Oh yeah, we helped our co-campers here as well to put the camper in the spot because they had this like, little, cool little trailer… I actually like it, it was so fun. And then and then they let us keep our bear bag in the car because there’s no bear bag storage here. Um, yeah, then we actually went to town, we went past one road, past a bunch of restaurants and places that we were recommended, that there’s events and things happening and then we kept going up just to see what there is beyond that point. We turned upwards, I keep thinking…

(Brent) South I think?

(Tomasz) Yeah, probably, yeah we turned south and then just like one block over there was like, a farmer’s market festival thing.

(Brent) A massive, yeah, a lot of people.

(Tomasz) It was a lot of people, it was a lot of nice things, a lot of like local/organic/handmade things, we had a nice like locally made bubble tea. Um, we went up and down to see what there is, we managed to get some gifts for Klara and Ida, there was a nice lady with a store where she was, she was making her own like, pouches and like, waist bags, and I got a couple of um, like, chalk bags for climbing from her, which are very nice. Made from like, recycled old material. And then, yeah, we spotted the bicycle shop where you could rent bicycles but we’re not going to probably. Too much. We’ll probably rent bicycles at your place if we want to, tomorrow or something. Or we do something else, whatever, I don’t know. Could also go off-road motorcycling tomorrow.

(Brent) Yes.

(Tomasz) Either or, there’s plenty of things to do. The important thing is I just pack everything, right? Get everything ready, clean up my cabin, and then, we can do whatever in the afternoon. And yeah, then we went and found the coffee shop so we know where where it is from here, that we’re gonna go later, we got the idea to find you shoes, we need to go and find a shoe shop. And we found I think one outdoors place that might have gopros, we’ve spotted one. But yeah everything was closed at that point, it was already late and the whole street was closed anyway for, the for the farmers market, so we went uh, we went back down to the to the first street to find food, and actually the first restaurant we went to seemed to have um, vegan and gluten-free accommodations, so we just stayed there, um, and then we were getting served. I decided it would be fun to just start engaging socially with the waitress, which was very… She was very pleasant and nice and actually we had like, she just kept coming back and we kept talking about different things, and I told Brent he has to get her phone number. That’s like base one. Base two is: get her to go out for coffee with us.

(Brent) Right.

(Tomasz) Today. Any bets on how that’s gonna go? Is she gonna come?

(Brent) Yeah, I don’t know.

(Tomasz) I’m gonna… I’m gonna say sadly probably no.

(Brent) Yeah.

(Tomasz) But it was a good effort and it was fun and I think like, I feel like, the like… If she didn’t want to come and chat she wouldn’t constantly keep coming and chatting. And also we kind of screwed up by just eating too slow so she just kept coming to chat when she had a moment but then we were still eating, and she would leave and you didn’t protest either. I tried but it was too late. It was like, by the time I tried to say something she was already gone, like, fuck… Because I think we could have talked much more if we… If we did, if we did it differently. But it’s okay, it was a nice, nice social experiment of just like, trying to see if you can just go talk to her to a random person and take it somewhere interesting. So yeah, she said she was going out with some friends from work and she was gonna get gazeboed.

(Brent) We did teach her that term.

(Tomasz) Which which might, yeah. Which might mean she won’t get up till like, late today so, I mean, I don’t expect her to write us now. I would hope she… She’ll at least say: “sorry I can’t come”. That would be very canadian of her.

(Brent) That’s so true.

(Tomasz) Around midday, you know, so at least then you get her phone number. Which is goal number one, you know, and then you can keep chatting if you want. Because she was really nice. So she was called Trey, and she was from…

(Brent) This is where I speak? Halifax area. Which is Nova Scotia.

(Tomasz) Yeah, I got introduced to some, to a concept of a map, on which I was shown where Halifax is.

(Brent) So far away…

(Tomasz) Yeah, so she’s apparently uh, on a crew of… Um, of what’s it… Air Canada Airlines? But she’s doing a five month kind of break and just being up here with some family and friends. Uh yeah, that was fun. It was fun and we gave her a nice tip at the end to make her feel good about it. Because if it was, if it was all just an act and she was just being uh, friendly because that’s her job, then she did a good job. I was, I enjoyed it, one way or another.

(Brent) Yeah, I was gonna say that’s not the sense I got, but…

(Tomasz) No, it’s not the sense I got but I’m just saying that, if that was it, then good on her, you know.

(Brent) Whatever.

(Tomasz) If it wasn’t then yeah… But I think she was more, she was more interested in talking to you than she was interested in talking to me so…

(Brent) Really?

(Tomasz) For sure. I mean you talked more than me, she talked…

(Brent) Talked more with her than I do with you?

(Tomasz) No. So I, I kind of tried to start conversations but then you would carry them and she would respond to you.

(Brent) Oh I see.

(Tomasz) Yeah. So, there you go. that’s our social experiment. And then Brent got semi-gaziboed, because he decided to order an alcoholic drink, which is unheard of in BrentLand. Yeah.

(Brent) It was a good cider!

(Tomasz) It was good i’m disappointed.

(Brent) Hoppy.

(Tomasz) Disappointed in Brent, alcoholic Brent is weird.

(Brent) It was literally the only thing I could have.

(Tomasz) Yeah no, I’m just joking. I like cider, if I could have cider I would have cider, but uh, I cannot have cider so.

(Brent) I guess yeah, I’ve never seen a non-alcoholic cider.

(Tomasz) It’s just fucking apple juice.

(Brent) Hmm, lame.

(Tomasz) Yeah. Uh yeah, and then we go back to the tent and we’ve, uh… Gouged our eyes on more of BBB.

(Brent) So good! Alpine…

(Tomasz) Alpine. And that episode wasn’t even the most interesting, they just like… Failed because it was so cold.

(Brent) Yeah.

(Tomasz) But it was nice to watch them sufferer. And we went to bed.

(Brent) What a day.

Day 11

▶️ Day 11 recording

(Tomasz) So yesterday we woke up in the tent in Nelson.

(Brent) In the Nelson City campground.

(Tomasz) What time did we wake up actually?

(Brent) It was like 8:39, something that.

(Tomasz) No, it was not so bad.

(Brent) Maybe a little earlier.

(Tomasz) Yeah, I don’t remember exactly, but it was early enough I remember that we finished parking what, like quarter to 11? Or something like this. Like we set off 15 or 20 minutes before the checkout time. Yeah, but we parked, we actually managed to take a shower. So like, the campsite grounds were really well equipped, right? I mean, I haven’t been to a campsite yet with washing machines, microwaves.

(Brent) Right!

(Tomasz) And, you know.

(Brent) Well it is in the middle of the city.

(Tomasz) Yeah, so that was really great. And it’s great that they had showers, although showers had left a little bit to be desired. But it was more than a welcome having them, right?

(Brent) Utility.

(Tomasz) At least we had the flip-flops to like… To not expose our feet to the… To the “surfaces”. And we had our own little soap thingy, that came in handy for the first time. Like properly handy, because until now it was a nice to have, and then it was like, oh we actually have soap!

(Brent) Quite the highlight.

(Tomaswz) Then we went to have coffee. So we went to have coffee and I had breakfast at the nice coffee place. What’s the place called?

(Brent) John Ward.

(Tomasz) John Ward in Nelson Coffee was nice. It was somehow… the niceness of the , because the coffee was quite subtle, the taste was like, nice and smooth, but it was a little bit overshadowed for me by this brownie I had. So the panini was nice and smooth, then the brownie was very flavorful. And I couldn’t really feel the taste of coffee afterwards. It was just too much brownie.

(Brent) It kind of killed it?

(Tomasz) Yeah…

(Brent) Well, that’s their fault having a bad pairing for coffee.

(Tomasz) Well, not really. I mean it’s your choice. Also, I already had some of the coffee with the panini. So by the time I went to the brownie, I didn’t have much left and I was just like… All brownied up. I didn’t want to waste food though. So I finished it. Then we went to one outdoor place, did we buy anything there? No. We only bought things in one outdoor place, right?

(Brent) Yeah, yeah.

(Tomasz) So we went there, they didn’t have GoPro mounts, didn’t have running shoes or anything, and we went to this… Or did we go to the sports place first? We go to sports place first or the mountain? Yeah, then we went to a second shop, you bought the paracord for Pete.

(Brent) Ah yeah!

(Tomasz) Yeah, and then we went to a sports place and the first sports place just had basically enough stock to find something for you. So we got you nice Brooks, whatever they are called. Are they called adrenaline or something?

(Brent) You’re sitting on them.

(Tomasz) If it sits it fits.

(Brent) Yeah.

(Tomasz) Brooks Cascadia, kinda like Canadia.

(Brent) Just use a marker.

(Tomasz) Yeah, so you got Brooks shoes, which is nice because I am a fan of Brooks shoes. I’m wearing them now on.

(Brent) Right?

(Tomasz) And they told us that there’s a Best Buy that opened in the mall and they’ll have GoPro mounts. You called Best Buy and they just were like: “yeah of course, everything’s here for you, just waiting”. We drove down, was there, we bought it, installed the equipment and then we were… Yeah, good to go with recording. The only problem was we had basically one and a half battery left. Actually not even… One was full and the other was more like… Actually like 20%, so there’s very little recording from that day actually. There’s only a few clips. So we got the nice and really solid GoPro mounts but then didn’t actually to take advantage of it. But it’s not like it’s going away anywhere, I will keep using these so it’s fine. Then we were riding. How was the riding from Nelson? I think the beginning was decent but we were stuck behind cars most of the time, right? Actually no, the beginning to the ferry was good!

(Brent) Yeah.

(Tomasz) Then the ferry was nice and we got to speak to a racist grandpa, as you do.

(Brent) who’s lived everywhere.

(Tomasz) I mean that’s the thing, you’d kind of figure that a person who’s kind of… Who lived in a lot of places and moved up in society… To be a little bit more open? But at some point people just try to build a narrative that fits their world view as opposed to adjusting their worldview based on the world around them. And I mean if he lived in the two-and-a-half million pound house in London and he’s…

(Brent) A particular world view…

(Tomasz) He’s gonna be a little bit skewed yeah. And also really I find this fascinating how he, having been so wealthy while he lived in the in the UK was somehow bothered by the poor immigrants? As if that ever affected him in any way. And he probably has never fucking spoken to one of them or like lived among them. But it’s kind of… It’s hard because it makes you wanna like, debunk it, when you talk to someone like this, especially having lived that life and having been, you know, going through all the steps of getting like, child benefit and housing benefit and all of this in the UK, and just being on the receiving end of it, and just saying like: “sorry but you like… you don’t know what you’re talking about” but without being arrogant. Yeah? Because he has lived longer than us by a considerable amount and he has seen things right? So you can’t deny him experience and if you’re just gonna tell him like: “no you’re dumb”, then this is not gonna change his mind. So I try to be diplomatic about it, and it did seem like he was taken aback a little bit. Was he?

(Brent) Yeah.

(Tomasz) But yeah, it’s also a little bit hard when you move from a society like you were in the UK where people are mostly accepting of immigrants, to a place like Germany where they’re not, and when you’re being on the receiving end of the of the immigrant hate you just start thinking like, okay… It could be anybody. I mean it’s me now, but if you as a Canadian were forced now to live somewhere else and you would be on the receiving end you’d have a different opinion very quickly about what is acceptable and what is not acceptable when it comes to immigration policies and benefits and stuff and whatever… Because in the end there are way bigger problems that cause economic issues for a country than benefit scam. Benefit scams are dropping the ocean, right? And then that that is also juxtaposed with the fact that the guy loves trump.

(Brent) Yeah!

(Tomasz) Somehow in his opinion that’s the fix to the poverty problem. It’s just like, one millionaire to another: “Yeah, let’s fix the problems by… Uh… making corporations more money! :et me let me fire up my cigar!

(Brent) “That’ll solve everything!”

(Tomasz) Yeah… So that was fun. A shame we didn’t fucking record the sound of the conversation. That’s hilarious. So we just figured out all of the recordings we took with a little gopro didn’t have sound because I fucked up and turned off the sound recording by accident. Yeah, so the ferry was great, couldn’t see shit because it was so cloudy…

(Brent) It’s too bad!

(Tomasz) Yeah I mean, I don’t know what I missed because all I saw was… Like the intro into the “Alone in the Dark” movie where they go in and everything is just covered in mist, you know. Or was it “Silent Hill? No, it’s “Silent Hill” where it is so misty. Old “Silent Hill” movie, that’s what it felt. Like, and then we got off the boat and the grandpa just like, fucking raced off in his Yaris…

(Brent) I know, that was so funny!

(Tomasz) He just hammed it! And he even made us… He even like, make space for us to like, make us jump in the queue because he thought we’re gonna like, smash it down the road on our bikes,

(Brent) He probably wanted to watch us.

(Tomasz) Yeah… And then we just immediately parked. Then he was like, okay, screw it, I’m going! “Screw you guys I’m going home!” Um, Jesus, and then… And then the rest of the trip to… Was it Creston? Yeah, it’s kind of… It’s kind of okay. It’s kind of, not as exciting because we just had a lot of traffic in our way. Creston itself, we had a nice meal in a thai restaurant. I was starting to get really hungry at that point, so if we left the campsite at… We left the town at quarter past twelve? It’s actually been later I think. We left the town center around quarter past twelve and then we went to get the gopros. So wse left the town itself closer to one. So we were in Creston at about four or five I think?

(Brent) No wonder you were hungry.

(Tomasz) We had late breakfast, we had breakfast around 11:30 or something like this. Yeah, and then after Creston we decided to get lost because we had too much fuel in our tanks clearly, and we need to waste it a bit so we went to see… To wave to the, you know, American border crossing guards, and just leave them in the dust cloud.

(Brent) That was not the intention but it happened. (Tomasz) What do you mean, you’ve completely planted it, you’re just pretending now. You just wanted me to not have enough fuel to get home.

(Brent) It’s surpisingly close though.

(Tomasz) It’s a conspiracy. Then, uhm, yeah we went back, then we were just like, completely were just smashing it down, churning the miles, just to get home. It wasn’t even so bad, that road. It was okay. Better than the motorway on the first day and definitely better than the motorway the second day. Yeah, and then I was starting to get nervous because… Actually not even nervous, I didn’t care. If like, one of us has no fuel then the other one just goes on the bike and brings some fuel. So like, fuck it, right? Or we call your brother because we’re close by then and he can fix us.

(Brent) Yeah.

(Tomasz) Fix us up. But uh, let’s make sure we’re recording.

(Brent) Is there sound?! We saw that biker on the side of the road?

(Tomasz) Yeah we stopped for a biker to see if we can help him, because we saw someone broke down and it was obviously a Harley. I mean, who else would break down? And he just lost bolts because that’s something that happens frequently.

(Brent) I know! I couldn’t believe it! He’s like: “Yeah, this has happened a couple times…”. Bolts just fall out…

(Tomasz) Then he’s like: “Yeah I just need to fix my gear lever to the little shaft with some wire”.

(Brent) It’s like, dude they have thread locker, like it’s a thing!

(Tomasz) Yeah I wasn’t gonna like… We’re there to help him, not to like… To talk him down, you know… So it’s like: “Um, actually… You’re a dumb ass.”

(Brent) “You messed up”.

(Tomasz) So yeah, it was nice chatting. Everybody in general, everybody all the time is nice. Oh yeah, we had a nice chat with a cop and two motorcyclists.

(Brent) That’s true.

(Tomasz) In Nelson, right? About the BMWs they were riding. Yeah, it was fun. And then, yeah… On like… I just kept watching the signs for Crambrook, because it was like, 30km to Crambrook… My bike was showing 20. Great. Then 20 left to Crambrook, and my bike was showing 10. And then 10 to Crambrook and my bike was showing zero. I just kept going and going. Uh, yeah, and just to add to the fun my clutch lever started catching.

(Brent) Oh yeah.

(Tomasz) And it wasn’t dangerous, there wasn’t like, anything. You can always… I mean, in a pinch you can dry shift like, whatever. It’s not good for the bike but you will live. Um, but the worst thing was my screen just decided to have a fucking seizure. It was just strobing on and off the backlight.

(Brent) That’s terrible…

(Tomasz) Anytime I like, accelerate and being like, in the vibration range it would just go: “on off on off”. Yeah, so I think we made it home just in time before I went crazy with this bike. If I had to ride a full day like this it would start getting old very quickly.

(Brent) Well, we would have fixed the clutch.

(Tomasz) I was just gonna let you fix it now. But the screen, I don’t know, I mean, we could have probably taken apart the like, digital assembly and see if something is loose.

(Brent) Who knows.

(Tomasz) But the problem is when you stand static it’s not a problem. It’s only when you’re moving.

(Brent) Yeah, so to test it you have to get them going.

(Tomasz) Or, you’re just like: one person is doing the cables and the other is just shaking it. Yeah, and then we came home to a nice welcome. Warm welcome by Brent’s family and it was very lovely and I got a beer and you got, what did you get?

(Brent) Wine.

(Tomasz) I think wine and we were just like, reminiscing the fabulous Budget Bike Battle Banff.

(Brent) We didn’t actually go to Banff!

(Tomasz) Woops! Yeah, there you go. That was the day. Oh yeah, and then we went to bed.

Day 12

▶️ Day 12 recording

(Brent) Ah, journal time. Yesterday…

(Tomasz) Can you remember anything?

(Brent) Well I guess I podcasted yesterday, but before that we did breakfast.

(Tomasz) Did we do… Oh, yeah… All right, yeah, so, woke up, came to bother you a bit. Actually… I actually had breakfast in the morning already. Then we went to Kaylyn and Aaron, because they said they would make nice breakfast for us. Then I thought it would be way nicer if I actually run before, so I went for a nice run. So yeah, you do have some… Nice, more tame roads for running in the end, just have to, you know, go and explore and then, you know, make a little map of your favorites. I was actually tempted to send you out for a run yesterday. I wanted to see what your shoes are like. Are you gonna run today?

(Brent) Today would be a great day for a run.

(Tomasz) I think so. What does your watch say?

(Brent) It said that had a certain pace thing to do.

(Tomasz) Show us. No, you have to start an activity. Try to start an activity. Yeah, so you’re doing, uh, this is tempo. Yeah so this is like, medium pace. Yeah, I think you should do it because it’s a nice interval thingy. Nine minutes at like, medium pace, with three minutes recovery. This looks like fun.

(Brent) Nice.

(Tomasz) Good test for your shoes. So yeah, I went for a run on a nice gravel track, only had to do a brief moment on the road and it seemed like, every road user that did pass me respected me and gave me lots of space.

(Brent) Good.

(Tomasz) I mean, it’s not always easiest because it’s a double yellow there, and but there’s plenty enough space off the road for you to run on as well. Then I came back, had a quick shower, had some nice breakfast. We had like, fried eggs with onions and stuff?

(Brent) Yeah I think there was like, a vegan meat.

(Tomasz) Okay, cool. And yeah, some tomatoes and a very nice pineapple.

(Brent) It was perfect.

(Tomasz) So much pineapple. Oh yeah, and Aaron made us his nice coffee, it was also very tasty. And then you postcasted. I went to do my check-in for the flight, which was easy. I got to select my seats this time.

(Brent) Yay!

(Tomasz) Yeah, because last time when I checked in every… Any seat that was selectable I’d have to pay. Yeah, so I didn’t, right? Yeah, and this time there was still a bunch of empty ones…

(Brent) Great.

(Tomasz) …that I could pick for free. So I just picked something by the window on all three flights. Um, then I went to pack most of my stuff, except the stuff I was wearing already. I washed my running clothes from earlier, hung them out to dry and then I went with Aaron and Kaylyn to town to just, you know help, them in the renovation efforts, since Brent was busy podcasting anyway, I thought I might as well go with them and just do something nice. Which was nice, got to learn a lot about practicalities of doing house maintenance with the infamous, what you call it, stud based wall thingies, that you can’t just drill wherever you want.

(Brent) Yeah, they’re massive spaces.

(Tomasz) Yeah, you just have plasterboard everywhere. Which, I mean, I’m no stranger to plasterboard, but… Um, yeah, every time I try to say: “oh let’s just like, screw, drill here” and I’ve caught myself out like, actually: “no, you can’t just drill here”… Here you have some limitations. That was funny, and you have funny square head screws as well.

(Brent) Yes, those are Canadian.

(Tomasz) Yeah, Aaron said it’s a Canadian standard, it only exists here.

(Brent) Pretty much.

(Tomasz) Uh, also Aaron has a funny drill, where like… So when I have just my battery hand drill, it’s smooth. But Aaron’s drill just goes in steps.

(Brent) Yeah, those are called impact drivers.

(Tomasz) Yeah.

(Brent) They… in the end make it easier. And they grip better and all this stuff. And they’re lighter, et cetera, et cetera…

(Tomasz) I found that to be the opposite at the beginning.

(Brent) Yeah, same.

(Tomasz) It just felt like it just keeps slipping because every step is so big, and then it stops and… But then I was like, oh you can actually change the speed.

(Brent) That’s true.

(Tomasz) So I lowered the speed and then it was nice. However, I mean, um yeah, it’s interesting. I need to see if they exist in Germany.

(Brent) They do.

(Tomasz) They probably do.

(Brent) I owned one for the longest time because I bought a kit that was like, a normal drill and one of these impact drills, and I didn’t use it for a couple years and Aaron convinced me. He’s like: “No no like, try it!” And sure enough like, once you get used to it, it’s amazing, and you never go back.”

(Tomasz) Yeah, so I might i might be up for another purchase. Another thing that we don’t have time to get, I need to see if they exist in Germany, is those inflatable pillow things that you can wedge stuff up

(Brent) Those are for leveling things.

(Tomasz) Because I really struggle with leveling washing machines and stuff.

(Brent) Super useful for that.

(Tomasz) And this would be perfect, and they only like, we said like 20 dollars a pack that’s like, a couple of them. And he was like: “Yeah, we can go to the shop and get it.” And we forgot. But they must exist in Europe because the ones he has are made in Denmark.

(Brent) Okay, that kinda gives it away, doesn’t it?!

(Tomasz) So yeah, that was fun. It was a fun experience helping him and, you know… He’s just living that the lifestyle that… That I at least only got to see in youtube, whenever LTT does their stupid home makeovers.

(Brent) Oh man…

(Tomsaz) Now I get to see, I get to see how it works.

(Brent) Did aaron show you a stud wall? Like, did you go into the mechanical room and see how…?

(Tomasz) Yes we did, I mean, I have kind of seen it because, like I said, I have built, helped build plaster walls before.

(Brent) Okay.

(Tomasz) But they’re usually small connecting walls when you divide the room, they’re not just your…

(Brent) Structural…

(Tomasz) Yeah, I mean obviously these are not structural walls like, this, the structure is carried by some pillars and stuff I guess or whatever? No? Okay. So if you have a multi-story building, how is the second floor propped?

(Brent) Yeah, it’s built on the same…

(Tomasz) …on the same wooden frame. Yeah, okay, fair enough. Yeah that doesn’t exist, it’s just dividing walls. But we have it, if you want to split a room or something, I actually built, designed and built one with a door inside, in berlin. I always overbuild them, I always add way too much wood and way too much, uh…

(Brent) We have like, such… We don’t even think about it, but they are such a well understood standards that for doing things like doors and windows…

(Tomasz) It’s almost like Lego, it’s just like “block block block block block…”

(Brent) Yeah but also you have to think about like, force distribution, so above doors and windows you usually have like, a bit of a beam, that transfers those particular members over to the next ones.

(Tomasz) Yeah.

(Brent) That’s kind of interesting actually.

(Tomasz) I kind of did that when I built my plaster wall with a door, I actually had a hanging door, so the door was hanging off from the top beam.

(Brent) Okay.

(Tomasz) Um… But yeah, super fun actually. Considering like, it was just a simple chore thing. Yeah. And yeah, he was… Fun fact, before you came Aaron was just on his phone and I did all the leveling and adjusting and screwing, and just when you came…He’s like: “Oh, you’re just watching”. I did all of it, and then Aaron like just stepped in as you came in.

(Brent) That’s hilarious.

(Tomasz) He just constantly kept getting phone calls from Vance, because Vance was on some sale somewhere.

(Brent) Yeah yeah, for the tires or whatever.

(Tomasz) There was some construction company going… Going under and they were doing a sale of all of, whatever they owned, and just found so much, and he was like: “Yeah do you want this? Do you want this?” And he was just like, on the phone the whole time and I was like, plucking away.

(Brent) That’s funny.

(Tomasz) Yeah, then we went and had lunch/dinner, in the fire hall again, and oh my God you can have poutine as a side! I was like: “Oh yes!”.

(Brent) I know, right? And you can pick any poutine!

(Tomasz) I can have… I can have a second meal as part of the first meal? Oh yes please! So yeah, the burger was nice, it wasn’t like, the nicest burger I’ve ever had, um… Actually a lot of like, the burgers in Berlin we order are usually a little bit larger in diameter, and they are… This one was a little bit on the dry side.

(Brent) Okay.

(Tomasz) But that has… That’s like, it’s whatever you like, right? Um, usually when you order something that has more sauce in it, it eventually dissolves, so it can be messy. So yeah, that’s what you want, but all in all it was a very tasty meal. This like, poutine with pulled pork is just so, so amazing. And you even managed to wrangle them into making you a vegan, gluten-free one.

(Brent) Yeah, took a bit but we got it.

(Tomasz) Yeah so now you know what to do, you can just go there and have it.

(Brent) Although mine was so big, I could have just used the side. But it was like…

(Tomasz) Yeah, when I looked at it, I at first thought that this was the plate to share between Kaylyn and Aaron…

(Brent) Yeah, I know, right?

(Tomasz) And I was like, “Oh no! That’s just Brent’s meal…” Oh my God, okay… I mean, you still have it, right?

(Brent) I still have half of it.

(Tomasz) Yeah, uh, then we, well, we came home, and finished packing, I asked you for some cleaning supplies and then we watched the shit out of bbb.

(Brent) Yeah we did!

(Tomasz) I did it with sunglasses on to pretend like I care about sunlight blocking. But at least we got to spend some time outside because the weather was nice. It was just the right temperature to sit outside.

(Brent) It’s true, that was a great idea.

(Tomasz) And, following that, we did the finances. In the end I owe you like 250 euros.

(Brent) The total trip, I looked up, was uh… 2100 Canadian dollars, not including like, mechanical stuff for the motorcycle.

(Tomasz) Would you consider that reasonable? This is for two people right?

(Brent) Yeah, I thought so.

(Tomasz) And this includes food and fuel right?

(Brent) Food, fuel and hotels.

(Tomasz) And we did not hold back on food.

(Brent) No no, the food was… We could have did better on food. But we kind of splurged.

(Tomasz) Yeah, I mean, it’s holidays, right? You’re, you’re just, uh, doing things to to enjoy it and try new things.

(Brent) McDonald’s. So we could have spent a little less but I think we did pretty well, like, camping was a great way to save money.

(Tomasz) Yes for sure, because like…

(Brent) 20 to 30 dollars per night.

(Tomasz) Instead of 250.

(Brent) It’s like 10 times less.

(Tomasz) Yeah, so, I think for sure we did everything right. Everything worked out, you know, way… I’d say it was, in the end, set up perfectly for a balance of cost and enjoyment.

(Brent) Nice.

(Tomasz) I wouldn’t have changed any of it.

(Brent) I agree.

(Tomasz) Which is why I asked you, like, what do you think? Because like, every person has a slightly different outlook on finances, but for me this good.

(Brent) Yeah, I thought it was great.

(Tomasz) Yeah, I mean, this also included like, Senas which were 250 a pop.

(Brent) Also a splurge.

(Tomasz) Yeah, but well worth it. And then finally we went to bed right? Yeah, I thought I’m gonna shower but I thought, ah fuck it, because I probably have to shower in the morning anyway. And I already showered after the run.

(Brent) Okay.

(Tomasz) So I went to bed. Make sure everything is charged and plugged in and stuff. Yeah, that’s it. And we are here.

(Brent) Where are we now?

(Tomasz) Now we’re just sitting on a bench right next to a cliff, just on Aaron’s property. Just looking out down towards the the marshlands just below and the rail track in the distance. On the backdrop of mountains.

(Brent) I thought maybe we’d seen Angelette. No such luck.

(Tomasz) Next time.

(Brent) Maybe.

Day 13

▶️ Day 13 recording

So it’s not the end of the day yet but the trip is over and I’ve actually been, I guess traveling for more than a day or let’s say… I’ve been awake for probably 24 hours at this point or more. Morning was easy, no problems. Got up, cleaned up the cabin, had breakfast, washed the bathroom, especially just to make sure I leave it in at least the same condition as I found it. And I went to Brent’s cabin, the cats, they probably felt like… They could feel like I was leaving or something. But yeah after, two weeks of being there they finally got used to me. Which is kind of funny, just at the end.

Aaron and Kaylyn… Kaylyn was on a phone call in the morning, but before I left they got dressed in the sweaters that Eli made for them for Christmas. We took a group photo, which was very, very pleasant. Nice touch. We said goodbye, went to the airport. Again, no issues. The airport, as before, is not… Arriving there again, it seemed way smaller at the first sight. We got some water from the water fountain. First flight no problems, second flight also no problems, I sat next to a very wide lady, although it’s still much better than what I had to go through before.

Fun fact, on the first tiny airplane I’ve flown with to Vancouver my phone was detecting my luggage somewhere below us, I guess, in the storage hold, and also on the big Boeing jet! It would momentarily make me lose track of it and send me information that, you know, I was losing track of it. I will check where it is now, last time, we will see. No, I still don’t (detect my luggage).

This is beautiful, quite beautiful, really beautiful. (I just received the message that my luggage did not make it onto my flight from Frankfurt to Berlin). Which is ironic, because on the way to Canada I couldn’t detect it and I wasn’t able to track it the whole time.

So I had to fill out a form and now it (my luggage) is shipped (to my house).

Interesting fact. The way the plane flies… It goes far enough north that it never actually gets dark on the plane. So they just turn off the cabin lights. I guess everybody closes windows and they pretend it’s night time. So I did manage to get some sleep. Garmin says I only managed to get one hour sleep, but I was kind of on and off. Definitely not high quality sleep. But I got the impression that the day has passed. I have to say I was really happy to be back in Berlin, even without the suitcase.

Yeah, we are back. Not with everything, but this suitcase only had clothes, so that’s OK.

Hesitant at first, I decided to join as in this episode we touched on Garmin smart/sports watches, of which I am a big fan and a long time power user. Hope you enjoy this one and indeed my debut in audio format as much as I did. All the best to the whole JB crew, and to the next 500 episodes and more, cheers!

Links

• Jupiter Broadcasting: https://www.jupiterbroadcasting.com/
• Linux Unplugged: https://linuxunplugged.com
• My guest page: https://linuxunplugged.com/guests/tomasz
• My guest appearance episode: https://linuxunplugged.com/572

I am a casual programmer that enjoys the art, though not so much in a professional manner. Yaml engineers the call us, or terraformers. Administering the cloud rarely demands access to my programming skill set, though when it does call for it I always prefer it to be sharp. And so I sharpen it here, on these pages and over on my trusty teapot.octopusx.de.

This new idea for a small project sparked in my mind after a discussion with a friend about migrating away from the dreadful and (somehow) still alive Evernote and over to a more open platform. They asked me if Nextcloud notes supports voice notes, like Evernote does. Well, it does not have that feature, I said to them. But… why should it not? I decided to see what it would take to make this a possibility, and though the resulting plugin is not strictly comparable to the functionality in Evernote, I think you may still find it a fascinating plugin, perhaps even useful! Let’s dive in and set out some rules…

What is it we’re actually trying to do here…

So, there is no built-in flow to simply record a voice file and treat it as a note, with the ability to tag or add metadata. In case of Evernote, you can create a voice memo, then include it in a note, akin to an email attachment. This way you can tag it and add any content to said note that you could potentially use in order to find the voice note again in the future. This though gave me an idea. What would be even better than extra tags and text added to such a voice note manually? Why, the contents of the voice note itself, in form of text!

My goal was therefore as follows:

• Have the ability to easily record, then seamlessly upload the recorded voice memos into Nextcloud
• Have the notes transcribed automatically in the background as they are uploaded
• The final transcriptions should take shape of a Nextcloud note, with any tags and metadata generated and attached automatically, to make searching for it and its content easy using Nextcloud’s unified search feature
• Ideally have the note either spell out the name of, or include an actual link to the voice file itself, making it possible to find the original recording and listen to it by essentially searching its content via the proxy of its transcribed note

Step one, let’s record something

Since the Android Nextcloud application has the ability to watch a certain directory on the phone, then apply rules to said folder about when and where to upload its contents, I thought this should be easily leveraged to watch for voice recordings generated by another app. After a short search on F-droid I have found a simple and perfectly functional application that I could use to record voice notes. The Simple Voice Recorder.

As a bonus, the default file naming scheme for the voice notes allows for easily identifying and ordering voice notes to be ingested by the transcription application, more on that later…

Step two, find out what’s new

So, the voice note creation and uploading is sorted simply. The next step to design is to somehow detect that a new voice note was uploaded and trigger a transcription of it. At first I thought I can actually use the (relatively) new App API released last year by Nextcloud. This new API allows for coding and integrating new Nextcloud apps/extensions in most any programming language, which is great for me as I am not proficient in PHP.

Sadly, after a short bit of research, it turns out that in order to deploy such app one must dedicate an entire virtual machine as a deployment target, then give Nextcloud access to the docker socket on that host. In short, to use such an app I would have to spawn another “cluster” alongside an already complex Kubernetes cluster that I host and manage. This would mean that, if I wanted the same level of resiliency, scalability and security for this VM and the Nextcloud apps running on it, I would need to duplicate a lot of work that one normally doesn’t need to duplicate when running everything in Kubernetes. Even worse, I have a suspicion that a lot of other developers feel similarly to myself, as the “External App Store” is barren, with only a couple small project hosted there alongside the 4 examples that were presented at the conference last year. Truly disappointing.

So, from one janky solution straight into another! The fastest way to see if new files got added to a folder is just to check the filesystem, right? Thankfully I am already using an NFS share for the file storage as I am running multiple instances of Nextcloud in my cluster and this is how I provide the file access to multiple containers. Sharing the NFS volume with another container was therefore not a big deal at all.

My first stab was using inotify and looking for filesystem changes to a specific directory mounted from the Nextcloud volume. I called the directory Recordings, and it is where all of my voice notes land when my Android phone uploads them. then run whisper on any new audio file and use the Nextcloud notes API to post new notes. This worked locally, but had exhibited odd behaviour when I deployed it to my Kubernetes cluster. It turns out that NFS mounted file systems do not generate inotify callbacks when files change on them, as that is a low level system call that is only triggered on the actual hardware the original filesystem resides on.

If at first you fail, try, try again. So I decided to instead use the file names of the voice notes as a guide, as they are independent of file system mechanisms and include the date and time of each recording (as I mentioned earlier). In the first iteration I generate a timedate object based on current time and ingest files with timestamps newer than that one, then move the timedate object to a new timestamp after ingestion. This seems to work ok from my limited testing, though long term I will for sure implement some sort of time sync method, in order to preserve the last ingested time stamp before container restarts.

Step three, write it down

Now, that we have a way to find audio files to transcribe, we can feed them into the openai-whisper library. And wow, is this one easy to use and… just working? Much simpler to work with and lighter to run than I expected.

The documentation gives us this quick start example:

import whisper

model = whisper.load_model("base")
result = model.transcribe("audio.mp3")
print(result["text"])

And that is basically all there is to it. The only prerequisite is that we have ffmpeg installed and available on the system.

The first time you load up the library it will look for the model you’re trying to use locally, then proceed to download it if it is not already present. Fun hack, in order to prevent my final deployment to keep re-downloading the same model over and over again on each restart, I added this script:

#!/usr/bin/env python
import whisper
model = whisper.load_model("base")

to a step in my dockerfile, so that it always comes preloaded with the basic model.

I wish there was more to write here, but it is really this straight forward…

Step four, save it for later

As I already alluded to in step two, I didn’t end up using the external app API to integrate this app with Nextcloud. There are however many options available to us, and the simplest one I found was to use the Nextcloud Notes API. This simple wrapper gives us all of the functionality we need, with nothing we don’t, just take a look at this example block:

from nextcloud_notes_api import NotesApi, Note

api = NotesApi('username', 'password', 'example.org')

note = Note('Shopping List', 'Spam', favorite=True)
api.create_note(note)

I was really expecring this project to take longer to cobble together, but there are just so many community projects and libraries covering the functionality we need, it was positively trivial.

In summary

We made it! This is about it basically. We read the file system for new files with matching filename patterns, feed the audio files through whisper then post the transcriptions as notes. And boom! We can now use the search feature of Nextcloud to find our voice notes.

What we have so far however isn’t perfect. There is a number of improvements I am planning to make to this little app in order to make it actually useful to other people. In the near future I hope to:

• Allow for more flexible configuration, the ability to choose the source folder and the pattern/metadata of the resulting notes
• Move away from scanning for files on the NFS volume into either making some form of an API call to Nextcloud, or using webdav to query the directory state
• Cache the timestamp of the last note transcribed, in order to perform graceful resumes on Notescriber restarts

Notescriber, currently at version 0.0.5, is available publicly over on my Teapot instance, so feel free to try it out, or even reach out to me if you’d like to contribute. And with that call to action, I wish you all many well transcribed audio notes and a wonderful start into the new year!

With that in mind, I was not craving any sort of a replacement for those services, as many have been with the recent mass migrations to the Fediverse. I was enticed by the prospect to join this migration nevertheless, as social networks in their federated iteration are a much healthier fenomenon in my opinion, and as we know they live or die by the interactions and presence of their users. Joining was going to by my 2 cents added to the jar of reasons for the federated future to keep getting more realistic for all.

How does the Fediverse work?

The Fediverse is a decentralised social network, where users are not tied to any single platform or service. More importantly, they are not locked in by any one vendor or company. Any server can join or leave the network at any time, and user’s data is not stored by any centralised entity. One can begin to participate in the social network by creating a user account on one of many free or paid federated servers. The most common type of servers in the Fediverse are Mastodon servers. The main instance ran by the Mastodon team is mastodon.social, and another popular among the FOSS community is fosstodon.org.

I have created and used my account on mastodon.social for a few months now, and it definitely gives me a more community-like vibe. I discover and follow accounts of people that I listen to in podcasts or follow via github or even some that I have met in real life (what an idea, eh?).

Creating an account on someone else’s server though wouldn’t be the self hoster’s way now, would it! Having now been invested in the network somewhat and gained a basic understanding of how it can be used, I decided to start a search for a suitable way to deploy my own instance. First port of call was the full fat Mastodon server. Hosting these has become very popular in the recent months thanks to Techno King himself, and so many a person has tried it and there are a lot of documentation and information about what is involved. You can see this blog post as an example. I have listened to a Late Night Linux show too at some point where I remembered one of the hosts mentioning that they hosted a server at Canonical for a few people and shortly after they were using up as much 5mbit of bandwidth per second constantly and an extra 50GB of disk space per month! If this were true for my instance I would run out of space in my homelab after around a year and a half…

Why does this happen though you may ask? And is this a Mastodon issue, or is it inevitable no matter what server you? So, to an extent it is inevitable, though likely not as bad as it seems, let me explain. Let’s say, you’re creating your own server and on that server you create your Fediverse account. At that point the only data the server will store is anything you post. Any images or text you put in your status updates. Now, if you crate a few accounts for some friends/family, everyone on the server can see each other’s posts, no big deal, you’re still just storing only the posts of your own users. The data transfer and storage issues begin when you start following people on other Mastodon/Fediverse servers. Each time you follow someone on another server, your and the other server will form a bond, i.e. federate. In order for your account to get updates on the person you follow on another instance, your server will now permanently connect to the other instance and continuously download and locally store the posts of the other user. So, the more people you follow the more storage you end up using locally. And the more servers you federate with, the more bandwidth you will end up using.

What to host?

As I am a hipster at heart just hosting an instance of an alternative social network was too mainstream for me, I started looking for a lightweight Fediverse server to host instead, one that would hopefully help me keep the “operating costs” down. I am not paying for the resources used up by the server as I am self hosting, but I am limited by what hardware I have and I would rather have it last me as long as possible without needing to upgrade (again).

A cursory search has brought me to the GoToSocial page and yeah… This sounds like just the perfect thing to deploy! The software is still (as of writing of this post) in Alpha (more on that later), but the base functionality is there and people seem to run it successfully. The documentation states that it federates successfully with most common ActivityPub servers out there, so no blockers. Also very importantly the developers and community have a public Matrix channel where you can ask questions, get support and talk about the project, which is always a big plus for me :+1:! I joined to say hi and everyone seems to be super friendly and helpful, give it a try at #gotosocial:superseriousbusiness.org.

I assume you have read ALL MY OTHER POSTS RIGHT!? Sure you did… So you know what’s next… Helm charts :D Yes, there is a fantastic ready-to-use helm chart for GoToSocial written by the github user fSocietySocial. You can check it out in their repo here. It is fully featured and all of the important configuration parameters (as far as I am aware) are cleanly exposed via the values file:

replicaCount: 1

image:
  repository: superseriousbusiness/gotosocial
  pullPolicy: Always
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

{...}
# Just look at these options!?
  config:
    logLevel: "info"
    logDBQueries: false
    logClientIp: true
    logTimestampFormat: "02/01/2006 15:04:05.000"
    applicationName: "gotosocial"
    landingPageUser: ""
    host: ""
    accountDomain: ""
    protocol: "https"
    bindAddress: "0.0.0.0"
    trustedProxies:
      - "10.0.0.0/8"
      - "172.16.0.0/12"
      - "192.168.0.0/16"
    db:
      sqliteAddress: "/gotosocial/storage/sqlite.db"
      maxOpenConnsMultiplier: "8"
      sqliteSynchronous: "NORMAL"
      sqliteCacheSize: "8MiB"
      sqliteBusyTimeout: "30m"
      sqliteTmpdir: "/gotosocial/storage/"
      visibilitySweepFreq: "1m"
      cache:
        memoryTarget: "100MiB"
    web:
      templateBaseDir: "./web/template/" 
      assetBaseDir: "./web/assets/"
      accountsCustomCssLength: "10000"
      deliverToSharedInboxes: "true"
      injectMastodonVersion: "false"
    instance:
      federationMode: "blocklist"
      exposePeers: false
      exposeSuspended: false
      exposeSuspendedWeb: false
      exposePublicTimeline: false
      deliverToSharedInboxes: true
    accounts:
      registrationOpen: true
      approvalRequired: true
      reasonRequired: true
      allowCustomCSS: false
    # AND MANY MANY MORE!!!
{...}

Full file for reference: https://github.com/fSocietySocial/charts/blob/main/charts/gotosocial/values.yaml

Installation is the usual stuff for a helm chart:

helm repo add fsociety https://charts.fsociety.social
helm repo update
helm upgrade --install gotosocial fsociety/gotosocial --namespace gotosocial --create-namespace --set gotosocial.config.host='domain.tld' --set gotosocial.config.accountDomain='domain.tld'

As usual, I created a git repo for this on a private gitea server that I can run automation from and where I can override and version control my own values file. Then deployed!

Alpha

I dived head first into this, excited to try out my new, self hosted federated instance! And, as they say, got to break a few eggs. When I interact with a new helm chart I generally make a copy of the values.yaml and browse for a bit, trying to understand what the important options are and what the defaults seem to be. I used my best judgment and ran helm install gts . and BOOM! It worked XD I was able to create a new user, log in and configure my profile. Then I tried to search for users on other servers and… nothing. A bunch of errors in the logs later I have narrowed the problem to be my ingress configuration. I can reach my server form local network and externally, yet public Mastodon instances weren’t able to get the webfinger for identifying my user account on my server.

The culprit was the following block of config in the values yaml:

  config:
    host: "gts.octopusx.de"
    accountDomain: "octopusx.de"
    protocol: "http"

As I was just rushing for the goal I guesstimated what goes into these fields, however incorrectly. The host is supposed to be the full hostname where the server is reachable from. The accountDomain is what will be used to append to the end of your user’s URL to signify which server they are coming from. Now, you can make the two different, but then the second one needs to still server some traffic under that URL for federation handshakes. The easiest thing is to have them be the same.

Having fixed that I was still getting 404s from the other servers according to my GoToSocial logs. Though I spotted the source of this problem fairly quickly. The other servers were trying to communicate with my instance over http. I assumed that the protocol field config has to be set to https only if I let GoToSocial terminate SSL and not use a reverse proxy, and that was also an incorrect assumption.

The working config looked as such:

  config:
    host: "gts.octopusx.de"
    accountDomain: "gts.octopusx.de"
    protocol: "https"

Tada!

For a better explanation of all of the configuration parameters I encourage you to study the official example available in Github, richly embellished with detailed documentation and explanations, available here: https://github.com/superseriousbusiness/gotosocial/blob/main/example/config.yaml. Really well done on the GoToSocial team’s part!

Hopes for a brighter future

A lot of my blog posts come from stories of struggle, learning and overcoming problems. Setting up this little ActivityPub-compatible server was so straightforward and painless I almost didn’t write a post for it. But having chatted with Brent from the Jupiter Broadcasting crew made me realise, that this is actually a good thing! It is a success story worth sharing and GoToSocial is a project worth spreading some positivity about! So here we are. What will come next is checking storage and bandwith usage over time, I am keen to find out how heavy it really is to host a Fediverse server, I might do an update in the future with my findings if the data happens to be interesting. And so, with that, I shall see you again, on another page, or maybe even in the Fediverse!

Give yourself no excuse

When it comes to building positive habits, including trying to exercise regularly, I find it important to lower the barrier to entry for myself. One of the ways that I have achieved this is having an indoor cycling setup with a bike on a turbo in front of a TV screen. The bike is always ready, the turbo can be turned on remotely from my phone and controlled by a sports app that suggests me the optimal workout for each day. I try to always have fresh cycling clothes ready. No need for my brain to perform mental gymnastics and participate in a self indulgent competition of making up excuses related to all those minor inconveniences, seemingly rendering me incapable of just jumping on a bike and sweating out some of my frustrations and calories that I could really do with getting rid of.

As a techie the gadgetry side of sport science is very appealing to me and naturally I have eventually paired my turbo trainer with an expensive bluetooth fan which promised seamless integration into my indoor cycling ecosystem of gadgets. Alas, it was not meant to be. The fan is able to simulate wind speed of up to 25km/h. However the indoor trainer does not simulate correct speed in ERG mode, and that is the mode I predominantly use for my training (using TrainerRoad). The fan can also track your heart rate, but for whatever reason it would never reach the full speed no matter how high my HR had gotten during the workouts I had tried this feature on. This has left me stuck with the manual speed mode, where I must adjust the speed myself between the 4 predefined speed presets. That would not be a big issue had it not been for the fact that in order to use this option I have to either:

• Jump off the bike in order to reach the buttons of this, so called, “smart” fan
• Switch apps on my phone multiple times mid ride from TrainerRoad to the Wahoo app and back in order to adjust the speed

This was not ideal and definitely did not match my usual “low barrier to entry” vision for how I like to enjoy my indoor sweating time. And don’t get me wrong, this fan is fantastic (pun intended) and does a great job in the core area of its functionality - being a fan and cooling me down, but I decided to take the matter in my own hands and take it up a notch from good to great!

Research

First things first, we need to see what is already available. A search and a look around had uncovered a github repository which we can use as a code donor for our project: https://github.com/garanj/wearwind. Although it is a Java project, an application for Android wear, we can learn a few interesting facts from it. To start, it is controlling the fan by sending it byte arrays of data in the following format:

private val POWER_ON = byteArrayOf(4, 4, 1)
private val POWER_OFF = byteArrayOf(2, 0)

As seen here!

This was a good starting point and following some experimentation I have decoded a few more commands which I found useful, notably sending the unit to sleep:

SLEEP = [0x4, 0x1]

And setting a speed of choice:

# Second byte is the speed, values can range from 0x1 to 0x64
MIN_SPEED = [0x2, 0x1]
FULL_SPEED = [0x2, 0x64]

Here are some other commands that I figured out I could send to the device:

ON = [0x4, 0x4, 0x2]    # Wake up the fan
OFF = [0x2, 0x0]        # Shut the fan off, including BLE
SLEEP = [0x4, 0x1]      # Shut the fan off, leave BLE on
HR = [0x4, 0x2]         # Put device in auto HR mode
SPD = [0x4, 0x3]        # Put device in auto speed mode

The beginning of my journey with python and BLE started with doing research on existing libraries and implementations of the BLE stack, where I have discovered the following resources:

• https://github.com/pybluez/pybluez
• https://github.com/ukBaz/BLE_GATT

These seemed like they are used in various projects on github, and the pybluez even comes with handy examples for how to do BLE scans:

• https://github.com/pybluez/pybluez/blob/master/examples/ble/scan.py

Trying to use this library however I quickly learned an interesting fact: BLE is not supported on many laptop or desktop wifi/ble modules! Having had trouble running this code, on a whim I decided to try it our on a raspberry pi 4 I had laying around, and that worked without a hitch. I was able to scan for BLE devices nearby and read their advertised lists of characteristics:

service000a = "00001801-0000-1000-8000-00805f9b34fb"                    # Generic Attribute Profile
service000a_char000b = "00002a05-0000-1000-8000-00805f9b34fb"           # Service Changed
service000a_char000b_desc000d = "00002902-0000-1000-8000-00805f9b34fb"  # Client Characteristic Configuration
service000e = "0000180a-0000-1000-8000-00805f9b34fb"                    # Device Information
service000e_char000f = "00002a29-0000-1000-8000-00805f9b34fb"           # Manufacturer Name String
service000e_char0011 = "00002a25-0000-1000-8000-00805f9b34fb"           # Serial Number String
service000e_char0013 = "00002a27-0000-1000-8000-00805f9b34fb"           # Hardware Revision String
service000e_char0015 = "00002a26-0000-1000-8000-00805f9b34fb"           # Firmware Revision String
service0017 = "a026ee01-0a7d-4ab3-97fa-f1500f9feb8b"                    # Vendor specific
service0017_char0018 = "a026e002-0a7d-4ab3-97fa-f1500f9feb8b"           # Vendor specific
service0017_char0018_desc001a = "00002902-0000-1000-8000-00805f9b34fb"  # Client Characteristic Configuration
service0017_char001b = "a026e004-0a7d-4ab3-97fa-f1500f9feb8b"           # Vendor specific
service0017_char001b_desc001d = "00002902-0000-1000-8000-00805f9b34fb"  # Client Characteristic Configuration
service001e = "a026ee0c-0a7d-4ab3-97fa-f1500f9feb8b"                    # Vendor specific
service001e_char001f = "a026e038-0a7d-4ab3-97fa-f1500f9feb8b"           # Vendor specific
service001e_char001f_desc0021 = "00002902-0000-1000-8000-00805f9b34fb"  # Client Characteristic Configuration

Characteristics are like fields, or ports, that you can either read from or write to, and after some more experimentation I have identified that in order to control the fan I need to send data to characteristic a026e038-0a7d-4ab3-97fa-f1500f9feb8b.

For a much better explanation of how BLE works and what tools you can use to play with it, I highly recommend this blog post: The Practical Guide to Hacking Bluetooth Low Energy by Vaibhav Bedi.

Moving forward, I had struggled to use the basic pybluez library for this type of communication and have switched over to using a simple yet powerful, more abstract library, i.e. bleak: https://github.com/hbldh/bleak.

It is during this transition that I realized that the BLE protocol is only used for scanning for devices and exposing their capabilities. In case of my Headwind fan, once I have found out the characteristic and MAC address to send data to, I no longer rely on the BLE stack for anything and could go back to sending data to the fan directly from my laptop. I did not need to scp my code to my local trusty Pi4 no longer from here on out.

PoC

With what we have learned so far, I started off by bundling all of the bluetooth-related functionality into the Headwind class. This will be our means of talking to the fan and asking it for state information.

from bleak import BleakClient

# Config byte arrays
ON = [0x4, 0x4, 0x2]
OFF = [0x2, 0x0]
SLEEP = [0x4, 0x1]
HR = [0x4, 0x2]
SPD = [0x4, 0x3]
MIN_SPEED = [0x2, 0x1]
HALF_SPEED = [0x2, 0x32]
FULL_SPEED = [0x2, 0x64]
# Characteristic to read state and write config to
CHARACTERISTIC = "a026e038-0a7d-4ab3-97fa-f1500f9feb8b"

class Headwind:
    fanClient = None
    flaskApp = None

    def __init__(self, flaskApp, address):
        self.flaskApp = flaskApp
        self.fanClient = BleakClient(address)

Bleak abstracts the complexity related to establishing the bluetooth connection, and reading and writing data to the bluetooth device.

In case of the Headwind fan, we have established that to read the state of the fan we need to read the value of the CHARACTERISTIC like so:

    async def readSpeed(self):
        try:
            async with self.fanClient as client:
                result = await client.read_gatt_char(CHARACTERISTIC)
                # The result will be a byte array
                # In this case it will return something similar to: 0xFD-01-XX-04
                # Where XX is the current fan speed
                # Hence we return the 3rd array element to get the speed:
                return result[2]
        except Exception:
            return 0

Ok, so what about changing the speed? It is also fairly straightforward. Take a look.

    async def writeSpeed(self, speed):
        if speed > 0:
            # Create a byte array that includes:
            # - the mode (first element)
            # - the speed value (second element)
            value = [0x2, speed]
            try:
                async with self.fanClient as client:
                    # Then we write the prepared byte array to the same CHARACTERISTIC address
                    await client.write_gatt_char(CHARACTERISTIC, value)
                    return True
            except Exception as e:
                return False
        else:
            return False

Now that we have established an abstraction for our smart fan, we can just create an instance of it and start controlling it.

fan = Headwind.Headwind("<bluetooth address>")
fan.writeOn()       # Wake up the fan
fan.readSpeed()     # Read the current fan speed
fan.writeSpeed(100) # Tell the fan to run at 100% speed
fan.writeSleep()    # Send the fan back to sleep

Buttons

But how are we going to interact with this fan? What I need is a way to change its speed without getting off and back on the bike. The answer, to no one’s surprise, is Home Assistant. Initially I looked into writing a proper Home Assistant integration that would expose the fan as a device with various entities that can be probed and controlled. The process of creating new Home Assistant integrations, based on my limited quick research, is however somewhat convoluted. As someone who comes from a programming background it seems counter intuitive and I will be exploring it at a later date. For now, the fallback plan will be to implement a simple REST API in Flask that we can then expose to Home Assistant via its rest_command interface.

Flask

Flask is a fantastic framework for creating REST APIs and more, very quickly and simply. Here I have prepared a snippet of what we need in order to create an endpoint that will increase fan speed by 25%. This endpoint will be called each time I “press a button” in Home Assistant.

@app.route("/speed/increase", methods=["POST"])
async def setIncreaseSpeed():
    # First we read the current speed of the fan
    currentSpeed = await fan.readSpeed()
    newSpeed = 0
    # We define the new desired speed, depending on the current speed
    if currentSpeed < 25:
        newSpeed = 25
    elif currentSpeed < 50:
        newSpeed = 50
    elif currentSpeed < 75:
        newSpeed = 75
    elif currentSpeed < 100:
        newSpeed = 100
    # Lastly we send the fan our new speed setting
    fan_status = await fan.writeSpeed(newSpeed)
    if not fan_status:
        return f"Failed to increase speed by 25%", 503
    return f"Increased speed by 25%", 200

This is one of the endpoints I created, to see the others check out the views.py module in my repository: https://teapot.octopusx.de/accidentallycompetent/bluewind/src/branch/main/bluewind/views.py

Now that we have the endpoints implemented, we need to start our bluewind Flask server, so that it is ready to receive the REST API calls:

# If you haven't done so already, roll out the virtual env and configure bluewind via environment variables
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
pip3 install -r requirements.txt
export FLASK_ADDRESS='<Headwind bluetooth address>'
# Then start the server
python3 main.py

Home Assistant

Home Assistant does not currently support, to the best of my knowledge, a way to configure rest_command integrations via the web UI, so we need to hop into our configuration.yaml file and add a block much like this one:

rest_command:
  Headwind_on:
    url: "http://192.168.0.111:5000/on"
    method: "POST"
  Headwind_sleep:
    url: "http://192.168.0.111:5000/sleep"
    method: "POST"
  Headwind_speedup:
    url: "http://192.168.0.111:5000/speed/increase"
    method: "POST"
  Headwind_speeddown:
    url: "http://192.168.0.111:5000/speed/decrease"
    method: "POST"

This way we define a number of endpoints that can be called via the rest_command service. In our case the commands will be Headwind_on, Headwind_sleep, Headwind_speedup, and Headwind_speeddown.

In my Home Assistant setup, my Wahoo smart trainer is powered from an ikea tradfri smart socket, which I can also control from a tradfri button press. I can now easily attach the Headwind_on onto the state of the switch.ikea_socket_office_2 entity, turning the fan on whenever the turbo gets turned on, and the opposite with another automation working in the reverse.

alias: Headwind on
description: ""
trigger:
  - platform: state
    entity_id:
      - switch.ikea_socket_office_2
    to: "on"
    from: "off"
condition: []
action:
  - service: rest_command.Headwind_on
    data: {}
mode: single

For my remote control of both my turbo and the Headwind fan I am using a plain old Ikea Tradfri button that is linked with Home Assistant via a Zigbee2MQTT bridge. It is a wireless, coin cell battery powered button with 5 discrete physical buttons, The big middle one, as described above, toggles on and off the Wahoo Kickr Core indoor trainer socket power and in turn the fan. The top and bottom physical buttons, normally used for adjusting light brightness levels in an Ikea smart lighting system, will be repurposed as our speed changing buttons. Each press of the up or down button will increase or decrease the fan speed by 25% respectively.

Here is the yaml code defining my this automation:

alias: Headwind speed up
description: ""
trigger:
  - platform: device
    domain: mqtt
    device_id: a6a84a4ded13d8a3850a4c323ec737e4
    type: action
    subtype: brightness_up_click
    discovery_id: 0xccccccfffe5f0bd7 action_brightness_up_click
condition: []
action:
  - service: rest_command.Headwind_speedup
    data: {}
mode: single

With this setup, I am able to turn all of my indoor riding gear on and of with a single press of a button, and I am able to adjust the fan speed to my liking remotely from my bicycle while riding. In the end I deployed my bluewind API server on a Raspberry Pi 4 that lives permanently in my training room, but this software will work on any linux machine with access to a bluetooth receiver that is within signal range of your Wahoo Headwind.

As ever, you can find all of the project files available on my Teapot Gitea instance: https://teapot.octopusx.de/accidentallycompetent/bluewind

Future Plans

As with every project, posting an article on my blog usually means that it is about half way finished. Yes, the basic functionality is here and I can use my new automation already, but there are a few things missing for me to call it good and done.

For one, I would like to decipher the rest of the byte codes returned by the BLE characteristics of the Headwind fan. I would like to be able to read what mode the fan is in and select different modes, not just operate the manual speed dial remotely.

Secondly, my current solution omits the issue of scanning and finding the MAC address of your bluetooth fan entirely. This means you need to first scan the aether with some other BLE scanner app in order to find the MAC address and enter it into bluewind’s config via an environment variable. Something I would love to be able to do, is to automatically check if whatever device you run the server on is BLE capable or not. If the former, then scan for local Headwind devices and automatically start communicating with the first one it finds, in case a MAC address is not passed as configuration.

Lastly, once the above two points are fully addressed, I would like to create a proper Home Assistant integration for bluewind. One that reduces the barrier to entry for all users and is much easier to set up for anyone.

I hope you’ve learned something from this blog entry, I certainly learned a lot about the BLE and bluetooth stacks, and I wish you the best in your future training and coding endeavors! Please check in again at a later date to see what I am working on, I do my best to mix and match topics to keep it fresh and interesting!

Starting point

The starting point is our hand-written set of kubernetes templates that we wrapped into a helm chart to deploy in our kubes and gpus tutorial.

# Source: refact/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: refact
  labels:
    helm.sh/chart: refact-0.1.0
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - port: 8008
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
---
# Source: refact/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: refact
  labels:
    helm.sh/chart: refact-0.1.0
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
spec:
  serviceName: refact
  revisionHistoryLimit: 3
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: refact
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: refact
        app.kubernetes.io/instance: release-name
    spec:
      hostNetwork: true
      runtimeClassName: nvidia # IMPORTANT!
      dnsPolicy: ClusterFirstWithHostNet
      enableServiceLinks: true
      containers:
        - name: refact
          image: "smallcloud/refact_self_hosting:latest"
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: false
          env:
            - name: "TZ"
              value: "UTC+02:00"
          ports:
            - name: http
              containerPort: 8008
              protocol: TCP
          volumeMounts:
            - name: data
              mountPath: /perm_storage
      affinity: # This is how we tell it to only spawn on the node with our GPU
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: gpu
                operator: Exists
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: refact
---
# Source: refact/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: refact
  labels:
    helm.sh/chart: refact-0.1.0
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
  annotations:
    kubernetes.io/ingress.class: # insert your own ingress class to use
    # insert any other annotations you need for your ingress controller
spec:
  tls:
    - hosts:
        - refact.example.com
      secretName: "wildcard-cert"
  rules:
    - host: refact.example.com
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: refact
                port:
                  number: 8008

The new base

So, to host my helm charts based on the common library chart I made a new repository: https://teapot.octopusx.de/octocloudlab/chart-catalog. Nothing special here, basically just a charts folder where all of our, yes you guessed it, charts will live. So we create a folder in that folder, and it is called refact.

.
├── ATTRIBUTION.md
├── charts
│   └── refact
│       ├── Chart.lock
│       ├── charts
│       │   └── common-4.5.2.tgz
│       ├── Chart.yaml
│       ├── README.md
│       ├── templates
│       │   ├── common.yaml
│       │   └── NOTES.txt
│       └── values.yaml
├── LICENSE
└── README.md

What I normally did in the past when I was including a downstream chart I would add this to my Chart.yaml:

dependencies:
  - name: common
    version: 4.5.2
    repository: https://git.octopusx.de/api/packages/octopusx/helm

Then I would add a common (in this case) block in my values.yaml file, and all of the configs related to this dependency would live inside:

common:
  image:
    repository: smallcloud/refact_self_hosting

Inspecting the original charts in the k8s-at-home repo however shows us that they aren’t doing that. All of the parent chart values are defined in the top scope. How do they do that? The answer is in that extra templates/common.yaml file, let’s take a look.

{{ include "common.all" . }}

This simple template basically declares that it will render the common.all template declared in the common child chart, which itself consumes the .Values scope and we can skip the extra scope from the import. Neat.

# common: <- no longer needed
image:
  repository: smallcloud/refact_self_hosting

Publishing the chart

When I was starting work on this part I expected complications. Mostly because I am not very familiar with the actions CI/CD model and I wasn’t sure what I need to do to get it to deploy every chart in the charts directory, whenever there is a change. Aaaaaand… it wasn’t a problem at all. The same action I used in the Teapot blog post can be passed list of directories as a wildcard, and it will iterate over them and run the release flow on them all.

name: Publishing Charts
run-name: Publishing Helm Charts to the Teapot Registry
on:
  workflow_dispatch:
  push:
    branches:
      - main
    paths:
      - "charts/**" # So we updated the trigger slightly
jobs:
  Explore-Gitea-Actions:
    runs-on: ubuntu-22.04:docker://node:18-bullseye
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - name: Push Helm Chart to Gitea Registry
        uses: bsord/helm-push@v4.1.0
        with:
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PUBLIC_PACKAGE_WRITE }}
          registry-url: 'https://teapot.octopusx.de/api/packages/octocloudlab/helm'
          force: true
          chart-folder: charts/* # And another change in this line, and that's it

Achievement unlocked!

Now, we can use the standardized values.yaml as long as we are using the common library chart. The resulting set of kubernetes templates is functionally identical, with a few extra bits that I hadn’t bothered to include before, such as a full complement of labels across all the objects as well as status probes on in the deployment.

/chart-catalog/c/refact main ❯ helm template .
---
# Source: refact/templates/common.yaml
apiVersion: v1
kind: Service
metadata:
  name: release-name-refact
  labels:
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: refact
    app.kubernetes.io/version: v1.1.0
    helm.sh/chart: refact-0.1.0
  annotations:
spec:
  type: ClusterIP
  ports:
  - port: 8008
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
---
# Source: refact/templates/common.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: release-name-refact
  labels:
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: refact
    app.kubernetes.io/version: v1.1.0
    helm.sh/chart: refact-0.1.0
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: refact
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: refact
        app.kubernetes.io/instance: release-name
    spec:
      serviceAccountName: default
      automountServiceAccountToken: true
      runtimeClassName: nvidia
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      containers:
        - name: release-name-refact
          image: "smallcloud/refact_self_hosting:latest"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8008
              protocol: TCP
          livenessProbe:
            tcpSocket:
              port: 8008
            initialDelaySeconds: 0
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          readinessProbe:
            tcpSocket:
              port: 8008
            initialDelaySeconds: 0
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          startupProbe:
            tcpSocket:
              port: 8008
            initialDelaySeconds: 0
            failureThreshold: 30
            timeoutSeconds: 1
            periodSeconds: 5
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: gpu
                operator: Exists
---
# Source: refact/templates/common.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: release-name-refact
  labels:
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: refact
    app.kubernetes.io/version: v1.1.0
    helm.sh/chart: refact-0.1.0
  annotations:
    kubernetes.io/ingress.class: traefik-210
    traefik.ingress.kubernetes.io/router.entrypoints: websecure,web
    traefik.ingress.kubernetes.io/router.middlewares: default-redirect-https@kubernetescrd
spec:
  tls:
    - hosts:
        - "refact.octopusx.de"
      secretName: "wildcard-cert"
  rules:
    - host: "refact.octopusx.de"
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: release-name-refact
                port:
                  number: 8008

If you want to use my new Refact chart in your own Kubernetes cluster, feel free to add my new Teapot repository like so:

helm repo add teapot https://teapot.octopusx.de/api/packages/octocloudlab/helm
helm repo update

Then install it with:

helm install refact teapot/refact --values <your own values file>.yaml

Enjoy!

When I first started my homelab journey and decided to go the kubernetes route, the k8s-at-home project was not only a source of quality helm charts for me to deploy but also an inspiration for how to write reusable charts and manage my own helm chart repositories. Sadly the project has been archived since then as the main maintainer had struggled to keep up with the updates without sufficient help from the community. At the time I either didn’t have the time or the skills to support the project. Today, a few years later, I feel like I may finally have what it takes to bring back at least a small part of the project alive, starting from their library chart. In this blog post I will walk you through how I’ve set up a public Gitea instance to be used as a helm chart registry and git repository for myself and anyone else who wishes to use the charts.

Before we go further, a quick disclaimer that, for all intents and purposes, https://truecharts.org/ has taken the place that k8s-at-home used to have. It is a fantastic resource for helm charts not just to use with TrueNas Scale but on any homelab k8s cluster. I highly recommend checking it out. That is not however what this blog is about, we’re going to explore the fun but also the complex side of self hosting. So let’s crack on!

Primary Objective

I am still running a lot of helm charts from the k8s-at-home public repository (https://github.com/k8s-at-home/charts) and all of them are built based on their library chart (https://github.com/k8s-at-home/library-charts). Now, so far this hasn’t been a problem as K8S releases haven’t deprecated many APIs lately, but it is only a matter of time before a change will break compatibility with the library chart. As the project is no longer in development, we will host the library chart ourselves and build a simple pipeline to publish it to our own registry. This way we will be able to update the base library chart when needed and port the k8s-at-home charts on our new updated version.

For this we will:

• Deploy a dedicated instance of Gitea, and call it Teapot!
• Deploy Gitea runners and connect them to the Gitea instance.
• Clone and sanitize the library chart, then host it in our new Gitea instance.
• Write a Gitea action to build and release the library chart.
• Build and push the library chart to Gitea’s built in registry.

Gitea Deployment

We will be deploying Gitea on our usual K8S cluster, as always. We will use the official Gitea helm chart as a base. Their chart is very robust and highly customizable. All we need to do is import it into our deployment chart and run helm install on it.

Chart.yaml:

apiVersion: v2
name: teapot
description: A public instance of Gitea, notably a Teapot.
type: application
version: 0.1.0
appVersion: "1.20.5"
maintainers:
  - name: Tomasz Fratczak
    email: sre@octopusx.de
dependencies:
  - name: gitea
    version: 9.5.1
    repository: https://dl.gitea.io/charts/

Here is my values file. I disabled all of the unnecessary bells and whistles, configured ingress as well as the SSH load balancer service (for SSH traffic ingress), storage, disabled built in databases and pointed it at my existing postgres instance.

values.yaml:

gitea:
  image:
    repository: gitea/gitea
    tag: 1.20.5
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: traefik-210-external ## Class of dedicated Traefik ingress controller for external connectivity
      traefik.ingress.kubernetes.io/router.entrypoints: websecure,web
      traefik.ingress.kubernetes.io/router.middlewares: default-redirect-https@kubernetescrd
    hosts:
    - host: teapot.octopusx.de
      paths:
        - path: /
          pathType: Prefix
    tls:
      - secretName: <tls-secret-name>
        hosts:
          - teapot.octopusx.de
  service:
    ssh:
      type: LoadBalancer
  persistence:
    enabled: true
    size: 30Gi
    accessModes:
      - ReadWriteOnce
  redis-cluster:
    enabled: false
  postgresql-ha:
    enabled: false
  gitea:
    config:
      actions:
        ENABLED: true
      server:
        START_SSH_SERVER: true
        ROOT_URL: https://teapot.octopusx.de
      service:
        DISABLE_REGISTRATION: true
      cache:
        ENABLED: false
      session:
        PROVIDER: db
      database:
        DB_TYPE: postgres
        HOST: <postgres-url>:<postgres-port>
        NAME: teapot
        USER: teapot
        PASSWD: "<PASSWORD>"
        SCHEMA: ""

For the database connection to be established I had pre-created a teapot database and user in my postges instance, like so:

CREATE USER teapot
      LOGIN
      ENCRYPTED PASSWORD '<password>';
	  
CREATE DATABASE teapot
      ENCODING 'UTF8'
      LC_COLLATE='C'
      LC_CTYPE='C'
      template=template0
      OWNER teapot;

Armed with the database credentials, we can now deploy the chart:

helm install teapot . --set gitea.gitea.postgresql.password=<PASSWORD> --set gitea.gitea.admin.password=<PASSWORD>

With the following result:

We can now log into our new instance, in my case under https://teapot.octopusx.de, as gitea_admin with the password we just set and start messing around.

Gitea Runner

Now that we have our Gitea instance up and running, we need to create a runner to be able to deploy the library chart.

The Gitea act runner is easiest deployed as a container. I won’t get into too much detail on how you can confiture it as Gitea’s documentation is very detailed already: https://docs.gitea.com/next/usage/actions/act-runner. What I will point to in addition though is the examples folder in the gitea act runner repo, to be found here: https://gitea.com/gitea/act_runner/src/branch/main/examples. Having read this I have realised that it basically needs to orchestrate a docker daemon, docker-in-docker style. This is generally not a good idea inside of a Kubernetes cluster, for practical as well as security reasons. I have therefore opted to run this workload in its own LXC container on one of my proxmox servers instead. I installed docker inside of the of my LXC and ran the following command:

docker run \
    -v $PWD/config.yaml:/config.yaml \
    -v $PWD/data:/data \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e CONFIG_FILE=/config.yaml \
    -e GITEA_INSTANCE_URL=https://teapot.octopusx.de:443 \
    -e GITEA_RUNNER_REGISTRATION_TOKEN=<get your own token lol> \
    -e GITEA_RUNNER_NAME=act01 \
    -e GITEA_RUNNER_LABELS="ubuntu-22.04:docker://node:18-bullseye" \
    --name act_runner \
    -d gitea/act_runner:nightly

If you’re wandering how to get the registration token from your Gitea server, I recommend reading the quick start guide: https://docs.gitea.com/next/usage/actions/quickstart.

And here we go, a newly registered runner, fresh out of the LXC and still warm:

Library Chart Repository

I don’t think I need to tell you how to create a new repository in Gitea do I? To make scoping and access rights management easier in the future I created an organization first, called it octocloudlab. It is hard to find good names that aren’t already copy righted, so I thought this one is just nonsense enough to get the job done. I then created a new repository called library-chart under this new organization. I did not want to just create a fork of k8s-at-home library-charts repository, as that was indeed a complex project that would be somewhat hard for one person to wrap their head around. The original repository was shared under the Apache 2.0 license, so I started by creating a README.md with the appropriate attributions to the original project contributors (which you can find here). Then I tool just the stable common chart from here: https://github.com/k8s-at-home/library-charts/tree/main/charts/stable/common to use as the blueprint for my release. BTW, this is the first time I am basing a project on someone else’s work, if you think I could have done anything better when it comes to attribution or other license-related issues, please let me know on the socials.

Before I committed it to my new repository I made sure to update all of the metadata, removing any references to the original contributors and documentation from the code itself, in order to make sure there are no misunderstandings as to who the maintainer of this repository is and where one should go to seek help if necessary.

To update all of the auto-generated Helm documentation I then ran:

helm-docs generate -t README_CHANGELOG.md.gotpl -t README_CONFIG.md.gotpl -t README.md.gotpl

from inside the chart folder, which produced the updated README.md.

The final result is a clean and simple, publicly available repository with just the necessary basics to allow me to maintain and update the common library chart for my own benefits.

Should you wish to explore it, feel free to visit https://teapot.octopusx.de/octocloudlab/library-chart/.

Gitea Action

Gitea’s actions system is drop-in compatible with Github actions. Even better, by default they use the Github actions repository and you can use use the same workflows and ready-made actions that you normally see on Github. For me this is a fantastic opportunity to try out and learn Github actions, which my company uses at work but I had no opportunity to play with. So, the first thing I decided to do is run a test demo workflow to make sure that my runners are operating correctly. I found this demo example on Gitea’s blog here https://blog.gitea.com/feature-preview-gitea-actions/:

name: Gitea Actions Demo
run-name: ${{ github.actor }} is testing out Gitea Actions 🚀
on: [push]
jobs:
  Explore-Gitea-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by Gitea!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
                    ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."

I added this to .gitea/workflows/demo.yaml in my library-chart repo and… nothing. The action was waiting forever. This is the point where I have learned that actions and act runners need to have matching labels. In this case, the

runs-on: ubuntu-latest

has to match the label ubuntu-22.04:docker://node:18-bullseye on my action runner in order for it to pick up the action. The interesting and unusual thing about this system is that this label (or tag) not only identifies the runner but also provides the specification of what the runner is capable of. In this case:

ubuntu-22.04 : docker       : // node:18-bullseye
Runner OS    : runner type  : docker image used

So, once we fixed that and labeled the job and the runner correctly, it triggered on the next push and executed no problem. Let’s move onto the actual workflow that we want to build though. The strength of the actions model is that they are reusable and you can find a multitude of pre-configured workflows to use in your own pipelines straight on Github’s “marketplace”: https://github.com/marketplace?type=actions. What I tried to do was:

• check out the project
• package up the helm chart
• push the new package to the gitea registry

To check out the repo we can use the same action as the demo action, i.e. actions/checkout@v4. I then tried using plain run commands that allow you to execute arbitrary bash code, but my action was slowly becoming pretty complex. This is because I would have to:

• install helm
• install helm plugin cm-push
• add my gitea registry as helm repository
• package the chart
• publish the package to the helm repository

I started searching for a ready made Github action, thinking that someone must have needed this functionality at some point, and yeah, I found one pretty quickly. The bsord/helm-push@v4.1.0 does all of the above in a single step with minimal configuration needed. It had some problems itself, as for some reason all of the documentation for it was listing a non-existent 4.2.0 tag, and without the v, so it took me a while to figure out why the heck it’s failing. Finally though, I made it:

name: Publishing Chart
run-name: Publishing Helm Library Chart to the Teapot Registry
on:
  workflow_dispatch:
  push:
    branches:
    - main
    paths:
    - 'chart/**'
jobs:
  Explore-Gitea-Actions:
    runs-on: ubuntu-22.04:docker://node:18-bullseye
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - name: Push Helm Chart to Gitea Registry
        uses: bsord/helm-push@v4.1.0
        with:
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PUBLIC_PACKAGE_WRITE }}
          registry-url: 'https://teapot.octopusx.de/api/packages/octocloudlab/helm'
          force: true
          chart-folder: chart

This is basically all I needed to get this basic workflow running. Now, this workflow will be triggered each time a change to the chart folder in the main branch is made. In order to be able to download and use this library chart you can now do the following:

helm repo add octocloudlab https://teapot.octopusx.de/api/packages/octocloudlab/helm
helm repo update

And to include it in your helm chart add the following to your dependencies section in the Chart.yaml

## Chart.yaml
dependencies:
- name: common
  version: 4.5.2
  repository: https://teapot.octopusx.de/api/packages/octocloudlab/helm

What now?

Now that I have this library chart published and a way to modify it and publish patches reliably, I will slowly replace the current helm chart that I am using in my infrastructure with a version of them that is based on this library chart. Since at this point I haven’t made any changes to the k8s-at-home library chart this should be a drop in replacement. Then, going forward, I will start adapting the library chart to my needs, adding capabilities that I am missing in my homelab and trying to patch it for newer versions of kubernetes API.

The first victim on the chopping block however will be the chart I pieced together to deploy refact.ai, about which you can read in my previous blog entry. As far as I know there is no official helm chart for that project, so it will be interesting to publish a nice clean helm chart for them and see if anyone, besides me, actually ends up using it.

The Hardware

What GPU you should buy to get the most out of it depends on the models that you want to run on it. There do exist smaller coding-focused LLMs that can be run on a GTX 1060 6GB or an 8GB GTX 1070, which can be sniped on the second hand market for under a 100 euros. AMD has historically sold cards with higher VRAM sizes which would make their second hand units the best bargain, sadly it seems that their ROCm software platform for running AI compute tasks is still in its early days and not many, if any, consumer cards are supported at the moment.

Some of the available models and their estimated VRAM requirements:

I was actually about to pick up a second hand GTX 1070 with 8GB of VRAM for 90 euros when I found a good deal on a refurbished RTX 3060 12GB model and ordered immediately for 270 euros delivered. In terms of performance per euro the 1070 was a much better choice, however the 3060 has 3 things going for it:

• lower power consumption (lol nope)
• smaller physical footprint (I am trying to cram this into a mITX case)
• extra 4GB of VRAM (more VRAM more better models)

Fun fact, now that I already have the card and started writing this article, I realized that actually the 3060 has a higher peak power and similar idle numbers to the 1070… Oh well… According to techPowerUp anyway: 1070: 13W idle, 148 peak, 3060: 13W idle, 179 peak (https://www.techpowerup.com/review/evga-geforce-rtx-3060-xc/36.html).

A lot of models that I want to try out exceed the 8GB mark once loaded into RAM, notably starcoder. I was also curious to figure out what it would take to set up CUDA time sharing on a single GPU, where multiple smaller models are loaded into VRAM simultaneously. It is a shame that AMD is not an option, as they have a number of 16GB VRAM cards at a similar price to this one (RX6700XT for example), while for Nvidia we would have to jump up to an RTX 4060 16GB, which retails at around 500 euros.

The Platform

You know what you’re getting yourself into coming here right? It’s all about Kubernetes xD One of the Proxmox nodes in my server rack has a spare X16 slot that will happily house our refurb 3060, and we can pass it through to one of the Kubernetes node VMs. To do this we need to follow a few steps:

Prepare Proxmox

1. Drain the node to be shut down

kubectl drain k8s12 --delete-emptydir-data --ignore-daemonsets

2. Edit the hardware definition of the node (add the new PCI device)

3. Shut down the node so that it may pick up the hardware change

4. (optional) Make a snapshot/backup of the VM, in case we screw something up later

5. Boot it up again

Voila, the GPU should be now available inside the VM. We can check that by running:

~ ❯ lspci                                                                             octopusx@k8s12
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
00:05.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
00:10.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
00:11.0 Audio device: NVIDIA Corporation Device 228e (rev a1)
00:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge

6. Uncordon the K8S node to re-enable scheduling on it

kubectl uncordon k8s12

Fun fact

As much as the above steps are all that I “technically” had to do should my proxmox server be correctly configured from the get go, I was getting weird behavior at the start. The graphics card was passed into the VM the first time it booted, but the driver (I installed next) would tell me there is no GPU available, even after rebooting the VM. I started messing around, thinking it is maybe some weird property of PCIe passthrough on Nvidia hardware, went on to blacklist the nvidia kernel module/drivers inside the Proxmox host to make sure it doesn’t get initialized by the host, which in turn prevented the VM I would pass it through from booting altogether. In the end I jumped into the bios of the machine to make sure all of the PCIe-related settings are in order and… of course I left the bifurcation settings on the x16 slot in the x8/x8 mode from when I was using multiple NVME drives in that slot. Changing that back and reverting all of the trial and error driver shenanigans on the host got it to work, boot and be available inside the VM every time.

Install Nvidia Drivers

Now off to install all of that dirty proprietary Nvidia stuff. We only just gave them our money, so we’ve got nothing to loose by going all the way and also giving them our souls.

Nvidia has a pretty solid guide with versions for various popular server linux flavors:

• https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

Which amounts to installing the CUDA toolkit and drivers.

1. Install the CUDA toolkit

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-3

2. Install Nvidia drivers

sudo apt-get install -y nvidia-kernel-open-545
sudo apt-get install -y cuda-drivers-545

3. Install the Nvidia container runtime

The last step that we need to do to prepare our Ubuntu VM to run GPU accelerated tasks inside OCI containers is to install the Nvidia container runtime. This is a container runtime that is compatible with the OCI specification and is able to run GPU accelerated containers. It is also compatible with the Kubernetes CRI interface, which is what K3S uses to run containers. Once installed, K3S will automatically detect it once its service is restarted. You can find out the setup details (which are very simple) on the following documentation page: https://docs.k3s.io/advanced#configuring-containerd.

Since we are running K3S on Ubuntu we are using systemd-containerd and installing everything from an apt repo. Therefore we need to follow Nvidia’s instructions for enabling their container runtime for containerd, as found here: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-containerd. In my case that would be:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
  && \
    sudo apt update
    sudo apt install -y nvidia-container-toolkit

Once this is done you can run an Nvidia CUDA container to see that the runtime is functioning:

~ ❯ sudo ctr image pull docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04
docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:0b165c469e8b0a620ce6b22373ead52502ef06d3088ba35a6edb78582b5274f6:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1f0416abdf40fca3a4ce4e42093584664b4ac0dddd012571453c94e5c7a35937: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:f5861cf44882f74454e1a5915647321b5b344af925503515bcd8bb0728d84551:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4b46bd5b7766c7415446d5b866f709b45dbdb60b04fec7677343d8232ca2e427:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f038740a8e0591897bdd6280d24e1f89690cf80c28e77c14072f84eca205c7e6:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:b2dca9da14b18f8a959e7deed652de532f88f979d8d1279511b987e294699f3c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:998220aec9f5708b331daeb126f9d66fe39bbcc333224ca9ba6efabc9b563b7d:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 7.0 s                                                                    total:  83.6 M (11.9 MiB/s)                                      
unpacking linux/amd64 sha256:0b165c469e8b0a620ce6b22373ead52502ef06d3088ba35a6edb78582b5274f6...
done: 1.00494491s

~ ❯ sudo ctr run --rm --gpus 0 -t docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 12.2.2-base-ubuntu22.04 nvidia-smi
Thu Oct 26 07:30:54 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06              Driver Version: 545.23.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060        On  | 00000000:00:10.0 Off |                  N/A |
|  0%   48C    P2              46W / 170W |  11651MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

And also, once you’ve restarted your K3S agent service, check that it also picked up the change:

~ ❯ sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml              octopusx@k8s12
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

The Deployment

In order for any container to take advantage of the nvidia containerd runtime plugin we need to create a special runtime class object, which can be referenced inside our kubernetes deployment yaml.

It looks like this:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

Since (at least in my case) I am running a single GPU that is attached to a specific K8S worker VM, we need to somehow force the workloads that require access to said GPU to be deployed on that specific node. To do this we create a node label that we can then reference in the pod affinity specification:

kubectl label nodes k8s12 gpu=3060

We can check that it worked by invoking the describe command on the K8S node object:

~/ ❯ kubectl describe node k8s12                                                                                                                       ○ teleport-k3s
Name:               k8s12
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    gpu=3060
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s12
                    kubernetes.io/os=linux
                    kubernetes.io/role=worker
                    node.kubernetes.io/instance-type=k3s

You will see that I created a label with a non-binary value. This is to make this system a little more future-proof on my side. This way I can match either on the label just “existing”, so if I want any GPU present to bind a deployment onto, or to a specific value if I want to assign a workload to a predefined model. Neat.

To make sure that our assignment mechanism works, we can spawn a test GPU benchmark container. If we did everything right this should return us a benchmark score for our GPU:

apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark
  namespace: default
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: cuda-container
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    args: ["nbody", "-gpu", "-benchmark"]
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: NVIDIA_DRIVER_CAPABILITIES
      value: all
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: Exists

RefactAI Chart

Now that we have checked that everything necessary to deploy refact is in place, we deploy it. Here is an example output of helm template . command for the helm chart that I created:

# Source: refact/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: refact
  labels:
    helm.sh/chart: refact-0.1.0
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - port: 8008
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
---
# Source: refact/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: refact
  labels:
    helm.sh/chart: refact-0.1.0
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
spec:
  serviceName: refact
  revisionHistoryLimit: 3
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: refact
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: refact
        app.kubernetes.io/instance: release-name
    spec:
      hostNetwork: true
      runtimeClassName: nvidia # IMPORTANT!
      dnsPolicy: ClusterFirstWithHostNet
      enableServiceLinks: true
      containers:
        - name: refact
          image: "smallcloud/refact_self_hosting:latest"
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: false
          env:
            - name: "TZ"
              value: "UTC+02:00"
          ports:
            - name: http
              containerPort: 8008
              protocol: TCP
          volumeMounts:
            - name: data
              mountPath: /perm_storage
      affinity: # This is how we tell it to only spawn on the node with our GPU
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: gpu
                operator: Exists
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: refact
---
# Source: refact/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: refact
  labels:
    helm.sh/chart: refact-0.1.0
    app.kubernetes.io/name: refact
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
  annotations:
    kubernetes.io/ingress.class: # insert your own ingress class to use
    # insert any other annotations you need for your ingress controller
spec:
  tls:
    - hosts:
        - refact.example.com
      secretName: "wildcard-cert"
  rules:
    - host: refact.example.com
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: refact
                port:
                  number: 8008

For me an extra step here would be to add refact.example.com to my local DNS to expose this ingress entry (I just add it to my PiHole server). If all goes well, you should be able to go to refact.example.com (taking that you have DNS set up for this) you should see a screen much like this:

Here I have loaded 2 small models, one that provides code completion and one that provides chat functionality. Also, I have no idea why I am getting the VRAM warning message. It has been there for me since the beginning no matter what model I select and since everything is working for me I am going to assume it’s a UI bug…

VSCode Plugin

Let’s finally go harness the power of the AI in a practical manner. First, we install the refact.ai plugin:

Then we go to its config page to enter our local instance URL:

And off you go! Refact should now automatically start generating suggestions in your VSCode editor window as you type:

In my case, it will also allow you to chat with the llama7b model:

The End

All in all, I am quite happy with this setup. It was very straight forward and pretty much just worked. I may even sign up for their paid version just to support their project, well done refact.ai! I haven’t used this in practice much so I don’t feel comfortable giving a verdict on how good the small models that I am able to fit into my 3060’s memory are. However what I can say is that the models execute extremely fast on the RTX 3060 12GB gpu… The code generation is very fast and the llama7b models chat is pretty much instant. Much more responsive than my previous attempt at CPU hosted model and downright impressive, with such a relatively small investment in hardware. Something I really wish I was able to do is expose my llama7b model hosted by refact in other apps via an openAI API endpoint. Another would be to be able to host a single larger model that does both code completion as well as chatting. That would make me 120% happy… For the moment, I have browsed the refact’s github issue tracker as well as their discord server (https://www.smallcloud.ai/discord) which seems to be active and will see if I can find out more about the project’s roadmap and if the few functionalities that are missing will be added over time. Hope you enjoyed this tutorial and will come back again when the next entry rolls in.

Ask Llama!

Tell me a joke about large language models

Why did the large language model go to therapy? Because it had a lot of issues with its input data.

TLDR;

I am pretty new to this whole AI thing, so we will split this into a few sections and list what worked for me and what didn’t. K8S deployment process first, the basics of what models are available and which ones worked best for me, a few simple integration and what we are going to cover in the future. I got most of what I wanted to work, but to unlock the full potential I need to try this on a GPU…

K8S Deployment

Ask Llama!

What is Kubernetes?

Kubernetes is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. It helps to manage complex distributed systems and enables developers to focus on writing code instead of managing infrastructure.

The first well described github repository I have found that explains how to easily host openAI compatible API for an LLM was Umbrel’s: https://github.com/getumbrel/llama-gpt. They didn’t just provide examples and docker-compose files, but even a simple set of Kubernetes templates, which was a nice kick start for me. A few minutes of digging through that and we had a ready helm chart based on their templates. They actually built a slick little web UI that mimic chat GPT’s, very handy! What hasn’t worked for me though is their mechanism for downloading various models. To load a model into the containerized environment I first created a k8s volume manually, mounted it to a VS Code web server where I used the terminal to download all the models I wanted to use. The container that provides the openAI API is from the llama-cpp-python binding project found here: https://github.com/abetlen/llama-cpp-python.

The second project I found was definitely more accomplished and well maintained: the localai.io project. Where the Umbrel setup uses python c++ bindings, this one implements its own in golang: https://github.com/go-skynet/LocalAI. It does not provide any form of web UI add on, however, unlike the Umbrel repo, this one includes working helm charts, instructions for K8S deployment and a working model download mechanism! All you have to do is add URLs to a list in helm chart’s values.yaml file and an init container will take care of it. There is even a flag to force re-download any model that already exists on the volume in case that is necessary.

I ended up running both of them in my cluster to test and play with, and will likely continue to do so going forward. Having deployed and briefly used both of the setups I realized quickly that they are fairly slow on my current hardware, which means that if I can have 2 different setups with different models at the same time I can avoid long request queueing. Additionally it is worth noting that if you wish to use these LLM containers with multiple models, each model used will be loaded into memory, so you might end up with a container that consumes upwards of 16 GB of RAM. Going forward I decided to use the Umbrel project to host my chat-focused model and LocalAI.io for my code generation one, which means that I have 2 independent services serving distinct functions and 2 separate pods, each of which will use around 8GB of RAM and can run on a different node in my K8S cluster.

As I mentioned earlier, the llama-cpp-python base configured in Umbrel’s repo doesn’t come with a proper helm chart, however I will make mine public once I find the time to tidy it up and push it to Github. Stay tuned for that if you’re interested!

Models

Ask Llama!

What is a large language model

A large language model is an advanced artificial intelligence system designed to generate human-like text, understand natural language, and perform tasks such as translation, summarization, and question answering by analyzing vast amounts of data.

So, the APIs are not of very much use to anybody if you don’t give them a model to work with. Me being super new to all this, I started my research from the Umbrel’s llama-gpt repo’s readme. Handily, it did contain a nice table summarizing the models compatible with this setup:

What it however does not tell you, at least not explicitly, is what each of these models does or where to get them. I do understand that the B value stands for billions of parameters. The more parameters, the more accurate model, but the more memory and computational resources it needs to run (warning: vast oversimplification). If we dig a little and take a sneak peak at the docker-compose.yaml files in the repo, we will see the following in line 15:

(...)
      MODEL_DOWNLOAD_URL: '${MODEL_DOWNLOAD_URL:-https://huggingface.co/TheBloke/Nous-Hermes-Llama-2-7B-GGML/resolve/main/nous-hermes-llama-2-7b.ggmlv3.q4_0.bin}'
(...)

Aha! This is where they are getting the models from. Short duck duck go later, and we find out that huggingface.co is a fantastic repository for all kinds of AI models, ready to be downloaded for free. I recommend reading the repositories that you can find there, as they describe often in detail what each model does, what it is good at and what requirements it has. The key thing I learned is this:

• Models marked with GGML are an older standard, single-file packaged models that can run on a CPU
• Models marked with GGUF are meant to be a more versatile, improved version of GGML packaging for ML models

There are a lot of different parameters that each model is pre-configured with, but from reading the info in the repos, models marked q4_0 give the best balance of speed and accuracy for a given parameter size (usually).

Yes, this was a quick search and a bunch of trial and error, so if you want to explain this to me or know of a cool resource online that I could read or watch to understand these parameters better, hit me up on mastodon!

From my side, I also have to give a shout out to https://gpt4all.io/, as it also has a lot of useful information about the relative qualities and purpose of each model.

Now, to set out some expectations and context, I am running these LLMs in my K8S cluster on CPUs. My home lab has 3 nodes, each spotting a Ryzen 3000 CPU. This means AVX1 and AVX2 but not AVX512. For the uninitiated, those are vector instruction set extensions which help accelerate some of the AI tasks. On each of these boxes I have a VM with CPU passthrough configured to make sure the workloads see the extensions, and each with 6 virtual CPU cores (6 threads essentially).

Chat Models

In the couple of days that I played with all of these, there were 2 models that worked reasonably well on my machines:

• llama-2-13b-chat.ggmlv3.q4_0
• nous-hermes-llama2-13b.ggmlv3.q4_0

Each of those models are about 8GB in size (i.e. will take 8GB of RAM once loaded up) and will run reasonably happily from within the llama-cpp-python container. Most of the queries I asked these models returned quickly enough to be considered “interactive”.

Pro tip. When you configure llama-cpp-python, make sure to set the n_threads environment variable to the number of threads you want to make available to the model for computing. All of the parameters available can be found in the official documentation.

Coding Models

I had very high hopes for the coding models I have to say, however I found most of them temperamental and tricky to work with. For some reason a lot of them, especially the codellama designated ones, were falling into a loop of repeated output and would generate the same answer over and over before I manually terminated them. I have not found a definitive answer online what causes this behavior and was not able to make them work with the VSCode integrations that I used. This left me with the one model that has been the most consistently working as a coding assistant:

• ggml-model-gpt4all-falcon-q4_0

This model worked fine in the VSCode integrations that I tried with it, though bear in mind, it is not an instruct model. It is good if you need it to explain code for you or suggest code via a chat-like interaction. It will not however work for straight up code generation inside your editor. OpenAI API is apparently not designed for that and you need a dedicated instruct model and an instruct compatible API endpoint. And self hosted versions of such software and models require the use of a GPU which I currently do not have! We will have to explore this avenue at another occasion then… More details on code generation in the integrations section.

Integrations

Umbrel Llama UI

Ok, so we have ourselves a web API deployed, we downloaded our models. It is finally time to use this stuff! The easiest way to start in my opinion is the Umbrel Llama UI that we can get from here. This creates a ChatGPT-like web UI for you that you can use to create threads of conversations with different basic parameters, such as the system prompt and temperature.

Just as an FYI:

• Temperature: setting on a scale of 0.0 to 1.0, that governs how factual and to the point answers will the model try to give you (0.0) vs how creative you want it to be (1.0), where setting 0.5 could be described as neutral.
• System prompt: information you can feed the model that sets the tone and the desired outcome of the conversation you are trying to have with the model. It can be things like: “You are a helpful AI and will answer concisely” or “We are going to be role playing where I am Mario and you are princess Peach”, you get the idea… This is great for general chat usage but also to test that your models are working and performing.

Nextcloud Assistant

Ask Llama!

What is Nextcloud?

Nextcloud is a open-source, self-hosted file synchronization and sharing platform that allows users to access their files from multiple devices, share files with others, and collaborate on projects.

Nextcloud has recently been making noise around their brand spanking new OpenAI integration being made available to use with self hosted LLMs. Being able to try this out was a fairly large motivator for me. Partly to see actually how easy it was to set up and partly because I found the features advertised, if not useful, then at least fun to try. So, once you have your own API endpoint ready, you can go to the administrator settings panel in your Nextcloud instance and enter it somewhere there… And by somewhere I mean somewhere, because for some reason it is in the Connected Accounts tab, and not the Artificial Intelligence tab, but what do I know. So, I enter it and… I get a green tick in the top right corner saying admin setting saved. That’s good right?

Well, all of the AI-related functionalities fail… And they fail after a long long time, which indicates some sort of a timeout issue maybe? I browsed through the logs and all I find this this…

OpenAI API error : Host violates local access rules

Which tells me nothing, and duck duck going it ain’t helping much either.

A couple of days pass and I find this beautiful issue on the Github issue tracker for the Nextcloud OpenAI integration: https://github.com/nextcloud/integration_openai/issues/47

Well, I had no idea I am supposed to be able to select the model there. The UI does not show that as an option, not even grayed out. As you see I chimed into the conversation, reporting similar behavior. Low and behold, I get a quick response from a dev and my issue is resolved!

So, now I have two key takeaways from this interaction. One is, that the devs clearly know what they’re doing and they were super quick to help me and fix my issue. Two, however, is a big problem in that I needed this help in the first place!

Basically, in order for Nextcloud to allow you to talk to a locally hosted LLM you need to change this setting:

'allow_local_remote_servers'=>true

in the main PHP config file for Nextcloud. But (at least at the time of writing) this is not said anywhere on the plugin install page or in the documentation as far as I am aware. And it seems that this flag is set to false by default. Oh dear…

So, with that out of the way, now we know that you have set things up correctly once the model field magically reveals itself to us in the config UI.

We can get to work now! Let’s generate some text!

I press that magical sparkly “Free Prompt” button and… Poof! Nothing XD To cut to the chase here and not drag this out much longer, it seems that Nextcloud is trying to execute most of the calls to your local LLM endpoint asynchronously. Which is a good choice in my opinion. The problem for me however is that I am running Nextcloud in Kubernetes, and to facilitate job processing I had set up a web cron container, that basically gets spawned twice per second to pick up any currently queued jobs. This includes but is not limited to the OpenAI API calls. This makes it fairly unpredictable as to when your jobs will be processed, and though largely fine for strictly speaking background jobs, I was expecting my AI queries to complete faster than within an hour or two… The only call that I found so far to be computed synchronously (as in, while you wait) is the smart picker ChatGPT-like text generation (with LocalAI) option that I triggered inside a note that I was editing. There you get a “preview” button, which will lock the prompt popup until you get your result from the LLM API.

Sadly my experiences with the ready made prompts that run async are also rather bad, and your milage may also vary depending on which LLM you use. With llama-2-13b-chat for example I generally get gems like this:

As far as I am aware there is no way to adjust the wording of the queries that is used behind these “summarize/generate headline/reformulate” buttons, so whether they will actually result in a useful output is up to luck. This is obviously only based on a few days of using this model and integration, so maybe I am just missing something…

In summary, with my current setup the local OpenAI API integration with Nextcloud is not very useful or practical. I feel like it lacks basic functionality such as the inability to set model temperature or system prompt, as well as being able to choose if a command is executed by the LLM synchronously or asynchronously. I am relatively positive that Nextcloud will iron out this integration and make it more configurable over time so I will happily revisit it at a later date. Though I really wish that they would come up with some other way to run jobs for Kubernetes deployments altogether, other than web cron…

Visual Studio Code

Ask Llama!

What is Visual Studio Code?

Visual Studio Code is a free, cross-platform source-code editor that runs on Windows, Linux, and macOS. It includes support for debugging, task running, and Git control, among other features. It is popular among developers for its speed, flexibility, and extensibility.

Now we are getting to where it’s at for productivity tasks! At least for my use case. AI assisted programming was something that I was already familiar with by the virtue of using the CodeGPT plugin for VSCode. I wouldn’t call myself a power user but I certainly found it useful at more than one occasion and wanted to find a self-hosted alternative that will respect my data and not charge me a subscription!

In the end I found and tested two extensions which can be used with a local OpenAI API endpoint, Wingman and Continue.

Wingman

Wingman is the first VSCode extension that I found and I really like the project. What it essentially is, is a set of pre-configured but user customizable queries that you can ask your LLM, with a mechanism to automatically paste the responses into the file you’re editing.

You can select actions of various categories, such as completion, testing, refactoring or just chat. Where I think the real power comes from here is the customizability. You can add your own prompts in the json config file, here’s the example it comes with:

    "wingman.userCommands": [
        {
            "command": "exImprove",
            "label": "Suggest improvements",
            "userMessageTemplate": "I have the following {{language}} code:\n```{{filetype}}\n{{text_selection}}\n```\nPlease provide a list of suggestions for improving the code in markdown format. Consider aspects such as readability, performance, maintainability, and best coding practices. Explain each suggestion briefly, highlighting the specific areas that could be enhanced. Your suggestions should aim to improve the code without changing its behavior.",
            "systemMessageTemplate": "You are an assistant to a {{language}} developer.",
            "callbackType": "none",
            "category": "My prompts"
        }
    ]

This article is sadly not going to cover how well most of the features work. This is because most of the more interesting commands that I ran via Wingman (or Continue for that matter), resulted in my server CPUs spinning away and nothing else.

I was still able to do simple things, such as asking the LLM to generate comments for simple code blocks, or use the built in chat to ask programming-specific questions.

Continue

Continue is another alternative which is very close in functionality to Wingman, with a few subtle differences. For a start, Continue is more keyboard driven. You’re not selecting query types from a menu but rather type / followed by the type of command you want to run. This also includes code editing, generating tests, comments, etc.

You can also configure multiple backends simultaneously and switch between them from the UI, which is very useful if you want to split your use between a local and a public cloud API or just when you’re testing multiple models, maybe specific for different programming languages…

Also, where Wingman basically overrides the code you pasted into it when you ask it to modify it, Continue creates you a visual diff, which you can either accept, reject or ask for adjustments to. Very neat, and which one you prefer depends on your style of working with code and coding editors.

With Continue you can also simply add code to the context of specific queries by selecting it and pressing ctrl+m. A visual queue will appear that the selected block is attached and you can remove it at any point. All of your chat history and context is also logged for you to browse through later which I definitely like too.

The models you want to use can be easily configured via modifying a config.py residing in your home dir (~/.continue/config.py). Here’s my block for the 3 models available on my local-ai instance:

...
    models=Models(
		saved=[OpenAI(
			model="nous-hermes-llama2-13b.ggmlv3.q4_0.bin",
			api_key="",
			api_base="https://local-ai.octopusx.de"
		),
		OpenAI(
			model="starcoder.ggmlv3.q4_0.bin",
			system_message="You are a friendly AI that is helping me write high quality Python code. Give short and concrete answers.",
			api_key="",
			api_base="https://local-ai.octopusx.de"
		)],
		default=OpenAI(
			model="ggml-model-gpt4all-falcon-q4_0.bin",
			system_message="You are a friendly AI that is helping me write high quality Python code. Give short and concrete answers.",
			api_key="",
			api_base="https://local-ai.octopusx.de"
		)
	),
...

Sadly again, due to my current hardware limitations, I could not use this extension very much beyond asking it to generate me some simple example code. The same CPU pegging, timeout inducing computation of doom got in my way.

What’s next

Being mostly hardware restricted in most of my scenarios I think we will have to try GPU-based generation next. Though that will have to wait for another blog post, as I don’t even own a single dedicated GPU!

There are obviously many more extensions and integrations to explore now that I have this whole deployment going, the least of which would be making the Llama 2 chat bot available via chat rooms on my Matrix server. Exciting times.

So, stay tuned for future shenanigans in the land of generative self hosted AI models!

Ask Llama!

Generate me an outro section for my technical blog post about LLMs, inviting the reader to come back to the site for the fantastic content in the future.

Thank you for reading my blog post on large language models and their uses in self-hosted infrastructure. Be sure to check back for more exciting content and insights into the world of AI and natural language processing. Until next time!

After many iterations and failed attempts at having a sensible strategy for storing the data of my stateful applications I ended up sticking with Longhorn, which has proven to be reliable and flexible yet simple enough to fit my needs. For more information on how I use it I recommend checking out my previous posts: https://blog.octopusx.de/posts/backup_restore/ and https://blog.octopusx.de/posts/homelab_ha_edition/.

Building trust

Having been using Longhorn now for a good few years, surviving a few cluster and Longhorn version upgrades I decided it is time to start encrypting my volumes again. This is particularly important now that I have an offsite backup. Thankfully Longorn has a built in encryption mechanism that can be used to decrypt volumes on the fly while mounting them to their respective containers for use.

The setup

We will be following Longhorn’s official guide to set this up: https://longhorn.io/docs/1.5.1/advanced-resources/security/volume-encryption/

It states that we need the dm_crypt kernel module loaded and cryptsetup installed on the worker nodes. I can see that I have cryptsetup on my Ubuntu 22.04 server workers:

~ ❯ sudo apt search cryptsetup                                                                                    5s user@worker01
Sorting... Done
Full Text Search... Done

cryptsetup/jammy-updates,now 2:2.4.3-1ubuntu1.1 amd64 [installed,automatic]
  disk encryption support - startup scripts

However the dm-crypt kernel module is nowhere to be seen. Since cryptsetup seems to be working fine though, let’s see if it is indeed needed or if it is a soft requirement…

lsmod | rg crypt
# (sound of crickets, but in text form...)

First thing we want to create is the secret that will be used by longhorn to encrypt and decrypt all the volumes. It has to look like this:

---
apiVersion: v1
kind: Secret
metadata:
  name: longhorn-crypto
  namespace: longhorn-system
stringData:
  CRYPTO_KEY_VALUE: "super-secret-extra-long-key"
  CRYPTO_KEY_PROVIDER: "secret"
  CRYPTO_KEY_CIPHER: "aes-xts-plain64"
  CRYPTO_KEY_HASH: "sha256"
  CRYPTO_KEY_SIZE: "256"
  CRYPTO_PBKDF: "argon2i"

We want to create this secret manually by the means of running kubectl apply -f <filename>, although I recommend storing the CRYPTO_KEY_VALUE in a password manager as a backup. I was wandering as to the length of this key too but didn’t imediately find any recommendations, so I chose a 40 character random string for now.

Also, bear in mind, that whenever you restore an encrypted volume from a backup to a new cluster you will need to re-create this secret, otherwise the data on your volumes won’t be readable. I have created this secret on both my production cluster as well as my backup cluster already to enable me to test the backup/restore procedure.

Next we need to create a new storage class, which we will use when creating encrypted volumes, new and restored from a backup alike:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-crypto-global
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "2"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""
  encrypted: "true"
  # global secret that contains the encryption key that will be used for all volumes
  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"

You can see in the above example that they refer to our secret name. This way of creating volumes means that they all are encrypted with the same key. If you want to have a different key for each volume, the official Longhorn guide tells you how to do that too.

Conversion to encrypted volumes

There is no automated way to convert an unencrypted volume into an encrypted one in Longhorn as far as I know. In order to achieve this we will have to resort to a few tricks that will allow us to migrate the data from the old unencrypted volumes to the new encrypted ones.

The first thing I tried, thinking that I am cheeky, was to first make a backup of the Longhorn volume that I wish to have encrypted, then restore that backup with the “encrypted” flag selected. This however resulted in errors when trying to mount such a volume. It was worth a shot though, it would make our life super easy if this worked!

So, off to go be more tricksy. Whenever I need to perform any manipulation on data inside my K8S cluster volumes, I usually temporarily attatch said volume to my VSCode server container. This gives me a convenient way to move data around using the built-in VSCode terminal and edit any files then and there if necessary. This turned out the be the way to go.

Create a backup

First I would go and create an ad-hoc backup of the volume I am trying to migrate. This way I can migrate the current state of the data on each volume without interrupting the opration of the application using said volume.

Restore to a new unencrypted volume

Now that we have fresh data in our backup system, we can go ahead and use the restore feature while creating a brand new unencrypted volume. This will allow us to have a live copy of the latest data that we can subsequently mount to our VSCode container, again, without interupting the related application.

Create a new encrypted volume

Then I go ahead and create a new encrypted volume. This will be the target volume that we will be eventually attatching to the container once all the data is migrated.

Attatch the new volumes to VSCode

In order to attatch the 2 volumes we just crated they first have to have PVs and PVCs created. This does not automatically happen when you create a new volume either fresh of from a backup. That is unless you’re restoring a backup and select the “use previous PVC name”, which we do not want to do because that PVC still exists and is attatched to the original container.

Once you create the PVCs we go to the VSCode helm chart and modify it, adding the extra two volume mounts.

extraVolumeMounts:
  # Source volume, restored from backup
  - name: dev-01
    mountPath: /mnt/gitea-mariadb-02
    existingClaim: gitea-mariadb-02
    readOnly: false
  # Destination encrypted volume, freshly created
  - name: dev-02
    mountPath: /mnt/gitea-mariadb-04
    existingClaim: gitea-mariadb-04
    readOnly: false

Then apply the changes so that they take effect:

helm upgrade vscode .

Data sync

Our vanilla VSCode container is missing rsync, which is easily remedied. CTRL+j brings up the terminal and off we go:

sudo apt update
sudo apt install rsync

Just like that:

Once that is done, run a sync command to copy all of our data from the backup volume and into the new encrypted volume:

sudo rsync -avxHAX --info=progress2 --info=name0 /mnt/gitea-mariadb-02/* /mnt/gitea-mariadb-04

Once this is done you can do a quick sanity check, make sure that both the volumes have the same weight before moving on:

du -sh /mnt/gitea-mariadb-02
du -sh /mnt/gitea-mariadb-04

Now we are ready to detatch these volumes from VSCode and attach them to our target workload. I usually do this by rolling back the last change to VSCode:

helm rollback code-server 0

Do the switcheroo

The final step is now to swap the volumes from beneath our container. Most helm charts will have a persistence section in the Values file, where we can point it to our new volume using the existingClaim parameter. Here is an example from my mariadb helm chart (imported from Bitnami repo):

mariadb:
  primary:
    persistence:
      existingClaim: gitea-mariadb-04

Then we upgrade out deployment and fingers crossed we end up with a working deployment:

helm upgrade gitea-mariadb .

One caveat here is that sometimes (and it depends on how the helm chart is structured) if you are using a statefulSet K8S deployment type, it will complain that such changes are not allowed on an sts and the deployment will fail. To remedy this you can modify the sts directly using kubectl edit command, replace just the volume name in its config, wait for the changes to take place, then run the helm upgrade. This should help you roll out this change without having to reinstalling the whole deployment.

Epilogue

It definitely took me a few good sessions of manual tinkering to migrate all of my data to encrypted volumes. Some of them have been running for a few weeks now and I haven’t had any problems with them so far. Evident lack of dm-crypt kernel module on ubuntu 22.04 was apparently also not an issue, the mounted volumes get decrypted and attached just fine. So was it all worth the efford? For the peace of mind, I would definitely say yes…

If you can’t restore it, you don’t really have a backup…

Protecting your data doesn’t end at having a 3-2-1 backup. Whether you are self hosting services or handle company data in the cloud it pays to have a practiced plan for bringing your services and data back to life. If all you’re going to do is read this section of the post and move on with your life, I’d like you to take with you just this one key lesson: don’t leave testing of your restore procedure until you actually need to restore! For anyone more technically minded I recommend to read through the log of a GitLab outage from 2017, where in the middle of an outage an employee accidentally deletes a live database, then realises that the S3 backups they thought they had were never actually working! Yikes! Let us see how I deal with backups and disaster recovery in my homelab using the insights I have learned working in the industry. All using open and free software!

The chicken and egg situation

So as we are talking about a homelab situation in this article, we are always going to be limited by certain inavitable factors. Like, most people have just one internet connection. Single source of electicity, a UPS in a best case scenario. What most homelabers also have in common is that they’re building this entire stack in their own spare time. It often means that we can’t just throw more hardware nor more elbow greese at the problems we encounter.

In a pevious article I outlined how I improved both my uptime and reliability by building a multinode k3s cluster. In that post I have described a situation where I had to scrap my cluster almost entirely twice before I was able to get it just right. Something however that I did not do was have a decent way to protect my data while this all was happening, which meant that, at least for some applications. I had to rebuild my Nextcloud instance from scratch, but even worse, I lost my Gitea instance and all the automations I ran from it.

So, if the infrastructure needed to restore your infrastructure is gone, what can you do? Well, let’s go over our options.

How do I back up so far…

Logically my storage solution has 3 tiers.

Tier 1

Live data being accessed by my applications and services running in my k3s cluster.

This data is stored on fast NVME storage, and replicated across multiple physical Kubernetes nodes. This tier of storage is completely managed by Longhorn.

Tier 2

Backup of the live data hosted on the local network, using an NFS share on a virtualised Truenas Scale node.

This data is stored on a redundant RaidZ2 array on a single node. Data makes its way onto this Truenas instance using Longhorn’s nighly snapshotting and backup mechanism. Data on this tier is at most 24h old at any given time.

Tier 3

Offsite copy of Tier 2 to another Truenas Scale machine, using Truenas’ built in replication mechanism.

This data is also stored on a redundant RaidZ2 array on a single remote node. This data makes its way onto the offsite Truenas instance via a Wireguard server hosted in the public cloud. Data is pushed in form of incremental backups and it is at most 48h old at any given time.

Visualisation

Longhorn and friends

So as you see this backup strategy relies heavily on Longhorn, which is a kubernetes storage orchestrator with powerful snapshotting and backup capabilities. What I found particularly interesting was that you can point multiple instances of Longhorn, each maintaining the storage of a completely separate Kubernetes cluster, to the same backup location. What this allows you to do is essentially use the backup and restore mechanism as a migration utility to move storage volumes between Kubrnetes clusters!

The beauty of this solution comes likewise from the fact that our workloads are running in Kubernets and that we are using Helm charts to deploy our applications. Helm charts which, conveniently, we are treating as our infrastructure as code.

Having both, the portability of application data (Longhorn-managed k8s storage volumes) and application configuration (helm charts) means that we have a repeatable, reliable way to migrate our workloads off site and spin them back up at will!

What is the last thing you need to get started? A good friend who will allow you to spin up an offsite k3s cluster on his Truenas box ;)

The nitty gritty

Let’s go over the different moving parts that we need to set up to see how it all works.

Longhorn helm chart

I am using the official longhorn helm chart:

# Chart.yaml
apiVersion: v2
name: longhorn
version: 0.0.1
appVersion: v1.4.2
description: Longhorn is a distributed block storage system for Kubernetes.
icon: https://raw.githubusercontent.com/cncf/artwork/master/projects/longhorn/icon/color/longhorn-icon-color.png
dependencies:
  - name: longhorn
    version: 1.4.2
    repository: https://charts.longhorn.io

You can find all of the parameters in the github repo: https://github.com/longhorn/charts, but what we want to focus here is the following configuration values:

# values.yaml
longhorn:
  defaultSettings:
    backupTarget: nfs://scale.octopusx.de:/mnt/hybrid/longhorn-backup
    backupstorePollInterval: 60
    defaultLonghornStaticStorageClass: longhorn-static
    (...)

Notably, we need to set the backupTarget and the defaultLonghornStaticStorageClass. The target will be an NFS share made available by the Truenas Scale running on my local network. The static storage class will be a persistent volume class created inside of Kubernetes for all volumes that are created “statically”, i.e. not automatically as part of bootstrapping a new application. Using this method of provisioning the initial volume as well as restoring one from the backup means that the proces of bootstrapping this application is nearly identical for both the main instance and the backup one on the remote cluster.

Create a static volume for Gitea

To create a new static volume for our Gitea to use, we go to the Volumes tab in Longhorn then click on Create Volume and fill in the form:

Next click on the new volume and ask longhorn to create the PVC and PV for you inside Kubernetes:

Gitea helm chart

Normally you can provision new volumes dynamically, declaring the size and class. Since we are using a pre-provisioned volume and the PVC for it already exists, we can use the existing claim variable to pass it to our gitea chart.

gitea:
  (...)
  persistence:
    enabled: true
    size: 20Gi
    accessModes:
      - ReadWriteOnce
    existingClaim: data-gitea-02

You’ll obviously need the rest of the chart to deploy gitea, you can find the template I use here.

Deploying this chart should looks identically on my main Kubernetes cluster as well as the remote backup instance.

Longhorn Backup

If you are following along we should be able to access the Longhorn web UI, so long as you have enabled it in the helm chart:

If all of this worked you should be able to go to the Backup tab and not see any connection errors. In my case, that NFS mount already contains earlier backups of various volumes I have backed up from a number of kubernetes clusters.

Perfect! Now, we don’t want to make backups manually. This is a homelab after all, it’s is not our job to do all of the meneal tasks manually, it is to automate them away! The Recurring Jobs tab will let us do just that, here we can schedule nightly backups of whatever volumes we want. We can create a group label and label each volume with it to link it to a specific recurring task. For us, we will use the default group, which is already attatched to every single volume, like so:

And that is it for the Longhorn backups, this will trigger a kubernetes job every night at the selected time and do the work. The backups are incremental so it is storage efficient if we desire to keep multiple backups for each volume by setting the Retain value to anything above 1. Once a backup is performed you can verify it by going to the Backup tab or any volume specifically, like this gitea data volume that we use for this example:

Offsite Backup

With backups of kubernetes cluster PVCs securely pushed every night to our Truenas server via NFS, the next step for me was to do the push from tier 2 (local backup) to the 3rd tier (offsite backup). This I achieve via the use of Truenas replication feature. I won’t be covering this step in detail as the replication itself is a well documented feature, so I will send anyone interested to the official Truenas documentation.

As for how do my local Truenas host talks to the remote Truenas, I use Wireguard VPN. Technically Truenas scale doesn’t have the option to configure Wireguard in the UI (though OpenVPN is an option), but Wireguard binaries are preinstalled (at least they are at the time of writing, I am currently running TrueNAS-22.12.3.2 Bluefin). This means that you can easily follow my previous post and simply add both the local and remote Truenas hosts to your Wireguard network via your cloud-hosted Wireguard server.

Restore from offsite backup

Restoring from backups is very simple in Longhorn. On our remote backup K8S instance we will go to the Longhorn Web UI, the Backup tab, then select the volume to restore:

Create the volume with the same name as the previously backed up volume:

Ask longhorn to create the PVC and PV for you inside Kubernetes:

The rest of the steps is identical to bootstrapping the original Gitea instance! Just make it use the existing, restored PVC which conveniently already has the correct name :)

Switching traffic to the offsite backup

As it can be seen on the beautiful visualisation I produced, the way I access services in my homelab from the outside world is by hosting a HA proxy in the public cloud. A subnet router node is deployed on both my Homelab LAN and Offsite LAN (not actually represented on the graph) that is able to route traffic to the respective networks. This means that switching traffic from one deployment to the other is as simple as changing a few lines of code in the HAProxy config:

frontend traefik_public
        bind *:443
        default_backend    homelab # <- We change homelab to offsite here
        option             forwardfor

# Homelab LAN Backend
backend homelab
        # Use roundrobin to balance traffic
        balance            roundrobin
        # Define the backend servers
        server             backend01 192.168.50.50:443 check

# Offside LAN Backend
backend offsite
        # Use roundrobin to balance traffic
        balance            roundrobin
        # Define the backend servers
        server             backend01 192.168.150.50:443 check

Then we restart the HAProxy service, all done!

The neat outcome of it all

So, for those of you that are more familiar with Kubernetes, you will have spotted that I omitted a number of other things that have to be synced between the two clusters in order to be able to switch traffic freely between them. For a start, Gitea has a database, so you will need to backup its volume and restore that too. However, the neat thing about this solution is, the backup and restore procedures are identical for each and every volume, and they should work just as well, so I didn’t feel it was worth doubling up on the explanation.

Another, and crucially, untouched topic here would be certificates. One way to solve this cleanly would be to resolve TLS at the HA proxy, which means you don’t need to do anything with the certs when you fail over. Alternatively you would have to sync your certificates between the clusters in some other way, or use features of Traefik or another ingress controller to generate all of the certs you want on both systems in parallel.

Lastly, how I actually use this in practice, and how I recommend you do it - the failover is fancy and shows the power of this solution, but I just use the wireguard network I already have to access my backups if necessary. More importantly, thanks to this wireguard network I can perform these restores once a month or so, to make sure the whole process still works. Because, as I alluded to at the very beginning, if you haven’t tested your restore procedure, do you even really have a backup?

The Humble Beginning

In the beginning I hosted everything on a lone Unraid server, which I built by converting an old desktop machine I had laying around, a very common beginning for a homelab. Unraid gives you a very economical way to manage mass storage with some redundancy, with the ability to run docker containers and mount their volumes to either the spinning rust array, or an ssd cache drive.

This has worked ok for a few years, but in time 3 problems started grating on me:

• If you want to do any maintenance of the server, you must spin the disk array down. And by maintenance I meana changing most basic settings, such as network or storage configuration. And every time you stop the array, you must shut down the VM and docker engines for the duration.
• There were multiple times where failure to resolve dns meant that my containers would fail to update, then promptly “disappear”, together with all their configuration. These were configured manually and now had to be rebuilt manually…
• Every time one of these events occured, all of my homelab users would be inconvenienced. Long periods of unavailability is unacceptable if you want to build a system that multiple people rely on on daily bases.

Make it double

I decided to experiment with a HA setup and build a second unraid server. This involved abandoning Unraid’s built in docker engine (for the most part), and choosing an alternative way to orchestrate my workloads. My first choice was Microk8s from Cannonical, as it was the simplest way to create a HA multinode cluster.

The goal of this setup was to allow me to take one of the nodes down for “maintenance” and have the services remain available, running on the second node.

The initial idea for how to design this system was basically a mirror setup. Each of the Unraid nodes would have:

• 2 large HDDs for the main Unraid array, storing the data
• 1 NVME SSD used as a separate “cache pool” for storing the Unraid VM and Docker engine data
• A docker container running Pihole
• A number of VMs running a Kubernetes cluster

What are we running

The most important application that we want to host is Nextcloud, with the intention of using it as photo backup for all family phones. To achieve our functionality we needed to run at minimum the following applications in our kubernetes cluster:

• Metallb
• Traefik
• Nextcloud Server
• Redis
• Mariadb

Storage

3 out of the 5 services needed are actually stateful applications with requirements to have some form of either object or block storage. The most obvious solution that comes to mind is to leverage the NFS server hosted on our 2 Unraid servers. It was not a viable option because it would again create a single point of failure. You can not run pure NFS with any sort of failover or HA, which means that it doesn’t matter that you have 2 unraid servers, the moment you take the wrong one down for “maintenance” the whole system collapses.

I started researching different different distributed filesystem solutions that can be used to provision storage for kubernetes nodes, and I came across the following:

• CephFS + Rook
• GlusterFS
• Longhorn

I have looked into Ceph for a little while before deciding it is too complex for my simple setup. I tried to setup GlusterFS but failed to make it work. I also found information that the GlusterFS Kubernetes provisioner was getting deprecated, which in the end led me to just one realistic choice, Longhorn. I looked at Longhorn last because I was hasitant to use a non-generic storage solution from Rancher. I generally try to stick to CNCF and proven open-source projects in my homelab, as I did not want to invest time into setting up infrastructure that may get “deprecated” or “abandoned” and have to re-build everything. However, I did manage to make longhorn work on my MicroK8S cluster, and I was off to the races with my new HA setup.

The Longhorn documentation mentions that for best results they recommend a high IOPS NVME drive for storage, so I added a couple of NVME SSDs into each of my Unraid servers to pass through to the VMs for that purpose.

First attempt

For simplicity’s sake, I decided I not to create an overly complex and maintenance-heavy system and stayed away from fully featured enterprise-grade K8S solutions. Having played with MicroK8S from Cannonical for a while in a local VM I decided to give it a test run in cluster mode. I have initially created a very simple 2 virtual machine setup, one machine on each Unraid server. I passed each its own NVME disk and exposed it to Longhorn and everything was fine… For a while… Until it wasn’t fine anymore.

The first time I went to reboot one of the Unraid servers the MicroK8S cluster desintegrated. In order to maintain operation of the system I had set Longhorn to have 2 replicas of each persistent volume, so one on each node. This means that when one of the nodes comes back after being unavailable for a while, it will discard its copy of the volume and sync it from the nodes that remained available in the cluster. This syncing operation is very IOPS and CPU heavy, to the point that while the sync operation was taking place, the K8S control plane (ETCD) was starved for cycles and the cluster had gone out of sync entirely and failed to come back.

Some of you might say, well, you should have known this right? This is why you shouldn’t host the control plane and data plane on the same host! It is asking for trouble… Ok, what should we do about it then? To save some words, I tried creating separate MicroK8S nodes for the control plane, data plane and storage, so that I woud have one of each type of nodes, each on a separate VM across the 2 Unraid boxes, and TLDR, it doesn’t work because MicroK8S doesn’t let you choose which nodes are supposed to act as masters and which are to be workers. The first 3 nodes to be added to the cluster would form the control plane, as necessary to reach quorum in ETCD. Each next added node would not be a part of the control plane. The moment you reboot one of the “master” nodes though (as long as another node in the cluster is available) another one will pick up that role and you have no say in which one that would be. After a few weeks the cluster has died in much the same way as it did initially. We needed a different solution…

Move to K3S

It seems to be a bit of a theme at this point, but Rancher Labs seems to know what they’re doing. Aside from MicroK8S, their K3S distribution of Kubernetes is another viable and likely the most popular option for Kubernetes on the edge and in the homelab. It is lightweight and flexible, allowing you to dedicate nodes specifically to the control plane and even using alternative data stores for control plane state.

Final touches

Just like before, we have built a 6 VM Kubernetes cluster, 2 master nodes, 2 workers and 2 longhorn storage nodes. To keep the system as vanilla as possible I elected for the default storage technology, which is ETCD. Such a system however is not HA, to have quorum you need 3 ETCD hosts. To ensure that the cluster stayed up while one of the Unraid servers is taken down I decided to host the third ETCD server… on a spare RaspberryPi I had laying around of course!

The public endpoint

To be able to host a VPN you can connect to from anywhere you will need a machine with a public IP. As mentioned before, this can be solved by setting up DDNS instead, but if you’re like me and you don’t have that option this is a solid alternative. The easiest way to obtain a public IP address is to create a small cheap VM with a cloud provider to host your Wireguard VPN. In my setup I have used Linode, but you can use anything you like really, so long as it gives you that sweet public IP goodness.

Provision the host

We are going to build our Linode wireguard VPN on an Ubuntu linux VM, version 20.04. Our VM should have wireguard available as a package out of the box which make installation super easy:

sudo apt update && apt install -y wireguard resolvconf ufw

It is probably also a good idea to enable firewall on this host. For now, we can safely block all traffic except ssh and wireguard ports:

ufw allow ssh
ufw allow 51820/udp
ufw enable

If you’ve run the above and are still able to connect to your server over ssh, you’ve done well. If your server is now turning a blind eye to your ssh connection attempts, you may have to access it over linode shell, then double check you’ve blocked the correct ports and fix the firewall rules if necessary.

Configure Wireguard

Once we got the software cosily installed we have to configure the bad boy. We can start with generating a keypair for the server. To do that run the following command on your cloud instance:

wg genkey | tee private.key | wg pubkey > public.key

This will create the two key files for you, private and public. The contents of the private key will live in the config on the server while the public one has to be shared with every peer that wishes to connect to this server. Either way, it is a good idea to think about where you’re gonna store these keys for safekeeping. I recommend using a secure note in your password manager app or an encrypted backup drive. You’ve got one of these already, right? Right!?

The next step is to create the server configuration for your public endpoint. The location of all of the endpoints will be stored in /etc/wireguard. We are going to call ours wg0.conf. You will notice that once we are done there will be a new virtual network interface on your linux host also called wg0. That’s because wireguard will create a virtual interface per tunnel endpoint that you configure, but more on that later ;)

Take a look at the following example config file:

[Interface]
Address    = 10.0.0.1/24
DNS        = 192.168.1.10,my-search-domain.com
SaveConfig = false
ListenPort = 51820
PrivateKey = mysupersecretgeneratedprivatekey
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = mylesssecretbutalsosecretpeerpublickey
AllowedIPs = 10.0.0.2/32, 192.168.1.0/24 # The networks that traffic can be routed to over this peer connection

The above block is a good starting point, let’s go over its contents briefly.

The interface block is the configuration parameters for our wg0 virtual network interface. It also configures some server-side parameters, such as what is the address of the DNS server to be used to resolve domain names on our VPN network. This is useful if you are hosting a DNS server within your lan to resolve internal domain names.

Another field we can see is the port, which is the network port to be used by peers to establish a connection with the server. For safety reasons (security by obscurity) it is a nice idea to change it to anything else other than the default.

We will also need to whitelist this address on our firewall and any security group analogs your cloud provider offers. Lastly we have the private key where, yes, you guessed it, we paste the private key that you so diligently saved in your password manager, haven’t you?

PostUp and PostDown fields allow us to run arbitrary code triggered by the change of our WireGuard tunnel state. Names are quite self-explanatory. Remember, just because we are able to create the tunnel, doesn’t mean we will send any traffic through it. It is by modifying network policies after the tunnel is established (and subsequently tearing them down when we take the tunnel down) that we are able to funnel our traffic where it is meant to go.

The second block we can see is the peer block.

The first field would be a public key. And no, this is not our server’s public key, this is the public key of your peer. A peer in this case is any machine trying to connect to our vpn server. Therefore you will likely have many peer blocks in this secrion, as many as there are devices you want to allow to connect to your network.

The allopwed IPs field will tell the server what range of IP addresses (subnets) it will accept traffic from should the handshake with the matching peer be successful. This will be the IP address of the wg0 (or whatever you call name it) interface BUT ON THE PEER. Plus any other network this other peer will expose to us.

If this is a little confusing, don’t worry, we’ll be able to connect the dots a little later.

Reaching the Unraid server

Unraid is a very potent platform for managing your home server infrastructure and it makes it extremely easy to deploy services within your home network.

I myself host a Pihole and HomeAssistant containers on mine among others. It also has the ability to install extensions. When trying to figure out an easy and reliable way to set up a permanent tunnel to my Linode wireguard host I stumbled accross a nicely written extension which at first seemed like it would do the trick.

After some attempts to make it work however I realised that it won’t do the trick, I could not get my Unraid box to forward traffic from the VPN to the rest of my network, which meant my internally-hosted services would not be reachable by other devices on the VPN. Mind you this was the beginning of my journey and I was only learning about wireguard so quite possibly it’s entirely my fault. If you have managed to make it work, more power to you!

In the end I decided to go for a small VM instead. Seeing as setting up wireguard on an Ubuntu machine is so quick and simple this went very smoothly.

Provision the VM

First, use the Unraid UI to spawn a new Ubuntu Server 20.04 VM. I assigned mine 2 vCPUs and 2GB of RAM, which should be plenty enough for the job. You can adjust this later anyway if you decide it is under or over-provisionned, so no biggie…

Once you’ve got it up and running, install the wireguard package just like on the Linode:

sudo apt update && apt install -y wireguard resolvconf

Just like before, we will need some freshly baked key material, straight from the oven: To do that run the following command on your cloud instance:

wg genkey | tee private.key | wg pubkey > public.key

I called my VM and configured it to have the hostname of wireguard-gateway, so we will refer to it as such from now on. Yup, nothing more to see here, let’s move on…

Configure a permanent tunnel

On the wireguard-gateway VM we will need to setup the basic wireguard tunnel peer config with a few small extras. Let’s take a look at what my config includes:

[Interface]
PrivateKey = privatekeyofthewireguard-gateway
Address = 10.0.0.3/24
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = publickeyofthewireguardlinodehost
Endpoint = my-linode-wireguard.com:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 20

Well now that finally looks like some black magic. If you’re familiar with iptables it likely makes perfect sense, I had to learn through strategic use of duckduckgo and trial-and-error…

The important bits here are the PostUP and PostDown commands, which are hooks to be executed when the tunnel is set to be up or down respectively. Here we are:

• Enabling forwarding of any traffic that enters through the wireguard interface wg0
• Enabling NAT Masquarading when traffic is forwarded to eth0 to allow it to find its way back to the VPN subnet Then the PostDown reverts the changes, to make sure routing works correctly also when the tunnel is down.

Since this machine is going to be the one establishing the connection with the Linode wireguard host as it itself lives behind the nasty nasty NAT, we want to set the PersistentKeepalive = 20 here too. Wireguard’s noise protocol will not transfer any data whatsoever unless actual data is requested or sent by peers on either side of the connection. This means that your host behind the layer of NAT will become unavailable after a little while unless you artificially keep pinging it via this setting, extending the lifespan of this connection.

• Enable forwarding
• Use systemd to start and enable the wg service, make sure it auto-reconnects on reboot

Launch the guard of wires

There are a couple of more steps necessary to turn our wireguard-gateway into a defacto gateway router. First of all, generic linux will have packet forwarding disabled by default. We will need to enable it and permanently, so that the setting survives any planned (or indeed unplanned) reboots and shutdown.

Go and add the following line to /etc/sysctl.conf:

net.ipv4.ip_forward=1

To check that it worked run the following command:

sysctl net.ipv4.ip_forward

The expected return value if it worked is this:

net.ipv4.ip_forward = 1

We should be able to to turn on our wireguard tunnel at this point. And for that, we have a special magic command:

sudo systemctl start wg-quick@wg0

Why this magic command? wg-quick is a utility that comes with wireguard and it lets you manage wg tunnels easliy as a human being and not a soulless robot. The above will not only start the tunnel but also enable a systemd service for it, meaning that the tunnel will remain up even after reboots. Nice.

Quick reality check

Here if everything went according to plan we should be able to start testing this new tunnel. The first thing you’ll likely want to do is make sure traffic is routed within the wireguard network itself.

Let’s (IP) address all of the hosts from the start.

# wireguard server
wg0 = 10.0.0.1
eth0 = 171.172.173.174

# unraid vm
wg0 = 10.0.0.2
eth0 = 192.168.1.5

Let’s log into the Wireguard server and ping some stuff:

~ ❯ ping 10.0.0.5 -c 1
PING 10.0.0.5 (10.0.0.5) 56(84) bytes of data.
64 bytes from 10.0.0.5: icmp_seq=1 ttl=64 time=29.6 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 29.618/29.618/29.618/0.000 ms

This means we can communicate with our Unraid VM. Sweet.

The bigger question is, can we also access resources on our LAN, behind our Wireguard gateway? Well, we can test that too. Find a local IP address on your home network that will respond to ping and use that. We are going to assume we have a Pihole server running under 192.168.1.10.

~ ❯ ping 192.168.1.10 -c 1
PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.
64 bytes from 192.168.1.10: icmp_seq=1 ttl=63 time=20.9 ms

--- 192.168.1.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 20.922/20.922/20.922/0.000 ms

Now, pinging traffic is special, in fact it is so special it uses its own specific protocol called icmp. What I found while working on this setup is, just because icmp traffic makes it through, it doesn’t mean that you’re all good just yet. Since we are using a Pihole endpoint to test the ping, and we know Pihole has a web UI, let’s try to curl that too.

~ ❯ curl -I http://192.168.1.10/admin/
HTTP/1.1 200 OK
Set-Cookie: PHPSESSID=4215ktt1igaqf8qoj5llbhv376; path=/; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-type: text/html; charset=UTF-8
X-Pi-hole: The Pi-hole Web interface is working!
X-Frame-Options: DENY
Date: Sat, 27 Aug 2022 13:16:15 GMT
Server: lighttpd/1.4.59

See? Now we are pretty sure it works. At least it will do for now and we can perform more practical tests once we set up wireguard on our end client machine.

The rest

Client config on Ubuntu

Now, this one gave me a lot of trouble for a while but I am very happy that I managed to figure it out all on my own, like a propper big boy with access to stack overflow! Let’s take a look at my final configuration file first, as I assume that’s mainly what you’re here for and postpone story time for a moment.

[Interface]
Address = 10.0.0.20/24
DNS = 192.168.1.10,my-search-domain.com
PrivateKey = myclientprivatekeypleasedontshare
MTU = 1280

[Peer]
AllowedIPs = 0.0.0.0/0
Endpoint = my-linode-wireguard.com:51820
PublicKey = thepublickeyofmylinodevpswireguarserverendpoint

This configuration block lives on my laptop in the usual place, i.e. /etc/wireguard/wg0.conf.

We will start from the peer block for a change. Allowed IPs here should be set to 0.0.0.0/0, which means that all of the traffic generated by our host will go through the tunnel. We can use this setting to adjust what goes through the tunnel and if, for example, we limit the subnet to 10.0.0.0/24,192.168.1.0/24, then the VPN subnet and our homelab subnet will be routed through the VPN tunnel, while all other internet traffic will be ommited and shot out straight onto the interwebz.

Endpoint will be identical for the wireguard-gateway and all of the peers that are not the VPS machine itself. It is the public address of the VPS machine, followed by the VPS’ public key to allow this peer to authenticate.

The interface configuration, as always, refers to the virtual wg0 interface on my laptop. The address will be the IP my client wants to assume within the VPN subnet.

The DNS entry just as before is telling my computer where to reach my internal domain name server and what my private search domain is.

The private key is to be generated and stored safely, as all of the previously generated keys.

Story time

Last but not least, the MTU. This is an optional parameter and was at the center of my problems with this entire setup. Generally most people do not need to touch this, though hopfully this will be a cautionary tale for anyone experiencing weird issues with their wireguard tunnels. My first attempt of making my roadwarrior peer working omitted this parameter, as most guides and vanilla setup documentations do not mention needing it. Indeed my VPN tunnel would connect without it. I could ping all of the hosts on the VPN subnet as well as my private subnet behind NAT. I could even resolve DNS. But when I tried to perform any other type of data transfer over the tunnel however, it would fail.

To troubleshoot this I had tested reachability from and to each host on my network, and all would work besides connections to and from my roadwarrior peer. This would have indicated that either my wireguarad-gateway host is not forwarding traffic correctly to the VPN subnet, or that there is an error in the VPN tunnel configuration on the VPS. At the time however I already had one working android peer, which could access and use all of the services hosted in my private subnet via the wireguard tunnel, so neither of the aformentnioned was a likely conclussion.

I have resorted to take an even closer look at what is happening on the network when I try to establish an ssh connectino between my roadwarrior peer and an ssh server in my private subnet. I set up probes listenning for ssh traffic on all 4 contact points:

• the roadwarrior peer
• the vps server
• the wireguard-gateway
• the target server in the private subnet

What I observed was that connection request would make all of the hops starting from the roadwarrior and ending on the targert server. Return traffic originating from the target server however could be observed on the target server itself, the wireguard-gateway, then the vps server, but the trail would end there. This was an indicator that it is strictly a problem with my ubuntu roadwarriror host.

After some frantic duckduckgoing and many a-stackoverflow page overturned, I found our that my probems are sympthomatic for MTU-related issues. This means that my laptop is not accepting packets exceeding certain size, and therefore dropping most of the traffic returning to it from my wireguard tunnel. Setting a smaller-than-default MTU made it work instantly, and every tunnel and virtual interface lived happy ever after…

Epilogue

So since homelabs are pretty much living beings, by the time I finished writing this blog entry my Wireguard setup has evolved and is used for much more than what is described here. What I must say though, I have myself refered back to this guide when trying to figure out issues with my own setup as well as when helping friends erect their own self hosted VPNs, so I am quite happy with the contents here and hope someone will benefit from reading this document. As to the parts that I omitted in this entry, I guess you will have to check back in to find out the missing bits and peaces as I scatter them across the field of future blog entries ;)