An image with the TrueNAS and K3s logo

Install Kubernetes on TrueNAS 25.04

3 October 2025

After years of running a home lab in my basement with Ubuntu and Kubernetes managed through ArgoCD, I decided to upgrade to a new machine and switch to a different distribution. My choice: TrueNAS.

One deciding factor is that TrueNAS supported Kubernetes out of the box ... or so I thought. The embedded Kubernetes feature was removed in TrueNAS 24.10, which I missed during my research. TrueNAS's other features significantly outweigh the manual configuration effort required on Ubuntu. I decided to keep TrueNAS and began searching for a way to install Kubernetes.

Prerequisites

When looking at various Kubernetes distributions, I basically had three choices

  • K3s: A batteries-included Kubernetes distribution.
  • Talos: An immutable OS for Kubernetes clusters.
  • kubeadm: The official Kubernetes installation method.

I chose K3s for one crucial feature: external cluster datastore support, such as MariaDB.

Why External Storage Matters

TrueNAS leverages ZFS architecture, where the system itself is installed in a dataset and versioned. During updates, TrueNAS creates new datasets and performs a clean installation, allowing rollbacks if updates fail.

This design creates two important requirements:

  • Avoid system modifications: Changes made to TrueNAS get lost during updates
  • Protect cluster data: The cluster state must survive TrueNAS updates

A dataset for your cluster's configuration

First, I created an apps dataset and inside that I added two other datasets k3s and mariadb:

apps/
├── k3s/
└── mariadb/

This structure separates concerns and provides clear organization for your cluster components.

Installing MariaDB

Rather than using a Kubernetes Operator for MariaDB, I leveraged TrueNAS's built-in app system:

  1. Go to Apps -> Discover Apps.
  2. Select MariaDB and click Install.
  3. Choose a name for your application or keep the default
  4. Enter the configuration for your database.
    • User: user
    • Password: password
    • Database: default
    • Root Password: root_password
  5. Set Network mode to "Publish port on the host for external access"
  6. Use Host Path as storage and select your Dataset: /tmp/tank/apps/mariadb.
  7. Click Install

After a few seconds, your database is ready.

Creating the K3s Database

We will now create a dedicated database and user for K3s.

Open a shell to your MariaDB container (you may also use a DB client from your own machine).

mariadb -uroot -p

You will be prompted for the root password, which we set up earlier.

You can now create a database and a user, and then grant this user full access to that database.

CREATE DATABASE k3s;
CREATE USER 'k3s'@'%' IDENTIFIED BY 'k3s_password';
GRANT ALL PRIVILEGES ON k3s.* TO 'k3s'@'%';
FLUSH PRIVILEGES;

Installing K3s: The Working Solution

After multiple failed attempts (documented in the appendix), here's the command that successfully installs K3s on TrueNAS:

Run this command with an admin user on the TrueNAS shell.

curl -sfL https://get.k3s.io | \
  INSTALL_K3S_BIN_DIR=/mnt/tank/apps/k3s/bin \
  K3S_DATA_DIR=/mnt/tank/apps/k3s/data \
  K3S_TOKEN=secret \
  K3S_DATASTORE_ENDPOINT="mysql://k3s:k3s_password@tcp(127.0.0.1:3306)/k3s" \
  INSTALL_K3S_EXEC="--write-kubeconfig-mode 666 --docker" \
  sudo -E sh -

That looks quite unwieldy. Let's look at all the options that were set here and why they're important:

  • INSTALL_K3S_BIN_DIR=/mnt/tank/apps/k3s/bin: Stores the k3s binary in your persistent dataset
  • K3S_DATA_DIR=/mnt/tank/apps/k3s/data: Places working data in your persistent Dataset
  • K3S_TOKEN=secret: Sets the agent authentication token
  • K3S_DATASTORE_ENDPOINT="mysql://k3s:k3s_password@tcp(127.0.0.1:3306)/k3s": Connects to your MariaDB instance
  • INSTALL_K3S_EXEC
    • --docker: Critical setting that leverages TrueNAS's existing Docker daemon
    • --write-kubeconfig-mode 666: Allows non-root users to read the kubeconfig file

This takes a few seconds, and your Kubernetes cluster is started.

Verify the installation:

sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) ...

It's not entirely finished yet; you will need two more things to be in business.

Post-Installation Configuration

Accessing your cluster

I was not able to run Kubectl directly inside Truenas 
truenas_admin@truenas[~]$ export PATH="/mnt/tank/apps/k3s/bin:$PATH"
truenas_admin@truenas[~]$ kubectl get pods      
FATA[0000] exec /mnt/tank/apps/k3s/bin/kubectl failed: argument list too long

However it works fine when accessed from outside, for that you need to copy the kubeconfig from your TrueNAS server and change the IP address in the file.

export SERVER_IP=<server-ip>
scp user@$SERVER_IP:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i "s/127.0.0.1/$SERVER_IP/g" ~/.kube/config

Test your connection:

kubectl get nodes

Enabling Port Forwarding

socat is missing from TrueNAS, and it is required for kubectl port-forward. Install it using this workaround:

sudo docker run -it debian:12-slim bash
# Inside the container:
apt update && apt install socat
exit

# Copy socat from the container:
sudo docker ps -a  # Note the container name
sudo docker cp <container-name>:/usr/bin/socat /mnt/tank_nvme/apps/k3s/data/data/cni/socat

Handling a TrueNAS update

Important: Every TrueNAS update creates a clean system Dataset that doesn't contain the K3s systemd service file. After each update, re-run the original installation command:

curl -sfL https://get.k3s.io | \
  INSTALL_K3S_BIN_DIR=/mnt/tank/apps/k3s/bin \
  K3S_DATA_DIR=/mnt/tank/apps/k3s/data \
  K3S_TOKEN=secret \
  K3S_DATASTORE_ENDPOINT="mysql://k3s:k3s_password@tcp(127.0.0.1:3306)/k3s" \
  INSTALL_K3S_EXEC="--write-kubeconfig-mode 666 --docker" \
  sudo -E sh -

And you're back in business!

Your cluster data remains intact because it's stored in your persistent Dataset and the external MariaDB database. Regular server reboots don't require this reinstallation; only TrueNAS system updates require it.

Conclusion

You now have a fully functional Kubernetes cluster running on TrueNAS with persistent state storage in MariaDB. This setup survives TrueNAS updates (with minimal reinstallation required) and provides a robust foundation for your home lab applications.

Happy homelabbing!

Appendix: Unsuccessful experiments

It took me some time to figure out the correct way to install K3s. Here's everything I tried but failed to get working.

Run K3s inside a Docker container

My first attempt was to run K3s inside Docker as a TrueNAS app. This approach seemed straightforward as it allowed the use of only standard TrueNAS features and would thus be resilient to updates.

Here's the configuration I used 
services:
  server:
    hostname: k3s-control-1
    image: "rancher/k3s:v1.33.3-k3s1"
    networks:
      default: {}
      k3snet:
        ipv4_address: 192.168.1.6
    command:
    - server
    - --disable=traefik
    - --disable=local-storage
    - --disable-helm-controller
    - --advertise-address=192.168.1.6
    dns:
      - 192.168.1.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    security_opt:
    - seccomp:unconfined
    environment:
    - K3S_TOKEN=<secret>
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    - K3S_DATASTORE_ENDPOINT=mysql://k3s:<password>@tcp(172.16.0.1:3306)/k3s
    volumes:
      - /mnt/tank_nvme/apps/k3s/output:/output
      - /mnt/tank_nvme/apps/k3s/control:/var/lib/rancher/k3s

  agent:
    hostname: k3s-worker-1
    image: "rancher/k3s:v1.33.3-k3s1"
    networks:
      default: {}
      k3snet:
        ipv4_address: 192.168.1.7
    command:
    - agent
    dns:
      - 192.168.1.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    security_opt:
    - seccomp:unconfined
    volumes:
      - /mnt/tank_nvme/apps/k3s/worker-1:/etc/rancher/node
      - /mnt/tank_nvme/apps/k3s/storage:/var/lib/rancher/k3s/storage
    environment:
    - K3S_URL=https://server:6443
    - K3S_TOKEN=<secret>
networks:
  k3snet:
    driver: macvlan
    driver_opts:
      parent: br0
    ipam:
      driver: default
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

The apiserver started and was accessible, but containers were in CrashLoopBackoff with a cryptic error.

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "argocd": executable file not found in $PATH

My initial suspicion was that this issue was due to a missing or incorrect security option in the container. Still, I was already running the containers in privileged mode with seccomp:unconfined. I could not add more permissions.

Perhaps the solution was to not use Docker and move to the host directly?

Running K3s directly on TrueNAS

I was disappointed to discover that installing K3s on TrueNAS still failed with the same permission errors.

curl -sfL https://get.k3s.io | \
  INSTALL_K3S_BIN_DIR=/mnt/tank/apps/k3s/bin \
  K3S_DATA_DIR=/mnt/tank/apps/k3s/data \
  K3S_TOKEN=secret \
  K3S_DATASTORE_ENDPOINT="mysql://k3s:k3s_password@tcp(127.0.0.1:3306)/k3s" \
  INSTALL_K3S_EXEC="--write-kubeconfig-mode 666" \
  sudo -E sh -

It took me days of investigation to understand the issue. The reason is that when mounting the container's filesystem, the partitions would get a noexec flag, preventing the container from starting.

shm on /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6b0d3f5dc0426d3b774dc47e2beb062eb802e5d71dac5489af2a7f3206b1b2e3/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
shm on /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/f69b02fb47ba32527bc91e722149d3187f04bb23d1d52cfeb7303c85822cfb4e/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
shm on /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/5594e9d8ff604c4307b7373e278becb6290ed904ffe0b95ea42391e265523d41/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)

The noexec flag is inherited from another mount, which is a result of TrueNAS's advanced security hardening.

Some more hours of research led me to many voodoo tricks and some AI hallucinations, but to no avail.

I tried to find a way to request K3s to use a different path for these mounts, possibly under /mnt/tank/apps/k3s, but this is apparently not possible. A proposed alternative is to "use your own containerd".

It so happens that Docker uses containerd under the hood, which means that TrueNAS already has containerd running.

Run with --container-runtime-endpoint unix:///run/containerd/containerd.sock

That option didn't work either; The version of containerd used in TrueNAS 25.04 was too old and thus incompatible with K3s.

I hope to be able to use this option one day. I will post an update if this starts working on a more recent version of TrueNAS.

At this stage, I was ready to throw in the towel. That's when I found the --docker option, and it worked! That's the secret sauce that makes K3s play nice with TrueNAS, since Docker is already configured to work correctly.

© 2025 Onigoetz.ch