Skip to content
A-Line

Back to guides

Talos Linux: what an immutable OS changes for your nodes

By Arthur De WittePublished Updated 10 min read

On this page

Most teams patch their Kubernetes control plane on schedule and scan every container image they ship. The operating system under the nodes gets less attention: a stock Ubuntu install, SSH open for the day something breaks, and a configuration tool that runs now and then. That OS is where an attacker who escapes a container lands, and where small manual fixes pile up until no two nodes are quite the same. This guide covers Talos Linux, an OS built to run Kubernetes and nothing else, and what changes when you use it.

What “immutable” means for a node

On a regular distribution every node starts identical and then drifts. Someone installs a debugging tool during an incident and never removes it. A configuration run fails halfway on one machine. A kernel setting gets changed by hand to test a theory. After a year, the nodes in one pool behave slightly differently, and nobody can say exactly how.

An immutable OS stops this by making the system itself read-only. You don’t change a running node; you change its description and roll out a new one. How far that goes differs per OS:

  • Flatcar Container Linux keeps /usr read-only, but the rest of the system is writable and SSH works as usual.
  • Bottlerocket makes the whole root filesystem read-only and turns SSH off by default. For debugging you start a separate admin container.
  • Talos Linux runs entirely from a compressed, read-only image. The few files under /etc that Kubernetes needs are generated from the machine configuration, and extra drivers or tools are added as system extensions when the image is built, not installed afterwards.

What a general-purpose OS brings along

A Kubernetes node needs a kernel, a container runtime, the kubelet and some networking. A standard server install ships much more: shells, compilers, mail tools, language runtimes. Every package you don’t use still receives security advisories, and each one has to be read, judged and patched.

The kernel itself is busy enough. In 2025 it received 5,530 CVEs, 28% more than the year before (CIQ). Talos runs a Linux kernel too, so a smaller OS doesn’t make those go away. What it changes is the rest of the list, and how much work it takes to put a fixed kernel on every node.

NIST’s guide on container security recommends exactly this: run containers on an OS built for them, because a smaller OS leaves less to attack (NIST SP 800-190).

How Talos works

Talos replaces most of a normal Linux userland with one program. machined starts as the first process and runs everything else: the container runtime, the kubelet and a handful of system services. There is no systemd, no bash and no coreutils.

You talk to a node through an API. Every node exposes a gRPC API secured with mutual TLS, and talosctl is the client. What you’d do over SSH elsewhere is a command here:

talosctl --nodes 10.0.0.5 logs kubelet     # a service's logs
talosctl --nodes 10.0.0.5 dmesg            # kernel messages
talosctl --nodes 10.0.0.5 pcap --interface eth0   # capture traffic
talosctl --nodes 10.0.0.5 dashboard        # live overview of the node

The node is a configuration file. One YAML document describes the machine: its role, network, disks and the Kubernetes version it runs. A worker can look like this:

version: v1alpha1
machine:
  type: worker
  network:
    hostname: worker-01
    interfaces:
      - interface: eth0
        dhcp: true
  kubelet:
    image: ghcr.io/siderolabs/kubelet:v1.34.0
cluster:
  controlPlane:
    endpoint: https://10.0.0.10:6443

The real file is longer. talosctl gen config generates it, including the certificates and tokens the node needs to join the cluster. Keep it in version control like any other configuration, with the secrets encrypted.

The result is a very small system. Sidero Labs, who make Talos, counted the programs a default install puts on a node:

Programs on a default install, by node OSRough counts, fewer means less for an attacker to use. Counted by Sidero Labs
  • Ubuntu3,000+
  • Flatcar2,300+
  • Bottlerocket~200
  • Talos Linux12
Show the numbers as a table
Node OSPrograms (approx.)
Ubuntu3,000+
Flatcar2,300+
Bottlerocket~200
Talos Linux12

What an attacker finds on a Talos node

Say someone escapes a container and reaches the host. On a regular node they find a shell, curl, a package manager and often an SSH key that opens the next machine. On Talos there is no shell to start and almost nothing to run. The OS can’t be changed, and a reboot through the API brings the node back from its image.

This doesn’t make a node unbreakable. A kernel exploit is still a kernel exploit, and data under /var, such as container images and pod volumes, survives a reboot. But most of the steps that come after a break-in depend on tools and writable places that Talos doesn’t have.

What an attacker tries General-purpose distribution Talos Linux
Move to other nodes over SSH Works if a key or password is found No SSH server on the node
Use tools already on the node Thousands of programs to choose from 12 programs and no shell
Leave something that survives a reboot System directories are writable The OS is read-only and comes back from its image
Change configuration by hand Lasts until someone notices Only possible through the API, from the config file
Add a local user account Edit /etc/passwd There are no user accounts

Talos, Bottlerocket or Flatcar

All three are built for containers. They differ in how strict they are, and that decides where each fits.

OS What is read-only Shell access Runs Fits best
Talos Linux The whole OS None, API only Kubernetes only Clusters on bare metal, in the cloud or at the edge
Bottlerocket The root filesystem Off by default, admin container when needed Containers, built for EKS and ECS Clusters on AWS
Flatcar /usr SSH as usual Any container workload Moving away from a regular distribution step by step

If you want the strictest setup and run Kubernetes anyway, Talos goes furthest. If you’re on AWS and want the provider’s tooling, Bottlerocket fits better. If your team still relies on SSH and you’d rather change one thing at a time, Flatcar is the gentler start.

Upgrades that can’t stop halfway

On a regular distribution an upgrade changes packages one by one on a running system. If it fails halfway, you have a node that is neither the old version nor the new one.

Talos upgrades the whole OS image at once:

talosctl upgrade --nodes 10.0.0.5 \
  --image ghcr.io/siderolabs/installer:v1.12.6
talosctl version --nodes 10.0.0.5

Talos cordons and drains the node, writes the new image next to the one it runs from, and reboots into it. If the new version doesn’t boot, the node starts the previous one again. Either way the node runs a complete, known version. Flatcar and Bottlerocket use the same two-slot idea; Talos adds that the entire OS, configuration included, comes from that one image.

Press a button to upgrade worker-01 from v1.12.5 to v1.12.6, or to see what happens when the new version won't boot.

Kubernetes itself is upgraded separately, with talosctl upgrade-k8s --to 1.34.1, which updates the control plane and the kubelets one by one. Upgrade one node first, check it, then do the rest. Draining only works out if your apps survive losing a pod; the guide on keeping a Discord bot online shows one way to build for that.

Who runs it in production

PostFinance, one of Switzerland’s largest banks, moved 35 Kubernetes clusters in an air-gapped environment from kubeadm and Ansible to Cluster API and Talos, without downtime (Camptocamp). They published part of the tooling they built for it as TOPF, a Talos orchestrator.

JYSK, the retailer, runs Talos in more than 3,400 stores across 48 countries. Each store’s node is built from the same image and receives its configuration at first boot (Sidero Labs).

Both cases have the same need: many machines that must stay identical, with few people able to log in to them.

What you give up

Host tools. There is no tcpdump, strace or htop on the node. talosctl pcap, logs and dashboard cover most needs. For the rest, kubectl debug node/worker-01 starts a pod with your own tools on that node. When a node can’t reach the API server, a traceroute from such a pod is often the quickest first look.

Agents that expect a regular host. DaemonSets that mount host paths such as /var/log/journal, or that load kernel modules themselves, may fail. Most monitoring and security agents have a Talos-ready setup, but check each one.

Habits. “Let me just log in and look” stops working, for everyone. Runbooks need rewriting, and the first incidents take longer while the team learns talosctl.

A smaller community. The project is active (GitHub), but there are far fewer forum answers and blog posts than for Ubuntu. Expect to read the documentation more.

Kubernetes only. If a machine also has to run something outside Kubernetes, Talos is the wrong OS for it.

Before you move a cluster

  1. List your DaemonSets and look for host-path mounts, privileged containers and anything that expects systemd or journald.
  2. Check your monitoring and logging agents for Talos support. Container logs are in /var/log/pods; Talos’s own service logs can be sent elsewhere with machine.logging in the configuration.
  3. List the kernel modules you load with lsmod on a current node. Those that aren’t in the default image come as system extensions, added when you build the image.
  4. Rewrite your runbooks so every step that used SSH has a talosctl or kubectl debug equivalent.
  5. Plan the move side by side. Build a Talos cluster next to the current one and move workloads over one at a time, so going back is a matter of moving them back.

Test your reading

Three situations from a Talos cluster. Decide for yourself whether each one shows a real problem, then open the answer.

SSH is refused

$ ssh admin@worker-03
ssh: connect to host worker-03 port 22: Connection refused
Is this a real problem?

No. Talos has no SSH server, so this is how every healthy node answers.

Nothing is broken; there is simply nothing listening on port 22. Ask the node through its API instead, for example with talosctl –nodes worker-03 logs kubelet or talosctl –nodes worker-03 dashboard.

The upgrade that didn't stick

node       worker-07
asked for  v1.12.6
running    v1.12.5
status     Ready
uptime     4 minutes
Is this a real problem?

Yes. The node rebooted but runs the old version: the new one didn't boot, and Talos fell back.

The node is healthy and serving, which is the fallback doing its job. But the upgrade failed, and it will fail on the other nodes too. Read talosctl –nodes worker-07 dmesg to see why the new version didn’t start; a missing system extension or a wrong kernel argument are common causes. Fix that before you upgrade anything else.

The log shipper that won't start

daemonset  log-shipper
pods       0 of 6 ready
status     CrashLoopBackOff
last log   open /var/log/journal: no such file or directory
Is this a real problem?

Yes. The agent expects journald, and Talos doesn't have it.

This is one of the agents you find in the checklist above. Point it at /var/log/pods for container logs, and have Talos send its own service logs to your log system with machine.logging in the machine configuration.

Doing this on A-Line.cloud

Everything above is work you can do yourself, and for a team that wants to own its nodes it’s worth learning. If you’d rather not run the platform underneath, A-Line.cloud does it for you on Talos:

  • Your own Talos cluster. No other customer’s workloads run in the cluster. Storage and builds are shared platform services, and nodes are shared-vCPU by default.
  • No nodes to manage. You deploy apps; the nodes underneath are ours to look after.
  • Rolling deploys that only continue once the new version is ready, and roll back when a deploy fails.
  • Builds from GitHub and certificates set up for you, with no Kubernetes YAML to write.

A-Line.cloud is invite-only for now. If you run apps on Kubernetes and would rather not look after the nodes, request access and tell us what you run today.

Frequently asked questions

Is Talos Linux free?

Yes. Talos is open source under the Mozilla Public License 2.0. Sidero Labs, the company behind it, sells Omni, a service for managing Talos clusters, and support. You don’t need either to run Talos.

Can I SSH into a Talos node?

No. Talos has no SSH server and no shell. Everything goes through the API with talosctl, and for hands-on debugging you start a pod on the node with kubectl debug.

What happens when a node crashes?

It reboots from the same image with the same configuration and rejoins the cluster. The OS comes back exactly as it was; data under /var, such as container images and volumes, stays.

How do I move an existing cluster to Talos?

Most teams build a new Talos cluster next to the current one and move workloads over, one app at a time. Larger setups, like PostFinance’s, use Cluster API to replace machines gradually.

Where to start

Put a single Talos worker in a test cluster and run your DaemonSets on it. DaemonSets that expect a regular host are the usual stumbling block. Whatever breaks on that one node is your migration list.