Steve Francois
Steve Francis ran data centers for SaaS companies such as Citrix Online and Fastclick for years, then was the founder of LogicMonitor.com, a SaaS platform serving medium and large enterprises for data center monitoring. data. He is now the CEO of Sidero Labs.
For those unaware, immutable operating systems have recently gained popularity. An immutable operating system is one in which some or all of the operating system file systems are read-only and cannot be modified.
Immutable operating systems have many advantages. They are inherently more secure, as many attacks and exploits depend on writing or modifying files. Additionally, even if an exploit is found, malicious actors cannot modify the operating system on disk (which in itself will thwart attacks that rely on writing to the filesystem), so a reboot will erase any memory-resident malware and will restore a non-exploited state.
Immutable systems are also easier to maintain and update: operating system images are not patched or updated, but replaced atomically (in a single operation whose complete completion or failure complete is guaranteed – no partial upgrades!)
Immutable systems can also claim to be more stable than traditional operating systems, simply by eliminating many vectors that introduce instability into a system – most of which are human. No sysadmin can “just change this setting to fix things” – with unforeseen impacts that aren’t discovered until hours later. (I was that sysadmin.) No partially complete terraform or puppet executions that leave systems in weird states…
On the workstation side, there are immutable operating system approaches such as rpm-ostree. This attempts to create immutable, image-based deployments in the OS, but overlays a flexible filesystem architecture, so packages can still be managed and updated by RPM.
On the server side, there is a specter of immutability among container-specific operating systems. All support image-based OS updates and no package manager. Some operating systems such as Flatcar Linux do /usr read-only, but allows common runtime modifications such as dynamically loading kernel modules and overriding systemd configurations.
Bottlerocket, from Amazon Web Services, adds the read-only /root partition and disables SSH access by default (it can run in a privileged container). Bottlerocket allows kernel modules to be loaded in older versions, but has moved to only allow modules signed with the Bottlerocket image key – modules that ship with Bottlerocket.
Talos Linux, from Sidero Labs, takes the immutable concept one step further. It’s another OS with a fully immutable filesystem and full API for management, and unlike Bottlerocket, completely removes SSH and console access, and runs wherever you run Kubernetes – all cloud providers, bare metal, virtualized systems, even in Docker and on SBCs like Raspberry Pis. Talos Linux also requires kernel modules to be signed with the same key used to build the kernel – and since this key is ephemeral, it makes the kernel completely static and immutable.
Of course, as with all things, the inherent strengths of immutable operating systems are also their weaknesses. Not all systems need to run the same workloads, firmware, and packages. How do you customize an immutable operating system to suit a variety of applications?
Flatcar Linux, having less immutable characteristics (the root filesystem is writable and the nodes support SSH, for example), solves this problem with the Ignition project (often in conjunction with Afterburn), which runs at first boot of a node, and can modify the file system, add files, configure users and other operations. This is similar to a typical Linux workflow where tools like Puppet, Chef, etc. are used (except that it can only run once on first boot).
Similarly, Bottlerocket uses the concept of Bootstrap containers to customize how the node works. These are containers that are run by systemd after the node boots, who have access to the root filesystem and host devices. They can run on every boot or only on first boot. The order of these containers is non-deterministic. Thus, bootstrap containers are effectively unbounded by filesystem immutability, which somewhat reduces the effectiveness of immutability for security purposes.
System extensions
Talos Linux, being completely immutable, needs to take a stricter approach to modifying nodes. Talos Linux uses a concept called System Extensions. It can work while maintaining full operating system immutability because it requires such extensions to be configured when installing or updating the node. In order to make any changes to system extensions, Talos Linux must be reinstalled on the node.
This encourages (actually reinforces!) the idea of ”pets, not livestock”, because to add or modify a node via system extensions, the contents of the disk will be erased and overwritten. When installing/reinstalling, the system extension container images will be stored in the initramfs like squashfs pictures; when booting they are layered on top of Talos Linux squashfs read-only root filesystem squashfs overlays mounted. All file systems (except /var for Kubernetes data) are immutable thereafter.
The concept of extending systems keeps file systems immutable while allowing nodes to load different features, such as specific firmware or gVisor for additional container isolation.
These different approaches to enabling modification of immutable systems have different strengths, but allow the user to select what suits them: from managing systems in a familiar way (as with rpm-ostreeor using SSH) to new paradigms such as the one implemented by Talos Linux, where there is no SSH, all management is done by API, and security is paramount – with the requirement to learn a new way of manage systems.
The New Stack is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.
Featured image by Lutz Krüger from Pixabay.
