Spinbox / field notes

Timeline

An experiment, eight months away, and ten weeks of building

The short version: an idea in October 2025, and an experiment until January: a containerd shim that ran each container in a microVM, image layers handed to the guest as EROFS, and a first control plane. From January to September a full-time job left the project almost no time. What you see in those months was done in spare hours, and most of it was withdrawn. The project as it stands began in September 2026: a little over two months in which the pieces became one system, picked up a host image, and the host image turned into a desktop distribution.

Commits per month

7,557 commits across fifteen repositories. Don't read too much into commits, though: most of the July bar is a storage engine I withdrew in August.

The experiment

Containers in microVMs, image layers as EROFS, and a first control plane that agents wrote from specifications. It proved the idea, and then I put it aside: EROFS layers gave way to a qcow2 chain, and little of the code survived.

October 2025

  1. Spinbox

    The first idea: containers in microVMs, layers as EROFS

    A private repository called beacon. The plan is a containerd shim that runs each container in a VM, and a snapshotter that hands the hypervisor image layers as files instead of mounting them on the host.

December 2025

  1. Spinbox

    First commit: a containerd shim that runs each container in a microVM

    On Cloud Hypervisor, under the name "beacon". TAP devices and dynamic addresses the same day.

  2. Spinbox

    QEMU arrives as a second backend

    Three weeks later it is the only one.

  3. Spinbox

    CPU hotplug over QMP

    The first runtime resize of a running machine.

  4. Spinbox

    Three TAP strategies in one day

    Ends with the TAP opened inside the sandbox namespace and handed to QEMU as a file descriptor: the first FD passing.

  5. Spinbox

    Memory hotplug; beacon is renamed qemubox

    CI moves to self-hosted runners the same day.

  6. Spinbox

    Apache-2.0

January 2026

  1. Context

    The first idea, in public

    Posts on X about the EROFS snapshotter and the microVM prototype, on 2 and 7 January.

  2. Spinbox

    The EROFS snapshotter gets a repository of its own

    OCI layers converted to EROFS at pull, concatenated by a VMDK descriptor into one block device, mounted by the guest kernel. Archived in October: a disk is a qcow2 chain now.

  3. Spinbox

    "Initial design docs": the platform starts

    A .roadmap/ of 14 phase documents; control plane, runner, proxy, supervisor and dashboard follow in days. The first experiment in writing specifications and letting agents write the code.

  4. The factory

    A mock runner for the end-to-end tests

    Deleted nine months later, after it let a five-minute outage through every pull request.

  5. Spinbox

    First tagged release

    71 tags in six days.

  6. Spinbox

    The control plane goes quiet

    Its last commit until September. Snapshots are still "nerdctl commit and push to a registry".

January to September: a full-time job

The project got whatever hours a full-time job left, which for months was none. What you see here was done in spare hours, and the storage engine of July was withdrawn in August.

April 2026

  1. Spinbox

    Six commits in spring

    Shutdown is serialised; the guest file system is quiesced before power-off.

June 2026

  1. Spinbox

    Back: Ubuntu 26.04, QEMU 11, Linux 7.1; pause and resume

    Pause over QMP, file system freeze, and a hot commit of a running container: the first thing that looks like a snapshot.

  2. Spinbox

    Boot profiling; the README goes from "~300ms" to "~90ms"

July 2026

  1. Spinbox

    devcontainer CLI in Go

    A single static binary, checked for parity against the official TypeScript CLI at 0.88.

  2. Spinbox

    Storage: "architecture v5.1" and a simulation harness

    Remote volumes with cold storage in S3. An own block engine: write-ahead log, chunk addressing, deterministic simulation testing.

  3. The factory

    194 commits in one day

    Phases 01 to 13 of the storage plan: crypto core, recovery from S3, snapshot catalog, clone, drain. Then twenty-odd audit merges closing durability gaps.

  4. Spinbox

    An own vhost-user-blk backend

    Committed as "wip, unreviewed and unverified". The four failing tests that followed: a deadlock, a Serve that never returns, a 4 GB allocation from an unvalidated integer, a desynchronised socket.

August 2026

  1. The factory

    "Build thin end to end first"

    The agent instructions are rewritten after seven defects, all found after unit tests, property tests, simulation and invariant checkers existed. One of them: a promoted host that served zeros for a whole volume, with no error anywhere.

  2. The factory

    Agents sharing one git index overwrite each other

    Five parallel tracks: one commits a stale copy and deletes 121 lines of another ten seconds after they land. Ownership moves from sections to files, then to one worktree per branch.

  3. Spinbox

    v6 replaces v5.1: qcow2 plus immutable commits

    A month of block engine is withdrawn. The data path goes back to QEMU; what remains ours is the chain, the commits and the object store.

  4. Spinbox

    A snapshot under a guest that never stops writing

    The tip is sealed, a new one starts.

  5. Spinbox

    A volume rebuilt from the object store alone, on an empty host

The project, from September

A little over two months of full-time work: one repository, a hardened sandbox on AWS, a host image, and a desktop distribution that came out of it.

September 2026

  1. Spinbox

    Boot archaeology

    41.6 ms of initramfs unpacking that three rounds of initcall profiling missed; 14.7 ms asleep in a backoff; microseconds per byte on an emulated UART.

  2. Spinbox

    Machine templates: restore a frozen VM instead of booting one

  3. Spinbox

    spin-machine: QEMU, guest kernel and base image in one pinned artefact

    QEMU static against musl; virtio-mem replaces DIMM hotplug, so memory can be given back.

  4. Spinbox

    One repository: spinbox, spin and storage merge

    The merge audit finds that storage's CI had never run: its workflow lived in a directory GitHub does not read.

  5. Spinbox

    A network namespace per machine; nothing on the host

    TAP inside the namespace, the NIC hot-plugged on an inherited descriptor, one egress namespace per host holding every nftables rule. A pool of already-running machines the same day: 55.9 ms against 104.7.

  6. Spinbox

    "An image is a snapshot": no registry

  7. The factory

    "25k lines of markdown become 9k"

    The roadmap is deleted. One document said security hardening was "not started, 0/18"; all six critical items in it were closed.

  8. Spinbox

    Two thirds of the boot was systemd waiting for a terminal to answer

    TERM=dumb: 855 ms becomes 163–207 ms. The QEMU device allowlist the same day: 363 device types to 225, 23.7 MB to 8.7.

  9. Spinbox

    Network policy by tag

  10. Spinbox

    virtio-mem: memory granted and taken back on a real machine

    Two weeks later, asking for all of it back panics the guest ten runs in ten; the shrink gets a floor.

  11. Spinbox

    A drained host's workspaces resume elsewhere, from their memory

    Guest paused 40–54 ms for a checkpoint; 700 MB written in about 260 ms. Not live migration: suspend to the object store, resume on another host.

  12. Spinbox

    Egress reporting: what a workspace reached, by name

  13. Spinbox

    QEMU stops running as root

    A uid per machine, a chroot with no /dev, everything handed over as file descriptors.

  14. The factory

    Mutation testing becomes a gate

    After "tests that could not fail". Mandatory before pushing Go from then on.

  15. The factory

    CI moves to rented runners (Blacksmith)

    The first of four moves in five weeks.

  16. Spinbox

    An outside review of the QEMU boundary

    QEMU's own sandbox does not deny socket and connect; a network namespace does not scope vsock. Each reproduced on a real machine, then fixed: an empty namespace, vsock local mode, qemu-img under Landlock and seccomp.

  17. Spinbox

    First AWS installation

    Terraform for a control plane and a spot group of runners, moved to its own module the next day.

  18. The factory

    The first skill: measure CI from its real run history

    Test audit, test performance and a complexity review follow within the week.

  19. Spinbox

    The control plane's clock was 1.6 s off on AWS

    Found only on a real installation.

  20. Construct Linux

    Spin OS: a host image from Wolfi packages

    No cloud-init, no sshd, no getty. By the end of the day: EROFS root under dm-verity with no initrd, a UKI signed by a KMS key, and PCR values computed before the machine exists.

  21. The factory

    versions.yaml and the pin-keeper agent

    Every pinned input in one file; an agent that moves one entry per commit, runs its checks, and stops for a person on a major version.

  22. Construct Linux

    A host kernel with no module support at all

    Built with Clang, kCFI on.

  23. Spinbox

    Four kernel patches for 36 ms

    ftrace, tracing eval maps, PCI ECAM: 109.4 ms to 73.7 ms of kernel boot.

  24. The factory

    113 pull requests merged in one day

    38 of them in one hour; 300 workflow runs. "A test double is generated, never written": a 195-entry allow-list emptied in sixteen pull requests.

  25. Construct Linux

    Self-updating bare metal: built and deleted the same day

    A/B slots and boot blessing for a CI machine. Removed hours later. The code comes back the next day as the desktop's update path.

  26. Construct Linux

    The desktop starts: GNOME Shell 51 on Wolfi, in QEMU, on day one

    It begins with a tool that measures how far Wolfi is from a GNOME desktop. 73 package recipes by the end of the day.

October 2026

  1. Spinbox

    A runner joins only as an attested build

    NitroTPM attestation.

  2. Construct Linux

    A bootable USB stick: Secure Boot, UKI, verified root, A/B updates

  3. Construct Linux

    It is called Construct

    After "spin-desktop" and one day as "Spindrift". First commits for real hardware: a ThinkPad X9.

  4. Construct Linux

    A YubiKey opens the disk and answers sudo

    Fingerprint login is removed and restored the same day, with a rule: a finger stands beside the password, never in its place.

  5. Construct Linux

    First signed release, five days after the first commit

    197 package recipes.

  6. The factory

    CI leaves rented runners: $224 in nine days

    The pull-request gate moves to the developer's machine and reports a commit status. Specifications become issues.

  7. The factory

    A stop rule for security work

    "A month in which half the lines added were security": each mechanism had added a way for a workspace not to start. The rule is written for agents.

  8. The factory

    The end-to-end tests get a real runner; the mock is deleted

  9. Construct Linux

    The repository is prepared to go public

    Apache-2.0, a security policy, CI on pull requests. 125 commits that day.

  10. Construct Linux

    A regenerated certificate locks every enrolled machine out of a release

    Same keys, new serial numbers. Firmware stores the certificate, not the key.

  11. The factory

    CI had been red for days with nobody looking

    The free runners ran out of disk. Nothing could be released. CI moves to one self-hosted box.

  12. Spinbox

    Prebuilds

    On a real host each fix exposed the next: a snapshot with git refs still in the page cache, then sshd host keys on every disk.

  13. The factory

    The instructions file gets a byte budget

    Every session loads it whole. 34,635 bytes to 29,465, and a gate that fails it over 30,720.

  14. Construct Linux

    Wolfi silently replaced one of our packages

    Every package is pinned in a lock file from then on, with a weekly pull request to move it.

  15. Construct Linux

    Five file descriptors leaked per unlock

    After days of suspends a laptop could not be unlocked. Six commits across GDM and gnome-shell.

  16. The factory

    239 commits in one day

    The busiest day in the spin repository.

  17. The factory

    CI moves a fourth time: an EC2 spot instance per job

    The self-hosted box had three runner accounts and ran out of processes.

  18. Construct Linux

    Release AMIs are public

  19. The factory

    The merge train that never ran is removed

    63 runs in two days, all skipped: it waited for a GitHub App nobody made. Pull requests still land by hand.

November 2026

  1. Spinbox

    Bandwidth changed on a running workspace

  2. Spinbox

    A VS Code extension, installed by the CLI and signature-checked

  3. The factory

    A spot reclaim stalls every gate

    206 on-demand fallbacks, 2,765 launches refused by quota, gates waiting a median of 32 minutes.

  4. The factory

    REVIEW.md: rules for a reviewer, human or agent

    "A flake claim without runs" is itself a finding. At most three optional findings per review.

  5. Spinbox

    Release v20261110.01

    The 144th tag.