Two weeks ago we turned a spare MacBook Air into our CI. One 8 GB M1 laptop called CASE, a Linux VM, and GitHub's self-hosted runners. It worked. It also had a single point of failure with a keyboard attached.

So CASE got a twin, TARS, a second office M1 Air with 8 GB. Then we let the team's laptops help when they are plugged in and idle. This post covers how the three pieces fit, how we keep production off laptops, and the five things that went wrong. The last part is the useful part.

The shape of it

Two Airs, zero routing code

TARS runs the same setup script as CASE, with the machine name as a variable. The runner names differ (<org>-airlock on CASE, <org>-tars on TARS). The labels are identical.

That is the whole load balancer. GitHub hands each job to whichever matching runner is idle. To check, we dispatched three canary runs at once. Two landed on TARS, one on CASE.

Each Air runs the same stack:

  • a Colima Linux VM with 6 GiB of the 8 GB, and Rosetta for amd64 images
  • two job slots, enforced by a runner job hook (more on that below)
  • a watchdog that restarts the VM if it hangs
  • a canary workflow dispatched every hour

The label scheme

Labels decide where a job may run. There are three layers:

  • self-hosted, linux, airlock: any of our runners, laptops included
  • an org or lane label, like featureos or featureos-main
  • case: only the two office Airs carry it

The rule is short. Anything that deploys, migrates or touches prod secrets requires case. Laptop runners never get it. A lid closing halfway through a rollout is not a failure mode we accept, so we made it impossible instead of unlikely.

FeatureOS picks its lane from the branch:

jobs:
  test:
    runs-on: ${{ github.ref == 'refs/heads/main'
      && fromJSON('["self-hosted","linux","airlock","featureos-main","case"]')
      || fromJSON('["self-hosted","linux","airlock","featureos"]') }}
 
  deploy:
    runs-on: [self-hosted, linux, airlock, featureos-main, case]

PR jobs can land on a laptop. Main and deploys stay in the office.

Slots: two jobs per 8 GB

GitHub thinks of a runner as one job at a time. An 8 GB Air can do a bit more than that, but not much. So each machine has two "slots", and a runner job hook claims them before the job starts.

  • Light jobs (lint, unit tests, type checks) take one slot and share the machine.
  • Heavy jobs (Docker image builds, the Rails rspec suite) take both.
  • When jobs wait, they go in priority order: the FeatureOS main lane, then FeatureOS PRs, then SupportWire, then internal repos.

The hook is a small script, and it works. It also created incident number five, so hold that thought.

The laptop mesh

My MacBook Pro runs an OrbStack VM with 20 of its 24 GB and five extra runners. They only serve the two monorepos, FeatureOS and SupportWire.

A laptop is not a server. It goes to sleep, it goes on a train, it runs on battery. So a LaunchAgent we call the gate runs every 60 seconds and decides whether the laptop may take jobs. Every check has to pass for five stable minutes:

  1. Kill switch and approved machines. Both come from our internal portal. If the portal can't be reached, the answer is no. It fails closed.
  2. AC power.
  3. VPN connected.
  4. Network quality. Packet loss and latency under limits.
  5. GitHub reachable.
  6. Load. The Mac isn't already busy with my own work.
  7. Free disk.
  8. Demand. Both office machines are busy, or jobs are queued. No demand, no laptop.

When a check fails, the gate stops idle runners. It never stops a runner in the middle of a job. A job that started finishes.

The heavy lane

There is one more laptop runner, separate from the five. It only takes long main-branch jobs, like rspec and image builds. Before starting it, the gate sends a macOS notification ("starts on this Mac in 1 min, keep the lid open"), waits one tick, then starts it. Fair warning beats a surprise fan.

A menu bar app on every team Mac shows the mesh state and sends a notification when a job starts and finishes, with a pass or fail emoji.

Did it help

Queue wait for FeatureOS and SupportWire monorepo jobs, comparing the two days before TARS with the first hours after:

Before After
Median wait 3 s 3 s
p90 wait 27 s 4 s
Worst wait 9.5 min 46 s

The median was always fine. The tail is what hurt, and the tail is what moved. After the change, the load split was 9 jobs on CASE, 9 on TARS and 7 on the laptop.

That is a small sample. A few hours is not a month, and we will look again when we have more data.

Keeping it in line

More runners means more ways for a workflow to pick the wrong one. So:

  • Org-wide workflow templates for CI and Deploy, with the right labels already in them.
  • A Claude Code skill pushed to every team Mac through our device updater, so coding agents write the correct runs-on the first time. It is the same idea as the rest of our AI-native engineering work: put the rule where the agent will read it.
  • A weekly lint that flags GitHub-hosted runner labels, deploy jobs missing case, and long artifact retention.
  • The Actions budget is $0 with a hard stop. If a job asks for a hosted runner, it waits instead of billing us.

What went wrong

1. 8.7 GB of artifacts nobody needed

One repo uploaded a Docker image tarball, about 392 MB, for every target on every run, kept for 7 days. It also pushed the same images to the registry. That came to 8.7 GB, 90% of our plan's Actions storage.

The fix: 1-day retention, no tarball when the image is published anyway, and the org default retention dropped from 90 days to 7.

2. The disk guard measured the wrong disk

Docker's data lives on the VM's 100 GB data disk. The cleanup script checked the 20 GB system disk, which was 81% full of other caches. So it kept deciding the disk was full and wiping a build cache that wasn't even on that disk.

We moved the caches to the data disk and pointed the guard at the Docker disk. The system disk went from 81% to 60%, and the build cache finally survives the night.

3. FileVault parked TARS at the unlock screen

We rebooted TARS remotely to clear a leftover network extension. FileVault did what FileVault does. The machine stopped at the unlock screen, with no network, until someone in the office typed the password.

The rule now: remote reboots only through fdesetup authrestart, which unlocks the disk once on the way back up.

4. QEMU hid behind a warm cache

One product's image builds used QEMU emulation (setup-qemu-action plus a buildx container driver). With a warm cache it looked fine, 8 to 17 minutes. Then one cache miss turned an Elixir release build into 59 minutes. It held both slots on one Air, and other jobs sat in "Set up runner" for up to 53 minutes.

The same machine already builds amd64 through Rosetta, using the Colima Docker daemon with driver: docker. That takes minutes:

- uses: docker/setup-buildx-action@v3
  with:
    driver: docker   # no QEMU, no container driver

So we dropped QEMU everywhere. For the record, GitHub-hosted builds of the same image took 0 to 3 minutes. We fixed it on our own hardware instead of going back.

5. Slot waits are invisible

This one is our own doing. GitHub assigns a job to a runner that looks idle. Then our slot hook makes the job wait on that machine, even when the other Air is free. GitHub can't see the wait, so it shows the job as running.

We built a stuck-job detector. Every 5 minutes, plain rules flag:

  • slot waits over 10 minutes
  • jobs queued over 10 minutes
  • jobs running longer than twice their median and over 15 minutes

Only flagged jobs go to Jev, the System One model from TypeSafe, for a typed verdict: {stuck, probability, reason}, with the reason from a fixed list. If it says stuck, we get a notification. The rules do the cheap filtering, and the model only judges the cases that need judging.

The first run caught a 74-minute API test job that usually takes 16.5. Jev turned out to be cautious. A 53-minute slot wait scored 0.38, so after testing we set the threshold at 0.3.

Jev watches every deploy

That detector turned into more than a fix for our own hook. Every deploy runs on an Air with case, and every job on the mesh goes through the detector every 5 minutes. So Jev is now the watcher for all our deploys.

A deploy that waits too long for a slot, sits in the queue, or runs far past its usual time gets flagged by the rules. Jev reads the flagged job and returns the same typed verdict: stuck or not, how likely, and a reason from a fixed list. If it says stuck, we hear about it before anyone asks why the release is late. Nobody has to sit and watch a rollout.

What is still not solved

  • Memory. Two 8 GB Airs are still memory bound. Heavy jobs run one at a time per machine.
  • Power cuts. A power cut with FileVault on still leaves TARS waiting for a human. CASE has FileVault off, so one machine always comes back on its own.
  • Routing around a full machine. Next, we want runners on a full machine to look busy to GitHub, so it sends the job to the free one instead of letting it wait in our hook.

The first post was about saving money. This one is about what happens after: the second machine, the rules that keep laptops honest, and the gaps you only find by running it. If you missed the start, it is here: we turned a spare MacBook Air into our CI.