In October we added tests to every repo at Skcript. That was the right call. Our GitHub Actions bill tripled, which was the expected side effect. We could have paid it. Instead we took a spare M1 MacBook Air from the office, gave it a Linux VM, and made it our CI.

We named the laptop CASE, after the other robot in Interstellar. The runners on it are called airlock. This post covers what we saved, what got faster (and slower), how we kept it safe, and the engineering problems that ate most of the week.

The bill

Our hosted runners came from Blacksmith, a provider that runs GitHub Actions jobs on faster machines. From 1 to 26 October it charged us $93.25. When I opened the breakdown, 83% of that was "sticky disks", which is cache storage. Compute was the small part. We were mostly paying to keep caches warm between jobs.

There was a second problem. Invoices had gone overdue and nothing told us, because we had no billing alert. That one is on us.

GitHub's own minutes had a different trap. One of our orgs is on the free plan, and we had a synthetic health check running every 15 minutes. Each run took 41 seconds, and GitHub rounds every run up to a whole minute. Ninety-six runs a day comes to roughly 2,900 billed minutes a month, more than the free plan's 2,000. That check now runs once after each successful deploy, where it takes 76 seconds. Another org had used 905 of its 2,000 free minutes in October before we moved it.

Interactive

What would your CI cost?

Monthly cost on hosted runners vs airlock, over 30 days. The rate and free minutes are assumptions you can change.

Billed minutes per month4,800

Hosted runners$22.40

Airlock$0.00

Self-hosted minutes are not metered. Airlock runs on a laptop we already owned, so the line that stays is electricity.

We looked at three other options first:

  • Hetzner cloud VMs. A CX33 or CX43 would do the job, but it is a monthly fee forever, and we were trying to make a monthly fee go away.
  • A 32 GB Linux PC for the office. The right machine, at the wrong price. RAM is expensive right now.
  • Drone or Jenkins. Both work, but we would be running a second CI system and rewriting every workflow. Self-hosted GitHub runners keep the workflows we already have. You change runs-on and that is most of the migration.

After the move, hosted-runner minutes for the migrated repos are $0. GitHub does not meter self-hosted minutes. The hardware was a laptop that was already sitting on a shelf. Electricity is a rounding error.

The machine

CASE is an M1 MacBook Air with 8 GB of RAM. It runs a Linux arm64 VM through Colima, using Apple's Virtualization.framework (the "vz" type), Rosetta for amd64 binaries and virtiofs for file sharing. The VM gets:

  • 6 vCPU
  • 5 GiB of RAM plus 4 GB of swap
  • a 19 GB root disk
  • a 100 GB data disk for Docker

The swap matters. The first VM had 4 GB and no room to breathe, and it ran out of memory on a vite build. Swap turned an out-of-memory kill into a slightly slower build.

The runners live inside the VM as systemd services, one per GitHub org. Colima starts from a system LaunchDaemon, which waits for Rosetta and the network, clears any wedged state from a previous run, and boots the VM. Nobody has to log in. We tested an unattended reboot. SSH came back in 31 seconds, Docker in 48, and the runners were online in 96.

The power settings are what you would expect for a server that happens to have a keyboard. It never sleeps, wakes on LAN, restarts after a freeze, and macOS auto-updates are off.

One detail shapes everything below. Our servers are x86, so every image ships as linux/amd64. CASE is arm64. Every build crosses architectures.

Speed

Here is the median job time before and after, with queue time excluded. "Before" is GitHub-hosted or Blacksmith, depending on the repo.

Interactive

Before and after airlock

Median job time, queue excluded. Before is GitHub-hosted or Blacksmith. Slower jobs are in red.

Hover or tab through a row for the raw numbers.

The big wins come from one change. The cache is now just a disk. The Docker daemon's own layer cache (buildx with driver: docker) and the bun and pip caches stay on disk between jobs. On hosted runners every job restored and saved its cache over the network with type=gha. On CASE the cache is already there when the job starts.

That is how the dashboard (SvelteKit) build and deploy went from 193 seconds to 43, and its tests from 555 to 219. The AI agent service (TypeScript/Bun) image build went from 196 to 76, and its lint, test and type-check jobs from 42 to 52 seconds down to 16 to 19.

Some jobs got slower, and the chart shows them in red. The core API (Elixir/Phoenix) tests went from 177 seconds to 212. The widget (Svelte) tests went from 46 to 103, the MCP server (Python) tests from 25 to 47, and a tiny Node check from 7 to 16. Two causes. Some of those runs had to download a cold toolchain first, and on short jobs a faster hosted CPU wins. The MCP server image build was about even, 100 seconds to 107.

The feedback app (Next.js) lint had a Blacksmith median of 198 seconds over 16 runs. On airlock the median of 6 runs is 175. Its prod image build and deploy took more work, which is the next section.

There is one real trade-off. Each GitHub org gets one runner process, so jobs from the same org queue behind each other. Most of the time the wait is about zero. The day we merged several PRs at once, jobs waited between 3 and 25 minutes. We can live with that.

The one that got slower, and how we fixed it

The feedback app (Next.js) prod image build had a Blacksmith median of 167 seconds over 24 runs. The first, cold build on airlock took 530. Most of that was the dependency install at 272 seconds, with a separate x64-only install of 171 seconds running in parallel. next build took 93 seconds and the asset upload 40.

A cold build is allowed to be slow. The warm one was the problem. With the dependency layers cached and a code change on top, the build still took 216 seconds, because next build started from scratch every time (107 seconds). Docker's layer cache works per instruction. When the source changes, the layer that runs next build is rebuilt from nothing, and Next's own incremental cache in .next/cache goes with it.

BuildKit cache mounts fix that. A cache mount is a directory that survives between builds but never ends up in the image:

RUN --mount=type=cache,target=/root/.bun/install/cache \
    bun install --frozen-lockfile
 
RUN --mount=type=cache,target=/app/.next/cache \
    bun run build

A warm build with a code change now takes 142 seconds, and next build takes 34 of them. That is faster than Blacksmith's 167. A lockfile change also reuses the cached packages instead of paying for the 270-second cold install again. Most of what is left is the asset upload (about 40 seconds) and the image export and push (about 20).

The deploy did get slower, from 85 seconds on GitHub-hosted runners to 119 on airlock, measured on one run. That time includes a gate that loads the new build in a real browser and checks it hydrates before any traffic switches. We are keeping the gate.

The engineering problems

This is where the week went. Building amd64 images on an arm64 laptop works until it doesn't, and it usually doesn't in a new and specific way.

Bun under Rosetta crashes with SIGILL

Rosetta translates x86 instructions, but it does not support AVX2. Bun's x64 binary uses AVX2, so it dies with SIGILL as soon as it starts. Anything that ran bun install inside the amd64 stage of a Dockerfile failed.

The fix was to stop running Bun as amd64 at all. Every Bun stage runs on the build platform, which is native arm64. Production dependencies install natively, but for the target platform:

FROM --platform=$BUILDPLATFORM oven/bun:1.4.3-alpine AS deps
ARG TARGETARCH
WORKDIR /app
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile --production --ignore-scripts \
    --os=linux --libc=musl --cpu="${TARGETARCH/amd64/x64}"

The final amd64 stage only COPYs. It never executes anything at build time, so it never needs Rosetta.

Interactive

One Dockerfile, two architectures

Four stages run natively on CASE. Only the last one is amd64, and it only copies.

arm64, native on CASEamd64, the image we ship1. bun2. deps3. build4. target-deps5. runtimebun binarynode_modules.next/standalonex64 @img/sharp
FROM --platform=$BUILDPLATFORM oven/bun:1.4.3-alpine AS bun

A named stage keeps the native arm64 bun. COPY --from=oven/bun would have pulled the amd64 one.

Simplified for the diagram.

COPY --from pulls the wrong architecture

The first version of that fix copied the bun binary straight from the image:

COPY --from=oven/bun:1.4.3-alpine /usr/local/bin/bun /usr/local/bin/bun

That pulls the image for the target architecture, even inside a stage pinned to the build platform. So we were back to an amd64 bun. A named stage fixes it, because the named stage carries its own platform:

FROM --platform=$BUILDPLATFORM oven/bun:1.4.3-alpine AS bun
 
FROM --platform=$BUILDPLATFORM node:22-alpine AS build
COPY --from=bun /usr/local/bin/bun /usr/local/bin/bun

Next.js standalone only traces the host's sharp

Next.js standalone output traces the files the app needs, and that includes the sharp native binaries. It traces the ones for the machine that ran the build, which on CASE is arm64. The runtime image is amd64 and needs the x64 @img/* packages. We install those in a target-arch stage and copy them into the runtime image next to the standalone output.

The Erlang JIT and an already-compiled module

The core API image build failed under Rosetta with this:

could not call Module.put_attribute/3 because the module ... is already compiled

The error points at Elixir, but the cause was the BEAM JIT running under Rosetta. Telling the JIT to use a single memory mapping in the builder stage fixed it:

ENV ERL_FLAGS="+JMsingle true"

Rust under Rosetta takes 11 minutes

The docs engine (Rust) release built in 68 to 129 seconds on Blacksmith. On airlock, with rustc running under Rosetta, it took 661 seconds cold.

So we compile natively on arm64 and only link for x86_64, using the cross linker. Roughly:

FROM --platform=$BUILDPLATFORM rust:1 AS build
RUN apt-get update && apt-get install -y gcc-x86-64-linux-gnu \
 && rustup target add x86_64-unknown-linux-gnu
ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
RUN cargo build --release --target x86_64-unknown-linux-gnu

Cold builds dropped from 661 seconds to 256. A typical code change takes about 95 seconds, and a build where nothing changed takes 31.

vitest ran under Bun by accident

The VM had no Node installed. npx vitest still worked, sort of, because Bun stepped in. Module loading broke in a way that took a while to see. import { z } from 'zod' gave undefined. We installed Node 22 in the VM to match GitHub's images, and the tests went back to passing for the boring reasons.

actions/cache hung for 8 minutes

On the self-hosted runner, actions/cache restores sometimes hung for 8 minutes or more. We did not debug it for long. The runner keeps its caches on disk already, so the step was doing nothing useful. We removed it.

The disk filled up

A persistent cache is only an advantage until it fills the disk. The 19 GB VM root reached 87%. Most of it was a 5.5 GB bun cache and the 4 GB swap file.

We moved the bun and Playwright caches to the data disk, using symlinks into /mnt/lima-colima/ci-cache. The obvious place was next to Docker's data in /var/lib/docker, but the runner user cannot traverse that directory. A daily prune now:

  • deletes images older than 48 hours
  • caps the build cache at 30 GB, dropping the least recently used entries first
  • removes bun packages nobody has touched in 7 days

Root disk usage went from 87% to 64%.

Persistent runners find flaky tests

A hosted runner starts every job on a fresh machine. A runner that lives for weeks doesn't, and timing on it is different. One test assumed Postgres returned rows in insertion order. Postgres never promised that, and on CASE it stopped happening. The test was wrong, so we fixed the test.

Safety

A hosted runner is thrown away after every job. CASE is not. Anything a job leaves on disk is still there for the next job, so we had to make sure secrets never land on disk in the first place.

Build secrets. Our Dockerfiles used to take the npm token, the Sentry token and AWS keys as ARG or ENV. Those values end up in the build cache metadata, and on CASE the build cache is persistent. We moved them all to BuildKit secret mounts. The secret is available to one RUN step and never written to a layer:

RUN --mount=type=secret,id=NODE_AUTH_TOKEN,env=NODE_AUTH_TOKEN \
    bun install --frozen-lockfile
- uses: docker/build-push-action@v6
  with:
    secrets: |
      NODE_AUTH_TOKEN=${{ secrets.NODE_AUTH_TOKEN }}

Docker's SecretsUsedInArgOrEnv warnings went from 9 to 0.

SSH deploy keys. The repos that deploy over SSH write the key at the start of the job. They overwrite the SSH config instead of appending to it, so one job's host entry never leaks into the next, and they delete the key in a step that runs even when the job fails:

- name: Remove deploy key
  if: always()
  run: rm -f ~/.ssh/deploy_key ~/.ssh/config

We checked after a deploy, and nothing is left behind. Most of our deploys don't use SSH at all. They call a Coolify webhook, so no key is involved.

The network. CASE sits on a private network with access rules that let it reach nothing else. If someone gets a shell on it, there is nowhere to go from there. Remote access needs a fresh re-authentication every time, so even we have to prove who we are before we get in. On the office network it has a static DHCP lease in its own "CI Runners" group.

Runner tokens. Registering a runner needs a short-lived token. We pass it only in the environment of the one-off registration command. It is never written to a file.

No GitHub token on CASE. The dashboard described below reads the runners' own diagnostic logs. It does not call the GitHub API, so there is no token for it to leak.

Prod deploys stay gated. Moving CI did not change how we ship. The new image starts next to the running prod build. It gets probed, then hydration-tested in a real browser, before any traffic moves. If any check fails, the old build keeps serving. A synthetic check runs after every deploy.

The risks we still carry:

  • It is one machine. If CASE dies, CI stops until we fix it or switch runs-on back. A UPS and a wired ethernet connection are next on the list.
  • One runner per org means queues on busy days, as covered above.

Watching it

CASE runs a small dashboard of its own. Glances provides system metrics. A job that runs every 15 seconds parses the runner logs into JSON and keeps 90 days of history. The page shows jobs, pass rate, build time and the last deploy, filtered by time range and org. Daily usage metrics are backed up to R2, so if the laptop goes, the history doesn't go with it.

Would we do it again

Yes. Most of the bill was cache storage, and a laptop with a big disk is very good at storing caches. The builds that mattered got faster. A few short jobs got slower, and we are fine with that.

The hard part was the architecture gap. Every problem in the middle of this post came from building amd64 images on an arm64 machine. The fixes are short once you know them, so they are all written down above.

If you liked this, we did something similar for error tracking with OpenFault, our self-hosted Sentry-compatible tracker. We also measure before we trust, which is why we built an LLM latency benchmark. And for how far we have come on CI, here is why we moved from CircleCI to Netlify years ago.