Self-hosting CI gets harder every month | Blacksmith

Self-hosting CI gets harder every month

More code is being written than ever. CI volume is outgrowing infrastructure that used to work.

CI demand comes in bursts

A busy hour can leave jobs waiting, while a quiet one leaves machines idle.

Demand changes by the minute

A fleet sized for a typical hour can run out of machines when everyone pushes at once.

Bursts fill the queue

When a burst uses your available machines, other jobs wait for the autoscaler.

Idle machines cost money

You’re paying to keep machines ready, even when there are no jobs to run.

Why do jobs queue even with autoscaling?

  1. Jobs queue while machines boot
    Demand jumps 4,000 vCPUs in five minutes. The autoscaler requests more machines, but none are online yet. 2,900 vCPUs of work sit in the queue.
  2. Capacity catches up, late
    New machines come online about three minutes after they’re needed. By then, demand has moved again.
  3. Capacity drops late, you pay for it
    Demand drops 12,000 vCPUs in five minutes. Capacity stays up for the 15-minute cooldown. That leaves 15,000 vCPUs running with nothing to do.

What it costs

  1. You wait for machines

Jobs queue while each new machine starts and registers, which can take minutes.

"Our self-hosted runners were having autoscaling issues and they’d be in queue for 5, 10 minutes. People would report that."
— LangChain, self-hosted runners on GCP

  1. You compete for capacity

The instance type you request can be out of stock, or you hit your vCPU quota. Either way, jobs stay queued.

"At the start of the year our self-hosted CI runner cluster would get m8 instances with a 90 percent success rate. It has dropped to 15 percent"
— Arian van Putten

  1. You pay to avoid waiting

A warm pool skips the wait, but you keep paying for it when CI load drops.

"I got sick of spending the time babysitting the warm pool and just took the scaling policies out, but it means you’re provisioning for the peaks."
— Megaport, self-hosted runners on AWS

  1. You lose spot machines mid-job

Spot instances are cheap until they’re reclaimed, and your job is cut off mid-run.

"I’m spending a whole quarter on it. If this ends up being cheaper than all of this, then I can spend my time on actually useful measures instead of tuning a GitHub runner thing."
— Amplitude, ARC + Karpenter on Kubernetes

The biggest cost is your team’s time

You pay for machines and the time your team spends maintaining them.

CI infra backlog

GitHub

  1. Handling API rate limits
  2. Updating runner versions
  3. Handling webhook delivery

Capacity

  1. Tuning autoscaling policies
  2. Asking AWS for quota bumps
  3. Isolating noisy neighbors
  4. Spot interruptions killing jobs mid-run

Images and caches

  1. Upgrading toolchains
  2. Patching base image CVEs
  3. Maintaining OS images
  4. Managing Docker and runtime caches

Network

  1. Avoiding Docker Hub limits
  2. Maintaining egress allowlists

Observability

  1. Tracking queue times
  2. Attributing job costs
  3. Measuring CPU and memory

Customer data

How long do jobs wait for a machine?

Why Blacksmith

Keep your workflows, let us manage everything else

Install the GitHub App and move the jobs you pick with a one-line change. Switching back is a simple revert.

- runs-on: [self-hosted, linux, x64]
+ runs-on: blacksmith-4vcpu-ubuntu-2404

What stays in GitHub

  1. Workflows
    Your workflow files stay in your repo. You choose which jobs move to Blacksmith.
  2. Secrets
    Secrets stay in GitHub and work with Blacksmith runners just as they do with your self-hosted runners.
  3. Permissions
    Who can run what, branch protection, and required reviews all stay in GitHub.

What moves to Blacksmith

  1. Runners
    High-performance, purpose-built compute with unlimited concurrency at no extra cost.
  2. Caches
    Dependencies, Docker layers, container images, and Git checkouts are cached with zero configuration.
  3. Operations
    We patch CVEs in base images, keep toolchain and dependency versions current, and pull unhealthy machines before your jobs land on them.
  4. Observability
    Every job’s logs, resource usage, and test failures in one place, with no agents to install.

Stop managing your own caching

Zero-config caching skips as much of every job as possible. What’s left runs on the fastest, purpose-built infrastructure.

Cache & reuse

  1. Git cache
    Checkout: 18s without cache, 3s with cache.
  2. Container images
    Initialize containers: 12s without cache, 9s with cache.
  3. Machine images
    Install dependencies: 42s without cache, 7s with cache.
  4. Build cache
    Compile and test: 48s without cache, 8s with cache.
  5. Docker layers
    Build and push: 54s without cache, 9s with cache.

How did job runtimes change after moving to Blacksmith?

Job runtimes

Blacksmith by the numbers