Runbooks That Run Themselves

Turning operational playbooks into AI skills

Most infrastructure teams have the same quiet problem. The riskiest work they do runs a handful of times a year, carries real blast radius, and depends on details someone learned the hard way. A CNI minor-version upgrade. A Kubernetes control-plane bump. A storage class migration. A certificate authority rotation. Each one is a procedure, and each procedure lives in the worst possible place: partly in a wiki page nobody has opened in eight months, partly in an old chat thread, and mostly in one engineer’s head.

We call these things runbooks, but most of them are not runbooks in any operational sense. They are notes. The next person who has to run the procedure either re-derives it from scratch or misses the one step that mattered. This post is about a pattern that fixes that, by making the runbook something you invoke rather than something you read.

The missing link: alert to runbook to skill

Good on-call setups already have half of this chain wired up. An alert fires and points at a runbook. The runbook explains what the alert means and what to check. That linkage is table stakes, and it works: when something pages you at 2am, you are not starting from a blank page.

But the chain stops one step too early. The runbook tells you what to do. It does not do it. You still open a terminal, remember the sequencing, remember the safety rules, and hope you do not fat-finger the order. The knowledge of how to execute safely is still tribal.

The pattern is to add the next link:

Alert  ->  Runbook  ->    Skill
fires,     explains it,   executes it,
points     points to      with guardrails
to its     its skill      baked in
runbook

Every runbook gets a skill attached, one to one. The alert names the runbook that explains it. The runbook names the skill that executes it. Now the chain is complete: detect, understand, act. The same discipline you already apply to alert-to-runbook links, you extend one more hop to runbook-to-skill.

What a skill actually is

In this context a skill is a small, self-contained folder that an AI coding agent (or a human) can read and execute step by step. It is not a chat prompt and it is not a wiki page. It is code-adjacent: it lives in the repository, it is reviewed like code, and it is versioned like code.

The anatomy is deliberately boring:

skills/
  upgrade-cni/
    SKILL.md              # the orchestrator
    references/
      per-version-changes.md
      inventory.md
      verify.md
      incidents.md

SKILL.md is the orchestrator. It holds three things: the guiding principle of the procedure, the non-negotiable hard rules, and the ordered steps. It is written to be stable. Once the methodology is right, it barely changes.

The references/ folder holds everything that changes over time and everything that is a fact rather than a method: the list of per-version breaking changes, the inventory of what you are operating on, the verification checklist, the log of past incidents and how they were handled.

The split that makes it last

The single most important design decision is keeping methodology and data apart.

Methodology goes in SKILL.md. Facts go in references/. When a new version of the thing you are upgrading ships, you add a row of data to a reference file. You do not touch the logic. Supporting next year’s release becomes a data edit, not a rewrite.

This is the difference between a skill that rots and a skill that compounds. If the version-specific gotchas are baked into the procedure itself, every new version means editing the procedure, and every edit is a chance to break the parts that were working. If the gotchas live in data, the procedure stays stable and trustworthy while the data grows.

The repeatable pattern

Once you write a few of these, a shape emerges. Almost every operational skill (an upgrade, a patch, a migration) follows the same seven phases. A new skill fills in the specifics. The shape stays the same, which is exactly why they feel familiar to run.

1. Scope and plan. Read the current state, build the change set (for a version upgrade, that is often a sequential ladder rather than a single jump), announce it, and get sign-off on anything ordering-sensitive.

2. Preflight gates. Run the read-only baseline and the checks that must pass before you touch anything: permissions, dependency compatibility, and snapshots of any state that lives in no repository. Commit these as artifacts so there is a record.

3. Execution model. Decide, per target, how the change actually lands. Through the normal pull-request-and-pipeline path, or through a controlled local run with reconciliation paused. Encode which environments allow which mode, so the operator cannot accidentally do the risky thing in the wrong place.

4. Execute one unit. One target at a time, one rung of the ladder at a time. Make the change, diff it read-only, review for only-expected changes, apply, and watch the rollout live. Never batch.

5. Verify. Run a fixed checklist identically every time. Crucially, verify end-to-end behavior, not just that the pods came back up. “The process restarted” is not the same as “the thing works.”

6. Land it. Reconcile the source of truth to the new state through a reviewed change, with the standard reviewers and the standard commit conventions. The cluster and the repo agree again.

7. Report. After each unit, update the tracking ticket with what was actually encountered, and keep the working log current. The paper trail is part of the deliverable, not an afterthought.

Guardrails, baked in

This is the real payoff, and it is worth being explicit about.

The safety knowledge that used to depend on the operator’s memory becomes part of the procedure. It holds even when the operator is tired, new, or under pressure at 2am. A well-written skill does not just describe the safety rules. It refuses to violate them.

A few examples of the kind of rule that gets encoded:

• Sequential only. If the migration path assumes you did not skip a version, the skill refuses to skip a version. The rule is not a sentence in a doc. It is a gate.

• Blast radius ordering. Test environment first, production last, and the production order confirmed with a human every single run. No exceptions buried in a rushed afternoon.

• Local execution is fenced. By default, direct hands-on execution is allowed only in the lowest-risk environment. Anywhere else requires an explicit override and a controlled setup.

• Notification and waiting periods. Production changes carry the required notifications and the required objection window, inherited from the repository’s shared rules rather than re-litigated each time.

• Verify is a gate, not a formality. The next step does not start until the current one is green.

None of this is novel as policy. Every mature team already has these rules written down somewhere. The shift is that the rules stop being aspirational. They become executable, and they travel with the procedure.

Repo-specific by design

Skills are not a central library that one team owns and everyone else queries. They are repo-specific.

Each repository carries its own skills, scoped to that repository’s services and infrastructure, maintained by the team that owns it. A skill loads only when work touches that repository. The runbook sits next to the code it operates on. A change to how a procedure works is a change to that repository, reviewed by the people who understand it.

This matters for two reasons. First, ownership is unambiguous: the team that runs the procedure owns the skill that encodes it. Second, it scales without a bottleneck. There is no central runbook team that becomes the constraint on everyone else. Every repository grows its own set, in the same shape, on its own schedule.

Lifecycle: how a skill is born and kept alive

The cheapest time to write a skill is right after you do the work the first time, while the gotchas are still fresh in your mind. Not before (you would be guessing), and not six months later (you would have forgotten the part that mattered).

• Capture. Do the work once, carefully. Then write the skill from what you actually hit, not from theory.

• Review. It ships as a normal change and gets reviewed like any other code.

• Run. The next time the operation comes up, invoke the skill instead of improvising. Dogfood it. Every point of friction becomes an edit.

• Extend. A new version or a new target is a data edit in references/, not a rewrite of the logic.

• Retire. When a procedure is genuinely dead, delete the skill in a reviewed change. No stale runbooks lingering to mislead someone in a year.

House rules

A few conventions keep a growing library coherent:

• Owned and reviewed. Every skill has an owner and lands through the normal review process. A change to a skill is a change to how the team operates. Treat it that way.

• Naming and shape. Verb-first, kebab-case names (upgrade-*, patch-*, migrate-*, rotate-*) and a consistent folder shape. Predictability is a feature.

• Compose, do not duplicate. Skills defer to the repository’s shared rules for things like notification, source control, and destructive-operation policy rather than restating them. Shared sub-procedures (for example, safely draining and replacing a node) become their own skill that others reference.

• Data out of logic. Anything that changes per version, per target, or per run lives in references/. The orchestrator stays stable and readable.

Rolling it out

You do not need a big initiative for this. You need one real skill and a convention.

1. Seed it. Write the first skill out of a real piece of work you just finished. Agree on the folder convention. One skill, real and in use, beats ten hypothetical ones.

2. Capture as you go. Every time you run a repeatable operation, write or extend its skill in the same change. Two or three high-frequency, high-risk procedures are the natural first targets.

3. Default posture. Over time, repeatable operations work becomes skill-first. The library quietly becomes the place your team’s operational knowledge is stored, versioned, and shared.

The compounding payoff

Each skill is written once and pays back on every run: faster, safer, more consistent, and teachable to the next person without a shadowing session. The library becomes the team’s operational memory, held in the repository, under review, and impossible to lose when someone changes teams.

And the framing that makes it click is the chain. You already tie alerts to runbooks. Extend that discipline one more hop, tie every runbook to a skill, and the procedure stops being a document you hope someone reads correctly. It becomes something you invoke, that carries its own guardrails, and that gets a little better every time you run it.

Detect, understand, act. The first two links you probably already have. The third one is the one worth building.