GitHub's August Was Fire

I only found the story for this issue by going looking at my own run history. The Vigilant is a daily OSINT letter I publish at 1:30 p.m. Pacific, and its dead-man's switch is a GitHub Actions cron that runs at 22:15 UTC, asks Beehiiv whether an issue published in the last three hours, and fails loudly if the answer is no; loudly, so nobody has to wonder whether the system that does the publishing woke up (it hasn't always). Three of the last three runs had failed. The issue those runs were checking on had gone out at 20:30 UTC each day, on time, with real content.

The timestamps explained the rest. From Aug. 15 through 25 the dead-man fired within 20 to 30 minutes of its slot, which is the normal lag for a scheduled workflow. On the 26th GitHub ran the 22:15 slot at 03:17 UTC the next morning, five hours and two minutes late; on the 27th it ran at 06:06 UTC, seven hours and 51 minutes late; on the 28th, 03:42 UTC, five hours and 27 minutes late. A check that arrives eight hours after the send and asks "anything in the last three hours?" fails by construction. My newsletter had a perfect week; my monitor had borrowed GitHub's clock, and GitHub's clock had a bad one.

A small-potatoes bug in a one-person cron, and the sane fix is a wider window. Right? The window is the symptom. The instrument measuring whether my pipeline survived the day was itself running on the platform that was having the day. When GitHub degraded, the first thing to fail was the thing whose whole job was to tell me if something failed. A dead-man's handle on a locomotive only works because the spring that trips it doesn't draw power from the engine; wire it to the engine and it goes limp in the one situation it exists for. Call it a BORROWED CLOCK: any monitor, gate, or guard whose sense of time, truth, or trust comes from the system it's supposed to be watching.

That's a design principle worth carrying past cron.

Because August gave GitHub two kinds of bad news, and they rhyme. The first is availability: a seven-hour, 47-minute outage on the 17th, Actions down again on the 26th, and by The Register's count 23 incidents for the month with a week left. The second landed at Black Hat on Aug. 5, where Novee Security showed that an AI coding agent wired into an official repository's CI can be walked, by a stranger filing an issue, into handing over the runner's secrets. Same mistake, two axes. I trusted Actions with my clock; a lot of teams are trusting its runner with their keys while an agent reads the public internet.

The outage on Aug. 17 ran from 13:28 to 21:15 UTC, and Vlad Fedorov, GitHub's CTO, gave the plain version three days later: "The outage began when traffic reached a new peak, and a critical infrastructure component in our Central US data center failed to scale with it." He also wrote the sentence every status page owes its readers and few deliver: "If you were trying to ship software that day, we let you down." The Register noted he never used the word "sorry," which is fair; a specific admission still beats a generic apology any day of the week.

The RCA GitHub posted to its community forum the day before has the machinery. An Istio sidecar hit its concurrency limit and, because of a misconfigured policy, didn't scale; four HAProxy nodes exhausted their flow limits and degraded the gateway's authentication path. On top of that, a latent retry bug in VS Code amplified Copilot traffic by roughly 10x, pushing the Copilot Token Service from its normal 7,000 to 9,000 requests per second to somewhere between 70,000 and 100,000. At the peak, about 20% of web and API requests failed and about half of archive and raw downloads did. Lorin Hochstein's read of the same report is the one to hand anyone tempted to name a single culprit. The failure came from the interaction of changing traffic, the autoscaling policy, sidecar saturation, retry logic, HAProxy saturation, and auth traffic; six parts, each behaving as configured. Nobody flipped the wrong switch; the switches had never been tested together at that load.

The context Fedorov supplied reframes the month. "Today, Azure serves roughly 58% of GitHub's platform load and half of all Git operations, up from 12% of platform load in May." That's a migration of the majority of a platform's traffic in about three months, in production, while the traffic itself set new peaks. His commitments are the right ones: more than 3 million CPU cores and 120 petabytes of storage added, consistent retry limits and retry budgets across service-to-service calls, and "isolating critical systems and removing shared dependencies between them." That last one is my dead-man's bug stated at a scale of billions of dollars: shared dependencies between the thing and the thing that's supposed to catch the thing.

Six days after the CTO's post, Actions went down again, this time for about two hours and 49 minutes (15:11 to 18:00 UTC on the 26th). The cause was a database primary issue where failover to a replica "did not fully mitigate the degradation"; GitHub pointed at upstream Vitess. The Register's Vigliarolo counted 23 incidents for August with roughly a week still to go, against 37 in February and 32 in March, and wrote the line the industry was thinking: "Six days after he promised to fix things, here we are with reliability slipping and the issue count growing." Around those two headline events the status mirrors show, roughly, a six-and-a-half-hour Larger Runners incident on the 18th and an 11-hour stretch of broken agent task creation on the 20th and 21st. Then a 19-hour billing incident on the 27th and another couple of hours of Actions and pull-request trouble on the 28th.

The reader's objection here writes itself, and a month ago it was mine: scheduled workflows have never fired on schedule, GitHub's documentation has carried a delay caveat for years, and my own render workflow has a comment in it about "sidestepping GitHub Actions cron lag" because the 12:45 UTC slot reliably runs at 13:24 or 13:38. Correct on every count. The repository's 96 runs from Aug. 15 through 28 include the scheduled runs of the 17th and 26th, every one of which succeeded (the five failures are the dead-man false alarms and their siblings). A one-person cron slept through a seven-hour, 47-minute outage at a company with a CTO-level apology. So for most of us the availability story was never the outage; best-effort is the contract, best-effort mid-migration means eight-hour lag on a good week, and anything you built assuming "close to on time" was measuring the platform with the platform's own ruler.

Actions is a shared compute pool, extraordinarily useful and priced like a commodity. It isn't a control plane, and the month it had is what happens when you treat one as the other.

Now the second axis, because the runner you're borrowing a clock from is also the runner you've handed an agent and a wallet. Elad Meged of Novee Security presented "Trusted Enough to Run: Breaking AI Agents in Official Workflows" at Black Hat USA on Aug. 5, and the Cloud Security Alliance published the written version three days later. Meged's framing, quoted by The Hacker News: "The harness is the code between the model and the real world." The model didn't get jailbroken in any of the three chains; the harness around it did the leaking.

The Claude Code chain is the one to study. An unprivileged user files an issue in a repository whose CI runs Claude Code on incoming issues. The payload sits inside single quotes in the receive-pack flag of a git push. The command validator strips single-quoted text before it evaluates the command, so the validator sees something harmless and the shell sees something else. That's remote code execution on the runner, with the runner's environment readable: the GITHUB_TOKEN, the Anthropic API key.

When direct exfiltration was blocked, the payload told the agent to create 64 model repositories on Hugging Face (huggingface.co was on the pre-approved list for WebFetch) and leak the key one character at a time through the public download counters. Anthropic went three rounds of patch-and-bypass before closing it in 2.1.163. The CVE is 2026-54316, published June 17, scored 9.1 by NVD under CVSS 3.1 and 6.0 by Anthropic under their own v4 assessment. Both numbers are real and they disagree, which tells you something about how new this attack class is to the people scoring it.

Gemini CLI fell to a crafted .gemini/.env file that produced host-level code execution before the sandbox started, on headless CI runs; the agent read GITHUB_TOKEN, GEMINI_API_KEY, and the rest of the CI environment straight out of /proc/$PPID/environ. That's CVE-2026-12537, published June 24, CVSS v4 10.0, fixed in 0.39.1 and run-gemini-cli 0.1.22. Google called the headless folder-trust change a "breaking change" (Dan Lisichkin of Pillar Security found the same hole independently).

Codex got no CVE: issue content was written into AGENTS.md, and invalid JSON forced a second pass that loaded the poisoned file. The CSA note characterizes OpenAI's position as intended sandbox behavior; OpenAI hardened the openai/codex workflows within three days anyway, separating workflows and making the sandbox read-only. Both CVEs went public in June. The Aug. 5 talk was the demonstration, not the disclosure, and the repositories running these workflows in July had a patch available and a public issue tracker open to everyone.

The belief that broke: "an agent in CI only sees what I hand it." It sees what the runner sees; the runner sees everything in its environment plus everything the agent's tools are allowed to reach, and one of those tools was allowed to reach a site with a public write-and-count primitive. GitHub's June 11 changelog helps at the margin (agentic workflows "no longer need a personal access token"; permissions: copilot-requests: write on the GITHUB_TOKEN does it), because a scoped token that dies with the job leaks less than a PAT that doesn't. But it's the same borrowed-trust shape as my clock: the agent's sense of what it's allowed to do came from the environment it was running in, and that environment was also where the attacker's text arrived.

Give every guard its own clock and its own trust root. For the dead-man, that means two changes. The assertion becomes "an issue with real content published today, after 20:30 UTC," which is true at 22:15 and still true at 06:06; the workflow's first step measures its own lag against the cron slot and reports "GitHub ran this 7h51m late" as its own distinct failure, so an eight-hour lag reads as GitHub news and not as newsletter news. The heartbeat that decides whether the publishing Mac is alive shouldn't depend on GitHub being alive to ask the question. A $5 VPS cron, a Cloudflare Worker cron, or any scheduler on a different failure domain is the right home for a switch whose entire value is independence; it costs almost nothing, and it should've been done in July.

For agents in CI, the same rule reads: the agent's environment shouldn't contain anything it doesn't need to finish the one task in front of it. Run the agent step with an explicit env: block and nothing inherited; pass a job-scoped GITHUB_TOKEN with the narrowest permissions: that works; keep model API keys off the runner by proxying them through a service that rate-limits and logs. Treat any tool with network egress as a channel out (a download counter is a channel; so is a DNS lookup). Pin actions to a commit SHA, and never let pull_request_target or an issue-triggered workflow run an agent with write access. Every one of those is boring, which is the point; the interesting part of the Black Hat talk was how ordinary the setups were.

Fedorov's third commitment, isolating critical systems and removing shared dependencies between them, is the same sentence at both scales. GitHub is spending 3 million cores on it. You can spend an afternoon.

🟢 Easy (~10 min): Grep every workflow in one repository for the three moves the Black Hat chains needed: grep -rnE 'pull_request_target|\$\{\{ *github\.event\.(issue|comment|pull_request)\.(title|body)|uses: [^@]+@(v[0-9]+|main|master)' .github/workflows/. Every hit is either a public-text expansion, a privileged trigger, or an unpinned action; fix or justify each one in a comment.

🟡 Medium (~1 hour): Move one monitor off the platform it monitors. Put the dead-man for your most important scheduled job on a different failure domain (a small VPS cron, a Cloudflare Worker cron, a Lambda on a schedule) and make it assert "happened today after HH:MM," never "within the last N hours."

🔴 Advanced (half day): Rebuild one agent-in-CI workflow from an empty env: block up. Grant a job-scoped GITHUB_TOKEN with the minimum permissions:, route the model key through a proxy the runner can't read, allowlist tool egress to the hosts the task requires and nothing with a public write primitive, pin every action to a SHA, and then file an issue against the repo yourself with a payload in single quotes to see what the validator strips.

The MCP as a webhook, unless something more exciting or catastrophic happens between now and then. More next Tuesday, assuming my cron job fires.

GitHub had the worst month it's had in a long time, and none of it reached my subscribers. What it did reach was a monitor that had borrowed the platform's clock and then reported the platform's lag as my failure. The Black Hat chains are the same defect with more severe consequences; an agent whose trust and secrets came from the environment the attacker could write to. Give every guard its own clock and its own keys, and keep the shared dependencies out of the path between the thing and the thing that watches it; the next time GitHub lets you down (Fedorov's words, and they'll apply again) the only email you'll get is the one that says so plainly.

Bobby R. Goldsmith
Ambassador Extraordinary and Plenipotentiary of Bashmatica! by NodeBridge Automation Solutions

P.S. If your dead-man switch lives on the same cron as the job it guards, you now have company; I ran that way for two months. Last week's Issue #29 covered the interlock, and this one covers what the interlock plugs into. Forward it to whoever owns your .github/workflows/ directory, and if that's you and someone forwarded it, subscribe at bashmatica.com.