State
Firing
unassigned · never acknowledged
Open for
2h 14m
first seen 09:42 UTC · Jul 23
Blast radius
12stores
4.1% of fleet sessions · 1.72M/day
Collapsed from
38alerts
rule: Script fire-rate floor
Script fire rate · affected stores
00:0006:00
09:42 breach
now 11:56
Fell from 99.1% to 78.4% in 4 minutes — a step change, not a drift. Consistent with a deploy, not with traffic.
Affected stores
| Store | SDK | Theme published | Fire rate | Sessions lost / day |
|---|---|---|---|---|
| northwind-supply | 4.2.1 | 09:38 · Dawn 15.2.0 | 0% | 412,900 |
| atlas-outfitters | 4.2.1 | 09:41 · Dawn 15.2.0 | 0% | 338,140 |
| harbor-lane | 4.2.1 | 09:44 · Craft 6.1.1 | 31% | 201,470 |
| foundry-mercantile | 4.2.1 | 09:47 · Dawn 15.2.0 | 0% | 186,020 |
| pilot-goods | 4.2.1 | 10:02 · Dawn 15.2.0 | 44% | 154,880 |
| kettle-and-co | 4.2.1 | 10:16 · Refresh 9.0.4 | 0% | 129,310 |
6 more stores below 100k sessions/day
Timeline
Jul 22 · 14:00
SDK 4.2.1 rollout reached 62% of fleet
Automatic staged rollout. No alert at the time — coverage looked normal for 19 hours.
Jul 23 · 09:38
northwind-supply published theme Dawn 15.2.0
First of 12 stores. Publish events observed on 11 more within 38 minutes.
Jul 23 · 09:42
Rule Script fire-rate floor breached
Fire rate 78.4% against a 95% floor, sustained 10 minutes. INC-2291 opened.
Jul 23 · 09:44
37 further alerts collapsed into this incident
Same rule, same SDK version, same 38-minute window — grouped instead of paged separately.
Jul 23 · 10:07
Paged on-call · no acknowledgement
Escalation policy scripts-critical · re-page due at 12:07.
Jul 23 · 11:56
Still firing · 12 stores
No change in blast radius for 74 minutes.
Correlation
All 12 stores run SDK 4.2.1 and published a theme after 09:30. No store on 4.2.0 is affected, and no 4.2.1 store that skipped a publish is affected.
- Shared SDK
- 4.2.1 (12 / 12)
- Shared action
- Theme publish (12 / 12)
- Shared theme
- Dawn 15.2.0 (9 / 12)
- Shared region
- none — 4 regions
- Shared plan
- none — 3 plans
Confidence: two signals at 12/12
Runbook
- Confirm the script tag survived the publish on one affected store.
- If absent, re-run Install Coframe Scripts for the store from the admin API.
- If present but inert, roll the store back to SDK 4.2.0 and re-check fire rate.
- Hold the 4.2.1 rollout until fire rate recovers above 95% fleet-wide.
coframe scripts:verify \
--store northwind-supply \
--expect-version 4.2.1
Seen before
Scripts dropped after theme publish
INC-2104 · Jun 11 · resolved in 41m
8 stores
Scripts silent on SDK upgrade
INC-1977 · Apr 28 · resolved in 2h 06m
31 stores
Both fixed by re-running script install.