Reliability
Week 12 · Promises you can measure
Week 12 · Promises you can measure
flowchart LR A["Oct 19, 11:48 PM<br/>DynamoDB endpoint<br/>resolves to nothing"] --> B["2:25 AM<br/>DNS restored"] B --> C["until 5:28 AM<br/>EC2 lease manager<br/>retries the whole fleet"] C --> D["until 2:09 PM<br/>load balancer<br/>health checks fail"] D --> E["2:20 PM<br/>recovered"]
The defect was fixed at 2:25 AM. Why did the outage last until the afternoon?
You design it, or you discover it.
The requirement Available at least 99% of each term, deadlines prioritized.
Its own architecture document "No alerting rules or SLOs." Production telemetry not collected.
If Project Pulse were down every Monday at 9 AM, what would notice?
| Project Pulse invitations, before Sept 10 | |
|---|---|
| Fault | One bad address throws out of the batch |
| Error | The transaction rolls back rows already emailed |
| Failure | A student is invited, then told they were not |
Everything fails, all the time. (Werner Vogels, Amazon CTO)
| AVL-uptime, measurable | |
|---|---|
| SLI | Share of one-minute health probes that pass |
| SLO | At least 99% over the term |
| Error budget | 112 days × 24 h × 1% ≈ 27 hours |
A Friday deploy that breaks until Monday spends two-thirds of the term.
Budget left: ship risky changes. Budget gone: the next release is a reliability fix.
Project Pulse answers unknown paths with its app page and 200. A probe that checks only the status says "up" for an endpoint that does not exist.
The agent writes the happy path. It makes none of these four unless you specify them.
sequenceDiagram participant B as Browser participant S as Server participant G as Gmail B->>S: invite 77 students S->>G: email 1 ... email 77 Note over B: 10 s: "failed" G-->>S: all delivered S-->>B: (nobody listening)
Retry only what is safe to repeat. Then space the retries out, with backoff and jitter.
for (Section section : sections) {
try {
for (Student student : section.getStudents()) {
try { send(student); }
catch (RuntimeException e) { log(student, e); }
}
} catch (RuntimeException e) { log(section, e); }
}Specified LLM down: everything else works; the AI features say so, within 15 seconds.
Not specified Whatever the agent happened to write.
"Be more careful" is not an action.