IMS
Incident management that collapses alert storms into one incident
Error signals from monitored services hit a FastAPI endpoint that answers 202 straight away, buffer in Kafka, and are debounced by a worker into deduplicated incidents. Each incident follows a strict lifecycle with mandatory RCA, and a React dashboard updates live over WebSockets.
- 202 on the hot path
- 100 signals / 10 s → 1 incident
- Survives a Kafka outage
- 30-min SLA → P0
§1
The problem
When a dependency fails, every service that uses it fires the same error thousands of times. Writing each signal straight to the database overloads it, and opening an incident per signal buries the on-call engineer. The design target was 10,000 signals per second without losing any.
§2
My role
Sole developer: ingestion API, Kafka worker, debounce logic, incident state machine, React dashboard, failure-simulation scripts and the Docker + Caddy deployment.
§3
Architecture
§4
Key decisions
D/01
Accept fast, process later
The API validates, rate-limits in Redis, produces to Kafka and returns 202. Database speed never shows up in ingestion latency; bursts queue in the topic.
D/02
One store per job
PostgreSQL holds incident state, RCA and the audit trail (ACID). MongoDB takes raw signals of any shape, deduplicated by a unique event_id index. Redis does counters, rate limits and pub/sub.
D/03
Debounce in Redis
A per-component counter expires after 10 s. At 100 signals, SET NX lets exactly one worker create the incident, even with several workers consuming.
D/04
Lifecycle as a state machine
OPEN → INVESTIGATING → RESOLVED → CLOSED, one class per state. Closing without a complete RCA returns 400, and a partial unique index (state != 'CLOSED') stops a second open incident for the same component.
D/05
Severity by strategy
An AlertStrategy per component type (database, queue, cache, API, MCP host…) decides severity and dispatch, so a new type is a new class, not another if-branch.
§5
Challenges
- Broker outage: if the produce fails, the API pushes the signal onto a Redis list and still returns 202. A background task drains it back into Kafka when the broker returns. A script stops Redpanda to prove it.
- Poison messages (bad JSON, missing fields) go to an ims.signals.dlq topic instead of stalling the consumer, with a replay script to feed them back.
- Incidents still open after 30 minutes are escalated to P0 by a worker loop and logged as SLA_BREACH events.
- Live updates: the worker publishes snapshots on Redis pub/sub and the API fans them out to every dashboard over WebSockets.
§6
Stack
- Python
- FastAPI
- Kafka (Redpanda)
- Redis
- PostgreSQL
- MongoDB
- React
- WebSockets
- Caddy
- Docker
Next case study
Payeazie
Payment operations: async charging, recovery tools and a full audit trail