Design a Metrics Monitoring Service
Thousands of servers send small measurements, a few numbers each, many times a minute. The service has to take them in without dropping any, store them cheaply, answer dashboard queries, roll old data up into coarser summaries, and page someone through an outside provider when a rule trips. During an incident, dashboard queries spike at the very moment the data is least reliable.
Constraints
- Acknowledging a batch of measurements must take under 100 ms
- About 20,000 data points per second at peak, almost all of them writes
- No data point may be lost, and storage can be rebuilt by reading the stream again
- Dashboard queries are rare but expensive, and ten times busier during an incident
- Alerts go out through an external paging provider
- Keep infrastructure under $14,000 a month
- About 20 billion data points are kept, from 2 million separate series
Scenarios to survive
Typical peak
Steady ingestion from every agent.
Interviewer asks: Why does acknowledging a data point not need to wait for it to be stored?
A deploy storm (2x)
A rollout makes every server report twice as fast.
Interviewer asks: Where does the extra data wait, and what stops it from slowing the agents?
Incident dashboards (10x queries)
Everyone opens dashboards at once and queries multiply by ten.
Interviewer asks: Expensive queries and constant writes share a store. How do you keep one from starving the other?
A storage node fails
The metrics store loses a node.
Interviewer asks: The store loses a node. What happens to incoming data, and how do you catch up afterwards?
What a good design must do
- Data points are acknowledged within the latency target at peak
- The deploy storm passes without losing data
- Total monthly cost stays within $14,000
- Data goes through a Stream before storage
- Dashboards under incident load do not stop ingestion
- Something rolls up old data and can rebuild it from the stream
- Something evaluates alert rules
- The design explains why the store is built for appends and time ranges
- Every query is answered through a key or an index, never by reading a whole table