The situation
What needs investigation
An application has dashboards and alerts, but an incident still requires searching across tools to determine whether the problem is traffic, application behaviour, infrastructure saturation, a dependency, or a recent change. The review begins with the production questions the team must answer quickly.
Signals we look for
Service behaviour
- latency by endpoint
- error rate
- availability checks
- traffic and saturation
Application signals
- structured logs
- exceptions
- request correlation
- release version
Platform signals
- container / host health
- resource limits
- network errors
- database and cache behaviour
Response process
- alert ownership
- noise and thresholds
- runbook links
- incident timeline
How we investigate
Each step is validated against the workload before the next change.
- 01
Define questions
List the questions an on-call engineer needs answered during a production issue.
- 02
Inventory signals
Identify metrics, logs, traces, dashboards, alerts, and their owners.
- 03
Connect context
Link application, platform, dependency, traffic, and release context where they describe the same event.
- 04
Tune alerts
Remove non-actionable alerts and set thresholds tied to a response path.
- 05
Test investigation
Walk through a known failure mode using the available evidence.
- 06
Maintain
Update dashboards, runbooks, and thresholds after releases and incidents.
Architecture / decision path
Example engineering decision
Illustrative incident signal — not customer data
AlertAPI latency above threshold
Application CPUnormal
Database latencyincreased after traffic peak
Engineering interpretation
The alert identifies a user-facing symptom. The first investigation step is not to restart the API; it is to correlate the latency change with database wait time, connection pool behaviour, traffic, and recent deployments.
Possible next investigation
- Inspect endpoint latency and database query timing.
- Check connection-pool saturation and error logs.
- Compare traffic and release timing.
- Add a runbook link if the response path is repeatable.
Trade-offs we consider
Signal coverage
More telemetry is useful only when it supports an investigation or decision.
Alert noise
Low thresholds can hide important alerts in a stream of non-actionable events.
Retention cost
Logs and metrics need retention that matches operational and investigation needs.
Operational effort
Dashboards and runbooks require ownership and review after change.
What the engagement can produce
Typical outputs for this type of work
Scope depends on the environment and the problem being investigated.
- signal inventory
- dashboard and alert findings
- service and dependency monitoring plan
- alert ownership and escalation notes
- incident investigation runbook
- logging and retention recommendations