Operation In Practice

Operation keeps the system reliable, observable, and cost-effective. The guide covers what each operational practice is. This page covers how to apply them.

Relationship Map

Operation is a continuous loop. Production data flows back into Direction and Delivery:

Monitoring (3 tiers) ──► Incident Response ──► Post-Mortem ──► Feedback
      │                                                          │
      │                                                          ├── Direction: user behaviour changes Missions
      │                                                          ├── Delivery: incidents add Stories or tech debt
      │                                                          ├── Operation: cost trends trigger FinOps
      │                                                          └── Operation: SLO breaches shift to reliability
      │
      ├── On-call rotation (primary + secondary)
      ├── SLOs + error budget (features vs reliability)
      ├── Production Readiness checklist (new services)
      ├── Cost management (once per cycle, 15 min)
      └── Friction budget (estimated at retro)

Default Path + Branching

Default — in production:

Monitoring (3 tiers) + On-call rotation + SLO tracking
  → Incident response when needed
  → Post-mortem after significant incidents
  → Cost review once per cycle
  → Friction budget estimate at retro

Pre-PMF or no production users:

Skip monitoring infrastructure. Watch for pain manually.
Skip on-call rotation (everyone is on-call).
No SLOs. Measure latency when something feels slow.
Skip cost management (cost is negligible).

Incident-heavy period:

Tighten alerting → Reduce toil → Add runbooks →
Post-mortem every incident → Action items → Process improvement

Cost growing faster than users:

Add cost review → Identify largest driver → Optimize or cap →
Track cost per user → Act when one service exceeds 20% of budget

Role Map

ActivityPrimarySupportsWhen
Monitoring instrumentationSRE / Platform EngineerSoftware EngineerBefore first production traffic
Dashboard creationSRE / Platform EngineerProduct OwnerOne per audience — do not mix
Alert configurationSRE / Platform EngineerSecurity EngineerAlert on symptoms, not causes
Incident response (primary)On-call engineerSREWhen alert fires
Incident response (escalation)SRE / Incident Manager—When primary does not respond
Post-mortem facilitationIncident ManagerWhole team24-48 hours after incident
On-call rotation managementSRE / Engineering Lead—Continuous
Toil tracking and automationSoftware EngineerSREWhen toil exceeds 30%
SLO definitionSRE / Product Owner—Once per service; review quarterly
Error budget trackingSREProduct OwnerDecides features vs reliability
Production Readiness reviewSoftware EngineerSRE, Security EngineerBefore new service receives traffic
Cost reviewFinOps EngineerRotating team member15 min, once per cycle
Friction budget estimationWhole team—At retro; target under 20%

Technique Table

TechniqueUse WhenOutputSkill
Issue TriageIncoming production issuesClassified issue with owner and next stepissue-guide
Incident Post-Mortem (coming)After significant incidentsAction items (1-3) with owners and due datesThin skill planned
Incident Response (coming)Production incident requires actionService restored, timeline documentedThin skill planned
Feature Flag Lifecycle (coming)Managing flags from creation to removalVerified off → canary → rollout → removal storyThin skill planned

Procedure Table

ProcedureWhat It DoesWiki Link
Incident Triage (coming)Classify by severity, assign, respondWiki — planned
Incident Post-Mortem (coming)Timeline, action items, follow-throughWiki — planned
On-Call Handover (coming)15-minute sync at rotation changeWiki — planned
Implementation PipelineIncludes deploy, monitoring handoff, flag lifecycledelivery/procedures/implementation-pipeline.md

SLO Design

Service TypePrimary SLOSecondary SLOWindow
API (user-facing)p95 latency < 500msError rate < 1%30-day rolling
API (internal)p99 latency < 1sUptime 99.9%30-day rolling
Batch jobCompletion rate > 99%—Per execution
DatabaseQuery p99 < 100msUptime 99.99%30-day rolling
Web applicationPage load p95 < 2sError rate < 0.5%30-day rolling
Mobile appAPI success rate > 99%Crash-free rate > 99.5%7-day rolling

Do not track more than 3 SLOs per service. Use the error budget: full → ship features. Low → fix reliability.

Incident Severity

LevelMeaningFirst ResponseFix Target
SEV1User-facing outage15 min4 hours
SEV2Degraded subset of users30 min8 hours
SEV3No user impact, fix required1 business dayNext release
SEV4Cosmetic or internal1 weekWhen scheduled

Write a runbook for every incident type that repeats twice. If the same incident happens twice without a runbook, that is a process gap.

On-Call Design

ElementRecommendedAdjust When
Rotation length1 weekHigh incident frequency → shorter
Team size per rotationMinimum 2 (primary + secondary)Small team → adjust, but always 1 primary + 1 secondary
Secondary escalationName a person, not a team—
Handover15-minute sync at rotation change—
Toil threshold>30% → automate. >50% → rotate more people or change system.—

Production Readiness Checklist

CheckOwner
Logs are collected and searchableSoftware Engineer
Metrics are visible on the team dashboardSoftware Engineer
Alerts exist for likely failure modesSRE
Deploy process is documented and repeatablePlatform Engineer
Rollback process is documented and testedPlatform Engineer
Data backup strategy is definedSRE
Secrets and credentials are not in the code repositorySecurity Engineer
Dependencies are documentedSoftware Engineer
Contact for each dependency is knownSoftware Engineer
Runbook exists for known failure modesSRE

When to skip: Internal tools with fewer than 10 users. Short-lived experiments.

Cost Review

ItemQuestionAction If
Total infrastructure costTrend flat or growing?Growing faster than users → investigate
Cost per userGoing up or down?Going up → check for waste
Cost per serviceAny service exceeds 20% of budget?Yes → optimise or migrate
Largest cost driverWhat is the single biggest line item?Unknown → add cost tagging

Duration: 15 minutes. One person reviews. Not a meeting.

Artifact Table

ArtifactProduced ByConsumed ByFormat
RunbookSRE / Software EngineerOn-call rotationDocumented procedure for each failure mode
SLO DefinitionSRE + Product OwnerError budget, dashboardTarget metrics with measurement window
Error BudgetSREProduct Owner, Engineering LeadGap between target and actual
Operations DashboardSRE / Platform EngineerWhole teamOne per audience — do not mix
Incident Post-Mortem ReportIncident ManagerProcess ImprovementTimeline, root cause, action items
Production Readiness ChecklistSoftware EngineerSRE, Security EngineerVerified per new service
Cost ReviewFinOps EngineerTeamTrend analysis, biggest drivers, actions
Friction Budget EstimateWhole team at retroProcess ImprovementEstimated % — not precisely measured

Skip Conditions

ActivitySkip When
Full monitoring instrumentationPre-PMF. Watch for pain manually.
Formal incident responseTeam of 2-3, no external users. Fix it, talk after.
Formal post-mortemMinor incident with clear cause and no recurrence risk.
On-call rotationNo production users. Everyone is on-call anyway.
SLO trackingPre-revenue, team <5. Measure latency when something feels slow.
Full production readiness checklistInternal tools with <10 users. Short-lived experiments.
Cost managementCost is below a threshold the team sets.
Friction budget estimationTeam already discusses process costs naturally.
Runbook writingIncident happened once and is unlikely to recur. Write it after the second time.

Alert on symptoms, not causes. If an alert does not require a human to act, remove it. If a practice takes longer than the value it creates, remove it.

Comments

Comments on each page are welcomed and are visible here and under the discussions tab of the github repository.

View discussions on GitHub ↗