24/7 managed coverage for a commerce platform: median restoration down from 3.6 hours to under an hour
A retail commerce platform engaged TechSupport Team for night coverage: staffed shifts at peak hours and a managed on-call rotation for the rest. The work covered alert triage, vendor cases, and rollback and failover runbooks. Median restoration fell from 3.6 hours to under an hour over the first quarter.
Summary
For a retail commerce platform on AWS and Kubernetes, TechSupport Team staffed peak night shifts and ran on-call for the rest of each night. Four shift engineers and an on-call rotation of four covered every night, under an agreed severity model. Over the first quarter, median restoration fell from 3.6 hours to under an hour, and the client's duty manager was woken on 3 of 42 nights, per its incident records. The client kept status-page approval and all customer decisions.
Engagement facts
| Fact | Value |
|---|---|
| Industry | Retail commerce |
| Client location | North America, anonymized region |
| Application type | Storefront and order services with a payment gateway integration and a public status page |
| Stack | AWS, Kubernetes, managed queue and database services |
| Roles supplied | Four staffed-shift engineers plus a managed on-call rotation of four, led by a coverage lead |
| Seniority | Two senior, two mid-level on shifts; rotation mixed seniority |
| Engineer count | 8 across staffed shifts and the on-call rotation |
| Engagement duration | Ongoing, reviewed monthly against the severity model |
| Coverage hours | Staffed shifts at peak night hours; on-call for the rest of every night, weekend and public holiday |
| Time zones | Two North American time zones, aligned to the client's revenue hours |
| Reporting line | TechSupport Team coverage lead runs shifts and rotation; the client's duty manager owns Sev 1 decisions and status-page approval |
| Model | Managed 24/7 coverage: staffed shifts plus on-call |
Why the client needed engineers
The platform's revenue hours were the support team's off hours. Nights ran on a volunteer on-call rotation that woke people for false alarms and missed quiet degradations until customers found them. There was no severity model for nights, no triage notes and no written handover; mornings reconstructed the night from alerts and guesses.
The client chose managed coverage over extending its own rotation because night work needed a dedicated structure. That meant staffed peak shifts, a rehearsed escalation path and a coverage lead accountable for the hours its own team did not want to own.
Team composition and responsibilities
Four engineers staffed the peak night shifts. A rotation of four covered the remaining nights, weekends and public holidays on-call. A TechSupport Team coverage lead owned shift schedules, the drill program and the handover format. The client's duty manager received every Sev 1 escalation, approved the status page and decided what reached customers. Day engineers kept feature work; our coverage never held release authority.
Selection and onboarding
Assessment and severity model
The assessment agreed severity definitions, response targets and the escalation path to the duty manager, published to the night team as a one-page runbook.
Runbook and drill
Every engineer passed a night-shift drill before their first solo shift: alert intake, triage decision, rollback runbook and a status-page draft under time pressure.
Shadow shifts
New engineers shadowed two staffed shifts and one on-call wake-up handled by a senior before taking their own.
Independent coverage
The client accepted independent night coverage after three weeks, once drills, shadow shifts and handover quality were reviewed.
Technical work performed
- Triaged every night alert against the agreed severity model and recorded triage notes, so alert noise shrank week by week.
- Ran rollback, failover and feature-flag runbooks during incidents, with the duty manager approving customer-visible steps.
- Opened and tracked vendor cases with evidence packages during provider outages, including the payment gateway.
- Kept the status page moving through honest states: investigating, identified, monitoring, resolved.
- Wrote a handover for every shift and wake-up: what happened, what was checked, what changed and what to watch.
- Rehearsed the provider-outage runbook monthly after the first real event.
One representative incident
- Intake
Checkout failure rate crossed the Sev 1 threshold during an evening peak. Monitoring showed healthy application pods.
- Investigation
The on-shift engineer ruled out the last two releases from deployment logs and reproduced a failing authorization against the payment provider's test endpoint.
- Action
The incident was reclassified as a provider outage. A vendor case opened with request traces, and the failover runbook moved checkout to the secondary provider configuration.
- Approval
The duty manager approved the status-page move to identified with workaround steps and the customer communication.
- Restoration and permanent fix
The provider restored service and later confirmed the fault window. Checkout ran on the failover until then, and the review produced the rehearsed provider-outage runbook.
Checkout recovered through the documented failover steps, the provider confirmed the fault window, and the post-incident review added a provider-outage runbook that now gets rehearsed monthly.
Measured results
| Metric | Baseline | Result | Window | Sample | Source |
|---|---|---|---|---|---|
| Sev 1 responses inside the agreed target | Not previously tracked | 49 of 50 | First quarter | All Sev 1 alerts | Alert log against target record |
| Median restoration time, alert to recovery | 3.6 hours | Under one hour | First quarter | All incidents | Client incident records |
| Nights the duty manager was woken | Most incident nights | 3 of 42 nights | First quarter | All nights | Night log and rotation record |
| Root cause analyses for Sev 1 and Sev 2 | Inconsistent | Every incident | First quarter | All Sev 1 and Sev 2 | Root cause analysis (RCA) register |
Evidence and limitations
The metrics come from the client's incident records, the alert log and the rotation record for the quarter. Traffic grew during the window while the incident count stayed the same, so the faster restoration is not explained by a quieter period. Targets were agreed per severity and reviewed monthly; 24/7 coverage was never sold as a resolution-time guarantee. The client's head of operations reviewed this case study before publication; company details are anonymized with approval.
Questions buyers ask
What did staffed shifts cover that on-call could not?
Peak evening hours produce most alerts and most customer impact. Staffed engineers triage and fix in real time; on-call exists for quieter hours, so a ringing phone means a real decision, not routine work.
Who approved the status page and customer messages?
The client's duty manager approved every customer-visible step. Our engineers supplied the facts and drafts; approval never moved.
How did the handover work in practice?
Every staffed shift and every on-call wake-up ended in a written record: what happened, what was checked, what changed and what the day team should watch. The day lead could trace any night without asking anyone.
How were false-alarm nights reduced?
Triage notes on every alert, reviewed weekly with the coverage lead. Alerts that fired without action were tuned or reclassified, which is why the duty manager's phone now rings rarely.
What happened when a provider failed?
The runbook treated providers as dependencies with their own cases: evidence packages, honest status pages and rehearsed failover steps, as in the checkout incident above.
Contact us
Tell us which engineers you need, or ask a question. For a detailed request, use the engineer request form.
- We reply by email.
- We agree a call about the roles, your stack and the hours.
- If there is a fit, we propose engineers after an assessment.