A deployment topology for small distributed engineering teams
We designed and tested a deployment system for teams of three to eight engineers working across unreliable connections and time zones.
- Author
- Sidney Okine2SVII, co-founder
- Published
- Reading time
- 1 min
Deployment tooling is mostly designed for one of two shapes: a single developer pushing to a managed platform, or an organisation with a dedicated platform team. Teams of three to eight fall between them, and the usual advice — adopt the large-organisation toolchain early — costs more than it returns at that size.
Constraints
- No engineer can be a required participant in a deploy. Time zones and connectivity make that fragile.
- A deploy must be reversible by whoever is awake, without institutional knowledge.
- The system must degrade to a documented manual procedure when the automation is unavailable.
The third constraint is the one usually skipped. Automation that has no documented manual fallback converts an outage in the tooling into an outage in the product.
What we changed
The substantive change was making the rollback path the same path as the deploy path, rather than a separate emergency procedure. Rollback is deploying a previous immutable artifact by identifier. There is no distinct rollback tooling to be unfamiliar with at the moment it is needed.
A recovery procedure that is only exercised during an incident is a procedure you have not tested.
| Measure | Before | After |
|---|---|---|
| Median time to deploy | 18 min | 6 min |
| Median time to roll back | 41 min | 6 min |
| Deploys requiring a specific engineer | 31 of 62 | 0 of 62 |
| Rollbacks abandoned mid-procedure | 3 | 0 |
Related
Analysis
What edge runtimes actually change about request handling
A practical account of the constraints edge runtimes impose, and which familiar server-side assumptions stop holding when you move there.