Make failure boring before you make scale exciting

It's easy to get drawn into conversations about scale. The diagrams are interesting, the tools are impressive, and nobody wants their product to fall over when it finally gets some attention.

For an application approaching its first serious users, though, a more immediate worry is usually worth tackling first. Can someone save their work? Will a booking appear twice if they press the button again? If yesterday's release breaks something, can you get the service working without guessing?

These questions apply whether the code came from an AI assistant, a contractor or your own team. You can make meaningful progress on them without rebuilding the application.

Start with the promise you're making

Choose one journey people depend on. Imagine a small booking service: a customer chooses a slot, submits their details and receives confirmation. Write down what success means, including the awkward bits. A slot belongs to one customer. An accepted booking survives a restart. A delayed email doesn't make the booking disappear.

Then ask the business owner two plain questions: how long could this journey be unavailable, and how much recent work could we afford to lose? The answers give recovery work a purpose. If an hour of missing bookings would require a day of phone calls, a nightly backup deserves a closer look.

Keep this description short enough for everyone involved to understand. It becomes the basis for tests, alerts and the instructions somebody follows when a customer reports a problem.

Put an end to waiting

Remote calls need time limits. Otherwise, an unavailable dependency can leave requests hanging and occupy resources other customers need. A timeout means the caller stopped waiting; the other service may already have completed the operation. Amazon's guide to timeouts and retries explains this uncertainty, and why retries need limits, increasing delays and some random variation, usually called jitter.

For your booking service, decide how long a customer should wait for an answer. Check that connection setup and the response are covered, and that multiple attempts fit inside that overall budget. A screen that spins for a minute because several individually reasonable waits stack up is still a poor experience.

Retry only failures the service documents as recoverable, such as certain temporary availability or rate-limit responses. Follow its retry guidance, cap attempts, and check whether your client library already retries. Retrying a validation error won't fix the submitted details. Retrying an overloaded service immediately may add to its problems.

Make a repeat request safe

For operations that change things, establish how repeats are recognised. An idempotency key identifies one intended operation across multiple attempts; the service can recognise a repeat and avoid applying the effect again. Amazon's idempotent API guide covers this approach, including the need to record the key and the change consistently. Keys also need a defined lifetime and handling for conflicting request details.

In our example, the same booking attempt should keep the same key when the connection drops. A genuinely new booking needs a new one. Test the awkward case deliberately: the booking is saved, but the response never reaches the browser. What happens when the customer tries again?

If an external service offers no suitable protection, don't assume adding a key locally makes its operation safe. You may need to check the external result before proceeding or leave the item for someone to reconcile. Show the customer that confirmation is pending, with a reference they can use, instead of inviting repeated submissions into an uncertain state.

Practise getting the data back

A backup marked successful is encouraging. The next useful step is restoring it somewhere isolated and checking the application against it. Microsoft explicitly recommends periodic test restores to check that backups meet recovery needs.

For the booking service, try opening a recent booking, finding its customer and viewing any associated upload. Database rows alone might not include everything the application needs. Record how old the recovered data is and how long the whole process took, including finding credentials and getting the application running.

Use a restricted test environment with outgoing emails, payments and live integrations disabled. A restore drill shouldn't send yesterday's confirmations again. Handle the copied data with the same care as the original.

Keep a short recovery note: where the backup lives, who can access it, the steps taken and the checks that show it worked. Have another person follow it where possible. If only the original author can complete the exercise, add the missing details while they're easy to remember.

Rehearse the way back

Keep the previous working release available and practise returning to it in a test environment. Remember that changing application code back doesn't undo changes to stored data. Amazon's rollback safety guidance explains why old and new versions need compatible ways to read and write data.

Suppose you rename the booking's status field and delete the old one in the same release. The previous application may now fail against the updated database. A staged change can keep both versions working while you move the data and switch readers. Remove the old field only after the versions that need it have been retired and the agreed rollback window has passed.

Decide before release whether recovery means returning to the previous version, disabling a feature or applying a forward fix. Include a check of an actual booking journey afterwards. A green deployment indicator doesn't tell you whether customers can finish their task.

Give every alert a job

Google's monitoring chapter offers a useful starting point: latency, traffic, errors and saturation. It also distinguishes conditions that warrant interrupting a person from information that belongs in a ticket or log.

Translate that into the service's everyday work. Track whether bookings complete and whether confirmations are waiting unusually long. A healthy homepage tells you little about a stopped email worker. On a quiet service, a scheduled safe check may reveal a fault sooner than waiting for customer traffic.

For each urgent alert, name the recipient, explain the likely customer impact and link to the first checks. Send a test alert through the real route. Agree who covers absences and what can reasonably wait until working hours.

Keep a request or booking reference in diagnostic records so an engineer can trace a problem. Avoid filling logs with customer details or credentials. After an incident, adjust noisy alerts and write down anything that would have made the response easier.

A useful first week

For a small service, I'd start with this sequence. Treat it as a set of priorities; a difficult restore may quite reasonably take the time you'd planned for something else.

1. Choose the critical journey. Agree what must survive a failure and what recovery time would be acceptable.
2. Trace one failed request. Check time limits, retry behaviour and duplicate protection with a controlled test.
3. Restore a backup. Run the journey against the restored application and record the gaps.
4. Rehearse a failed release. Check the previous version against the current data structure.
5. Test the alert and instructions. Make sure the right person can see the problem and take a useful first step.

You won't have covered every failure. But you'll know whether the backup restores, whether a repeated request is safe and who gets the alert. Those are useful things to know before the next busy day.

Further reading

• Amazon Builders' Library: timeouts, retries and jitter (PDF).
• Amazon Builders' Library: making retries safe with idempotent APIs.
• Microsoft: protecting and validating backups.
• Amazon Builders' Library: ensuring rollback safety (PDF).
• Google SRE: monitoring distributed systems.