Your backups succeeded. Can you actually restore the service?

The backup job is green, but the service is still unavailable

A backup dashboard answers a narrow question: did the scheduled process produce the expected backup artifact? During an outage, the business asks a much wider question. Can customers use the service again, with an acceptable amount of lost work, and can the team explain what happened to transactions around the interruption?

This guide follows a fictional appointment-booking service. It stores bookings in a database, sends confirmations through a queue, keeps uploaded documents in object storage, and uses external identity and payment services. The recovery exercise is illustrative. It contains no measured customer results or promised recovery times. Its purpose is to help a team define what it will actually test.

Google Cloud's disaster-recovery guidance frames planning around business requirements, recovery objectives, and the systems needed to restore operation. This article builds an editorial exercise around that foundation. Use the primary references to check the supported recovery behavior of your own database, storage, identity, and deployment tools.

The central change in mindset is to test a restored service through a real user task. A database that accepts a connection is necessary for many applications, but it does not prove that a patient can book an appointment, that staff can see it, or that a confirmation will arrive exactly as the business expects.

Choose a business journey before choosing a failure

Start with a service journey that someone outside infrastructure recognizes. For the appointment service, choose creating a booking and viewing it from the staff dashboard. Name the user roles, data, and external actions involved. That gives the exercise a meaningful end point and prevents it from becoming a collection of unrelated component restores.

Define the smallest acceptable restored experience. Perhaps existing bookings must be visible before new appointments reopen. Perhaps document uploads can remain temporarily unavailable while staff can still access schedules. The business owner should make those priorities explicit, because the recovery team will otherwise infer them under pressure.

Record what successful completion looks like. A synthetic test account can create a booking in the isolated environment, a staff account can find it, and a test notification destination receives the expected message. If payment is part of the journey, use the provider's supported test mode or a controlled substitute. Avoid creating real charges or customer communications during a drill.

Choose a failure scenario only after the journey is understood. A lost database, a region outage, a deleted deployment configuration, and compromised application credentials require different recovery actions. One exercise can cover a bounded scenario well. It should not claim to prove every possible outage from one successful restore.

Give recovery objectives a clear clock

Recovery time and recovery point objectives need a defined starting event, ending event, and data boundary. A statement such as restore within an hour is ambiguous if one team starts the clock at failure and another starts after a backup has already been downloaded. Align the measurement before running the exercise.

For the appointment service, the business might measure recovery from declared service interruption to the agreed booking journey becoming usable. Detection, decision-making, infrastructure preparation, data restoration, validation, and reopening all consume that time. Record each stage so the team can identify where delay occurred.

The recovery point concerns the acceptable loss of committed business work. A backup timestamp is part of the evidence, but the important question is which bookings and associated changes survived. A database restored to one point and document storage restored to another can leave an application with mismatched state.

Use explicit, illustrative objectives during the first rehearsal if formal targets do not yet exist. Label them as proposed thresholds for discussion. Do not turn an untested aspiration into a published customer commitment. The drill should reveal whether the team can meet a target and what investment or scope change would be required if it cannot.

Map the dependencies needed to return to service

List the components needed for the selected journey. Include the database, application deployment, configuration, secrets, certificates, network routes, name resolution, identity provider, queue, object store, and external integrations. A dependency omitted from the map can become the longest part of the recovery.

Describe whether each dependency is restored, recreated, reused, or temporarily replaced. The production identity provider may remain available while the application database is lost. A region-wide scenario may require a different assumption. State those assumptions so a successful exercise is interpreted within its real scope.

Separate control-plane needs from data-plane needs. The team may require access to cloud management, source control, an artifact registry, or a secrets system before it can rebuild the service. Those dependencies can be unavailable even when the backup itself is intact. Include the operators' path to recovery tools in the map.

Ask the team to explain one failure chain from the map. If the artifact registry is unreachable, can they still deploy the known application version? If the restored environment cannot reach identity services, how will test users log in? A useful map drives practical questions rather than serving only as a diagram for a review meeting.

Build an isolated place to make mistakes

An exercise should create room to fail without affecting customers. Choose a test account, project, subscription, or equivalent environment with the network and permission boundaries your platform supports. The isolation must cover external effects as well as infrastructure names. A restored worker can still send a real email if it retains production credentials.

Inventory every outbound integration before starting restored workloads. Replace customer notification destinations with a sink, use payment test credentials, and prevent background jobs from writing to production systems. Make the expected behavior visible in the exercise plan so an operator can verify it before enabling workers.

Decide which production data may be used and who can access it. Restoring a backup into a less protected environment can expand access to sensitive records. Follow your organization's data-handling rules and use a sanitized dataset when it can answer the exercise's questions. If real data is necessary, preserve the required protections.

Give the environment an owner, spending boundary, and cleanup deadline. A recovery rehearsal should not leave a second production-like system running indefinitely with unclear access. Include a safe teardown plan, but retain the evidence needed for the review before removing resources or logs.

Confirm that the backup can be found and read

Start the exercise from the runbook, not from a console page someone prepared in advance. Ask an operator to locate the appropriate backup, determine its creation time, identify its protection and retention policy, and verify that the restore identity can access it. Record every manual discovery step.

Check encryption and key access as part of recovery. Possessing a backup object is not sufficient if the required key is unavailable to the restore process. The correct treatment depends on the service and key-management configuration. Verify the supported procedure in the relevant provider documentation rather than assuming that copying an object preserves usable access.

Include the possibility that the most recent backup is unsuitable. It may contain a damaging application change, reflect an incomplete data migration, or fall outside the desired recovery point. The operator needs a way to select an earlier valid point and explain the resulting loss of work.

For the appointment service, choose a known booking marker created before the selected backup and another created afterward. Those markers help validate the restored boundary. Keep them synthetic and documented. A marker exercise is more persuasive than assuming that a database timestamp alone proves which business records are present.

Recreate infrastructure from a known definition

A reliable restore requires somewhere appropriate to put the data. Identify the infrastructure definition, application artifact, configuration revision, and access policy needed for the chosen recovery point. Rebuilding today's infrastructure against an older data schema may fail even when both pieces are individually valid.

Use the same supported deployment path the team expects during an incident. If the runbook relies on infrastructure as code, test it from the retained revision and required inputs. Note which values are discovered dynamically and which must be supplied by an operator. Missing configuration often becomes visible only when rebuilding from an empty environment.

Keep environment-specific names and endpoints explicit. A restored application should connect to the recovered database and test integrations, not accidentally resolve a production hostname from an old configuration. Verify effective runtime settings before opening access or enabling background processing.

Record the time spent waiting for resource creation, capacity, permissions, and operator decisions. A restore benchmark that starts after infrastructure is ready leaves out a large part of service recovery. The business needs to understand the complete path from interruption to usable operation.

Restore data with the application version in mind

Select the supported restore procedure for the database and backup type. Understand whether it creates a new instance, restores into an existing target, or requires additional log replay. Read the exact service documentation for your engine and configuration. Similar product names can hide different recovery behavior.

Pair the restored database with an application version that understands its schema. A recently deployed service may expect a migration that occurred after the selected recovery point. Decide whether to use an older application artifact or apply a controlled migration to the restored data. Test that choice before relying on it during an outage.

Inspect the restored state before allowing writes. Verify the expected markers, important record counts, representative relationships, and service-specific integrity checks. Counts alone are not enough. Ten thousand bookings can still be wrong if they reference missing users or point at documents that were not recovered.

Keep the initial environment read-only where the selected tools and workflow support it, or otherwise limit writers until validation is complete. This is an editorial operating recommendation. The implementation depends on your application and database. The aim is to preserve a clear validation boundary before new activity makes discrepancies harder to investigate.

Bring object storage and the database into agreement

The appointment service stores document references in the database and document bytes in object storage. Restoring only one side can leave a booking that points at a missing attachment or an object that no longer has an associated record. Decide which mismatches the restored service can tolerate and how they will be handled.

Choose representative test cases. Include a document uploaded before the recovery point, one replaced near the boundary, and one deleted according to normal policy. Verify that the application shows the intended behavior for each case. A generic bucket listing does not establish that users can access the right version.

Document whether object versioning, snapshots, replication, or application-level export is part of the actual protection design. Each feature has its own scope and retention behavior. Avoid describing replication as a universal substitute for a recoverable history, especially when accidental changes may propagate.

If perfect alignment cannot be restored, define an honest degraded experience. Staff might see a clear missing-document state and a supported way to request a replacement. That decision belongs to the business and product owners. A recovery team should not silently hide inconsistencies behind successful infrastructure status.

Decide what to do with queued work

A restored application may find old messages, missing messages, or messages whose database effects no longer match the recovery point. Reopening consumers immediately can repeat notifications or apply work to records that were rolled back by restoration. Review the queue's state before enabling workers.

Identify the intended source of truth for pending actions. The database may contain an outbox, the broker may retain messages, and an external provider may already have completed some effects. Each system can describe a different part of the history. Reconciliation needs stable identifiers and a defined owner.

For the appointment exercise, create a synthetic booking whose confirmation was pending at the recovery boundary. After restoration, determine whether exactly one useful confirmation should be generated, suppressed as already sent, or held for review. Use a test destination so the drill cannot confuse real customers.

Document the restart order and validation checks for background jobs. A safe recovery can bring the application into read-only service first, reconcile queued effects, and then enable writes and consumers in stages. The exact sequence depends on the business, but it should be a deliberate choice rather than an accidental consequence of starting every container at once.

Recover identity and operator access

A restored service needs users and operators who can authenticate with the expected permissions. Verify the identity provider connection, application registration, redirect addresses, certificate dependencies, and role mapping for the recovery environment. A login page that loads does not prove that the correct staff role can perform its task.

Test one ordinary user, one authorized staff user, and one user who should remain denied. Recovery is a permission boundary as well as an availability exercise. An emergency configuration that grants broad access may make the demonstration easier while leaving an unacceptable state after reopening.

Include operator access to the cloud and deployment tools. If the main identity service is part of the failure scenario, use the organization's approved emergency-access process. The drill should verify ownership, authorization, audit evidence, and the procedure for returning to normal controls afterward.

Keep secrets and recovery credentials out of shared exercise notes. Record the location and owner of the approved retrieval process instead. A runbook should help authorized people find the right access without becoming a new store of reusable secrets.

Measure the whole recovery timeline

Create a timeline with explicit milestones. Record when the failure was declared, when the recovery decision was made, when infrastructure was ready, when data restoration finished, when validation passed, and when the agreed user journey became available. Include who made each decision.

Distinguish active work from waiting. A ten-minute restore can be surrounded by an hour spent locating a key owner, waiting for approval, or discovering a missing artifact. Those delays matter because they are part of the customer outage. They may also be cheaper to improve than the storage engine itself.

Capture uncertainty honestly. If an operator had a pre-opened console session or a resource was provisioned before the timer began, note that preparation. It may be a valid assumption for one scenario, but it should not disappear from the report. Future planners need to know what the measurement includes.

Use the first exercise to establish a baseline, not to prove a predetermined target. A result outside the desired recovery time is useful when it reveals a specific bottleneck and a realistic improvement. A polished success claim with hidden preparation makes the next incident harder to plan for.

Define the evidence for reopening

Before the drill, agree on the checks required to return the service to use. For the appointment system, that could include viewing existing bookings, creating a synthetic new booking, seeing it in the staff dashboard, and receiving a test confirmation. Add the authorization and data-boundary checks relevant to the scenario.

Name who can accept a degraded state. Perhaps document upload is temporarily unavailable while schedules remain usable. The business owner should approve that limitation and the communication plan. Infrastructure status alone should not decide that customers can resume a workflow whose important steps still fail.

Record the state of each required dependency and any deferred work. A recovered system may have a backlog, reduced capacity, or an external reconciliation task. Those conditions need an owner and a follow-up threshold. Reopening is a controlled decision with known limitations.

Ask a colleague who did not perform the restore to execute the validation steps. Independent use often exposes a missing permission, confusing hostname, or undocumented manual fix. The goal is not formal ceremony. It is evidence that the restored service works beyond the terminal where its author assembled it.

Keep recovery and failback as separate plans

Moving service to a recovered environment is one operation. Returning to the preferred environment later is another, especially after users create new data in the recovery location. A failback plan must explain how that new state moves and which system is authoritative during the transition.

Do not assume the old primary can simply be restarted. It may contain stale or conflicting writes, a damaged schema, or the original fault. Establish a controlled decision about whether to rebuild it, reconcile it, or retain the recovered environment as the new primary. Follow the service's supported replication and migration procedures.

In the exercise, write a small set of synthetic bookings after recovery. Then rehearse how those records survive the return path. The test should expose any step that would discard post-recovery business work or allow two writable systems to diverge without a resolution process.

Record the communication and rollback points for failback just as you did for recovery. A team may reasonably defer the return until it has enough evidence and staffing. Restoring customers quickly and returning infrastructure to its original shape are different priorities that deserve explicit decisions.

Test a missing dependency on purpose

Once the basic restore works, remove one supporting dependency from the exercise assumptions. Choose a bounded scenario such as an unavailable artifact repository, a missing configuration value, or a test identity integration that fails. This reveals whether the runbook describes a complete recovery path.

Keep the scope small enough to diagnose. Combining several unfamiliar failures in the first exercise can produce confusion without useful evidence. One missing dependency lets the team identify the symptom, determine ownership, and update the procedure based on a specific observation.

For the appointment service, suppose the application artifact is available but the deployment configuration points at an obsolete secret name. Ask the operator to find the mismatch using retained records. Measure the time and the access needed to correct it. The outcome may suggest better configuration versioning rather than more backup capacity.

Repeat the exercise after the improvement. The important result is whether the same failure is easier to recognize and recover from, not whether the team can memorize the solution during one meeting. A runbook earns value when another authorized person can use it later.

Exercise a destructive-change scenario separately

An infrastructure outage and a destructive application change have different recovery requirements. If a bad release corrupts data, the most recent replica may faithfully contain the same bad state. The recovery point may need to precede the change, and the application version may need to move with it.

Define how the team identifies the last acceptable state. Audit records, release times, synthetic markers, and business validation can all contribute. None should be treated as universally sufficient without checking the actual failure. A successful backup after corruption can still be a successful backup of unusable data.

Use an isolated dataset to rehearse this scenario. Apply a harmless, reversible change that makes a known record invalid, then test selection of a prior recovery point and compatible application version. Keep the exercise focused on decision-making and verification rather than causing real data damage.

Discuss the work that occurred after the selected point. Some actions may need to be reconstructed from external records or reconciled with customers. A recovery plan should acknowledge that restoring an earlier snapshot can require business recovery work beyond bringing the database online.

Estimate the cost of the recovery posture

Recovery design has ongoing costs and exercise costs. Warm capacity, retained backups, network transfer, test environments, and staff time all belong in the comparison. A cheaper steady-state design can require a longer rebuild, while a faster posture may need more resources and operating discipline.

Use a scenario-based comparison. For each option, describe which resources are already available, which must be created, and what the team expects to validate. Avoid comparing a fully prepared recovery environment with an unprepared restore as though their assumptions were identical.

Include the cost of rehearsal. A recovery posture that cannot be tested within the team's budget may drift away from its written design. Plan bounded exercises, cleanup, and retained evidence as part of the service's ordinary operating cost. That makes the recovery claim easier to maintain.

Ask the business owner what reduction in recovery time or data loss is worth funding. The answer may lead to better automation, a different backup schedule, or a more realistic service objective. The role of the exercise is to make that trade-off concrete with observed constraints.

Write the report so it changes the next drill

A useful report begins with scope and assumptions, then records the timeline, data boundary, validation result, and unresolved gaps. Keep the evidence links and exact revisions needed to repeat the exercise. Avoid a long narrative that makes it difficult to find what passed and what still needs work.

Turn findings into bounded tasks. Replace restore process needs improvement with an action such as retain the last approved application artifact in the recovery location or document the owner of the encryption-key access procedure. Give each task an owner and a way to verify completion.

Separate defects from deliberate limitations. If the exercise excluded a regional identity failure, that is a scope limitation. If the restored application accidentally contacted production email, that is a defect requiring correction before another drill. Clear categories help the team prioritize without overstating what the exercise proved.

Schedule the next drill around meaningful changes. A database upgrade, new identity integration, major schema change, or different backup policy can invalidate earlier evidence. Regular rehearsal matters, but change-driven review helps the team notice when its recovery story has materially changed.

Turn the runbook into a usable operating tool

Write steps in the order an operator encounters them. Start with declaring the scenario, locating the correct artifacts, confirming access, preparing the target, restoring data, validating behavior, and authorizing reopening. Include expected outputs and safe stopping points for each phase.

Link to provider procedures for exact commands and current service behavior. Keep application-specific decisions in the runbook, such as which version is compatible with the selected recovery point and which background jobs remain disabled. This division avoids copying a large vendor manual that becomes stale.

Make the runbook accessible during the failure it describes. If all instructions live behind a dependency assumed unavailable in the scenario, the team needs an approved alternate access path. Test that access during the drill rather than assuming an operator will have a cached copy.

Ask a new on-call engineer to walk through the document in the isolated environment. Where they need verbal guidance, improve the written step or its supporting context. Recovery readiness depends on what the team can execute, not on how confidently the original author can describe the design.

A worked recovery decision for the booking service

Suppose the exercise restores the database and application successfully, but the document store is still being reconciled. Staff can view schedules and create appointments, while attachments for a small set of synthetic bookings are unavailable. The team now faces a reopening decision, not a technical yes-or-no status.

Use the predefined acceptance criteria. If documents are essential to the selected booking journey, reopening waits. If the business agreed that schedules can return first, the service can enter a clearly communicated degraded mode with upload disabled and a named reconciliation owner. The correct decision follows the agreed user need.

Record the evidence and remaining boundary. Note which marker bookings were verified, which document cases failed, which notification destinations were tested, and which new writes are now permitted. That record makes the next phase safer because operators understand what has already changed.

Once reconciliation finishes, repeat the affected journey and remove the temporary restriction through the normal change process. Do not allow the degraded state to become permanent simply because the outage has ended. Recovery includes completing deferred work and restoring a maintainable service posture.

Questions for the business owner after the drill

Ask which parts of the observed outage would most affect customers. A technically short data restore can still create hours of manual rescheduling if the application loses recent bookings. A longer infrastructure rebuild may be acceptable if staff retain a reliable read-only schedule and a clear fallback process.

Discuss the cost of uncertainty. If the team cannot identify whether confirmations were sent, customer support may need to investigate individual records. Better event identifiers and reconciliation tools can reduce that uncertainty even without changing the backup interval. The drill can reveal investments outside the backup product.

Review the proposed objectives against the evidence. If the desired recovery target is unrealistic under current assumptions, decide whether to fund a different posture, narrow the supported journey, or revise the commitment. Leave the meeting with an explicit decision and an owner.

Finally, ask what must be retested before the next major product release. New integrations and data flows can quietly expand the service's recovery boundary. Keeping business ownership involved makes the recovery plan evolve with the product rather than remain attached to an older architecture.

Prepare a minimal exercise pack

Create a small pack of material before the rehearsal. Include the scenario, intended user journey, proposed recovery objectives, dependency map, authorized operators, test accounts, synthetic markers, and cleanup owner. Keep the pack short enough for everyone involved to read before the exercise starts.

Add a blank timeline and a place to record decisions. The observer should be able to note a delay without interrupting the operator to reconstruct it later. Record both the start and end of long waits. That makes it possible to distinguish work the team can automate from waiting imposed by another dependency.

Include the safe stop conditions. If a restored worker contacts a production destination, if unexpected access appears, or if the exercise exceeds its agreed boundary, the team needs a controlled way to pause. The conditions should be specific to the environment and approved by the responsible owners.

For the appointment service, the pack can identify one booking before the backup, one after it, and one with a document near the boundary. Those three records give the validation team concrete objects to inspect. A modest pack with useful markers often exposes more than a large document full of untested assumptions.

Treat configuration recovery as data recovery

Application behavior depends on configuration that may live outside the database. Feature flags, integration endpoints, tenant settings, and deployment parameters can determine how restored records are interpreted. Include the configuration state needed for the selected application version in the recovery inventory.

Distinguish settings that should return to their historical value from settings that must be replaced for the exercise. A production endpoint might be historically correct but unsafe in the isolated environment. The runbook should explain that deliberate substitution, along with the evidence that validates it.

Review who can reconstruct configuration if the usual management system is unavailable. Exporting sensitive values into a general document may create a new risk. Prefer an approved recovery method that preserves access control and gives operators a reliable version reference.

During the drill, compare effective runtime configuration with the intended recovery configuration. A successful deployment can still inherit an unexpected environment variable or default. Inspecting the running service's behavior and sanitized settings helps catch a class of failure that database validation alone cannot reveal.

Account for work performed outside the application

The booking platform may interact with people and systems that do not roll back with its database. A staff member may have called a customer, a payment may have settled, or a calendar integration may have created an external appointment. Restoring earlier application state does not reverse those actions.

List the external facts that must be reconciled after recovery. Use stable booking and payment identifiers to connect them. Decide which source is authoritative for each fact and who can approve a correction when records disagree. This is business recovery work that should have a place in the plan.

For the isolated exercise, use recorded synthetic external outcomes. Ask the team to resolve a booking that is absent after restore but present in the simulated payment ledger. The expected action might be recreation, refund review, or a hold for investigation, depending on the business contract.

Avoid automatic repair until its consequences are understood. A reconciliation job that treats every mismatch as a missing transaction may create duplicates or resend communications. Start with a visible discrepancy report and a bounded, reviewed repair path, then automate only the cases the team can define safely.

Turn recovery readiness into a maintained record

Keep a compact readiness record for the service. Include the last exercise date, scenario, tested application and data versions, observed recovery milestones, and unresolved findings. The record should link to evidence without forcing every reader to inspect all logs.

Use it during architecture and product changes. If a new document service becomes essential to booking, add it to the recovery journey and plan a test. If an old integration is retired, remove its obsolete steps. A maintained record makes recovery ownership part of normal service evolution.

Do not use the record as a permanent certificate of reliability. It describes what the team tested under stated conditions. A later dependency failure or unsupported restore path can reveal a new gap. That is a reason to improve the evidence and procedure, not to defend an old green status.

The practical habit is simple. After a consequential change, ask whether the last successful recovery exercise still represents the service people use today. If the answer is uncertain, identify the smallest next rehearsal that can resolve the uncertainty.

Close the exercise with an explicit handover

Before removing the test environment, confirm who owns each unresolved finding and where the retained evidence lives. Record which temporary credentials, access grants, and external test destinations were used, then retire them through the approved process. Cleanup should leave the ordinary service and its records in a clear state.

Give the application owner a short handover containing the tested journey, observed recovery boundary, reopening decision, and deferred work. Give the business owner the practical consequence of the result, including any data loss or degraded capability represented by the scenario. Both should be able to describe what the drill established.

Ask the next on-call engineer to locate that handover and explain the first recovery step. If they cannot find it, improve discoverability before calling the exercise complete. The durable output is a team that can repeat the process, not a single successful restore remembered by its author.

Retain a small set of repeatable markers

Keep the synthetic booking and document markers used in the drill as documented test cases. Their purpose is to make the recovery boundary visible, so record when each marker was created and what the restored application should show.

Use a fresh set for later exercises when needed, while preserving the earlier results for comparison. A marker that changes silently can make a restore appear incorrect or conceal a real mismatch. Treat the markers as part of the exercise definition.

This small discipline helps another operator repeat the validation without relying on the original team's memory. It also makes the report more precise about the business state that was actually checked.

Decide what the next exercise should prove

After the first recovery, choose the most consequential untested assumption. That might be a longer outage, an older recovery point, unavailable infrastructure capacity, or the loss of a supporting system. Select one question and define the evidence that would answer it.

For the appointment service, a useful second exercise might focus on queued confirmations around the recovery boundary. Another might test rebuilding the service without an existing artifact-cache entry. The choice should follow the actual findings and business risks rather than an arbitrary desire for a more dramatic scenario.

Keep the first exercise's markers, timeline format, and validation journey where they remain useful. Change the scenario deliberately and explain what comparison is valid. That allows the team to learn across drills without pretending that different failures are equivalent benchmarks.

The next action is to put a rehearsal on the calendar with a named owner, isolated environment, and business validation step. A small exercise that restores one complete journey provides more operating evidence than another quarter of green backup jobs.