Resilience, Availability, and Physical Operations

The promise is a machine

Every recovery plan makes a promise. It promises that a process resumes in a stated number of hours and loses no more than a stated amount of data. Chapter 5 taught you how those promises are written: the business impact analysis, the recovery time objective, the recovery point objective, the site options, the tests. This chapter is about the machinery that keeps the promises: the redundant components that make failures invisible, the backup schedules that preserve a recoverable past, the recovery procedures that turn stored bytes back into running service, and the physical facility that holds all of it. A plan without machinery is a wish, and machinery without a plan is expensive decoration. The exam’s operations material tests the machinery, and it tests it at the level of the decision-maker: which RAID level survives which failure, which backup type restores fastest, why a snapshot is not a backup, what an immutable backup is for, what a Tier III data center can do that a Tier II cannot, and which control governs a vendor’s remote access to a production server.

The frame that holds all of it is a single sentence. Availability is an engineering property with a price, and the price is paid twice: once in design, when redundant capacity is bought, and once in operations, when that capacity is maintained, tested, and exercised. Most outages are not caused by the absence of a plan. They are caused by a component failing that someone forgot was single, a backup that nobody ever restored, a generator that had not been tested, or a badge that was never revoked. The discipline of this chapter is the discipline of noticing those gaps.

The arithmetic of availability

Availability is a fraction: the proportion of time a system is operational and usable. The exam expects you to work fluently with the numbers, because every design decision in this chapter is really a decision about where the fraction falls.

The standard measure is a percentage, and the percentages have names. 99% allows about 3.65 days of downtime per year. 99.9%, often called “three nines”, allows about 8.8 hours. 99.99%, “four nines”, allows about 53 minutes. 99.999%, “five nines”, allows about 5 minutes and 15 seconds. The sequence is the trap: each additional nine removes a decimal place from the allowed downtime, not a percentage point. A system at five nines cannot meet its target if it has a single six-minute maintenance window per month unless the maintenance can happen without taking the service down, which is exactly why the data center tiers later in this chapter exist and why “concurrently maintainable” is a selling point rather than a slogan.

The second piece of arithmetic is the component-level formula. For a repairable system, availability equals the mean time between failures divided by the sum of the mean time between failures and the mean time to repair: A = MTBF / (MTBF + MTTR). A component that fails once every 90 days and takes an hour to repair, MTBF of 2,160 hours and MTTR of 1 hour, delivers availability of 2,160 / 2,161, about 99.95%. The formula shows why redundancy works: when a second component covers the first, the pair’s effective MTBF grows enormously while the exposure to repair time shrinks, so the fraction approaches one without any single component being more reliable. Reliability engineering is not about finding perfect parts; it is about arranging good-enough parts so that no single failure matters.

Two vocabulary notes the exam likes. Mean time to failure, MTTF, applies to non-repairable items: the disk that is replaced rather than fixed, the battery that is swapped. Mean time between failures, MTBF, applies to repairable items. Mean time to repair, MTTR, is the whole repair interval, including detection, diagnosis, part arrival, and validation, not merely the minutes the technician’s hands are on the machine. And one measurement note: the denominator of a real availability measurement includes every minute the service was down, whatever the cause, including the “scheduled” ones. Customers do not distinguish planned maintenance from failure; the SLA does not care whose calendar the outage honored. This is why maintenance windows appear on the outage side of the ledger, and why design that allows maintenance without outage, the tiers again, is an availability control rather than a convenience.

Redundancy: buying back the failure

Redundancy is the answer to a question, and the question is: which single component, if it fails, takes the service down with it? That component is a single point of failure, SPOF for short, and the entire craft of resilience design is finding SPOFs and removing them layer by layer. The exam rewards the search: a scenario about a service that survived every outage until the network switch in rack 12 died is a scenario about a SPOF that the design forgot.

The vocabulary of spare capacity is precise. N is the capacity the load needs. N+1 means one additional unit, so the design survives the failure of one component: one extra power supply, one extra cooling unit, one extra switch. N+2 survives two simultaneous failures. 2N means two complete, independent sets of everything, so a whole path, an entire electrical feed or an entire cooling loop, can fail and the other set carries the load. 2N+1 means two full sets plus a spare on top. The escalation is about what the design tolerates: N+1 tolerates component failure, 2N tolerates path failure. The exam’s discriminator is usually that: a question offering “N+1 power” versus “dual independent power feeds” is asking whether you know the difference between surviving one dead unit and surviving a dead distribution path.

Architecture adds the second dimension. Active-active configurations run the load across two or more live components simultaneously, sharing traffic and failing over by rebalancing. Active-passive configurations run the load on one component with a standby ready to take over. The standby comes in the familiar temperatures from Chapter 5, at the component level instead of the site level: hot standby is running and synchronized, ready in minutes; warm standby is powered and configured but not current; cold standby is hardware with no current state, and its recovery time is whatever it takes to bring it up. Failover is the move from primary to standby. Failback is the deliberate, often more careful, move back when the primary is proven healthy. The exam expects failback to exist in the vocabulary: teams that practice failover and never practice failback have practiced only half the operation, and failback is frequently the moment systems break because the old primary comes back into a world that has changed.

Redundancy must be designed across every layer, and the exam likes to see you enumerate them: power, so a dead circuit or failed PSU does not end the session; cooling, so a failed unit does not overheat the rack; network, so a dead link, switch, or carrier does not isolate the service; compute, so a dead server or hypervisor host does not take its workloads down; storage, so a dead disk, controller, or array does not lose or strand the data; data itself, so a corrupted or deleted dataset can be recovered from a point before the corruption; and, quietly, people and process, so the one engineer who knows how to fail over is not the one on vacation. Every layer a design fails to cover is a SPOF by omission.

The stack: storage, compute, network, and the room

Storage: RAID and beyond

Redundant Arrays of Independent Disks, RAID, is the classic storage resilience control, and the exam asks about it at the level of “which level survives which failure”. RAID 0 is striping only: data is spread across disks for performance with no redundancy, and it survives nothing; it is not resilience at all. RAID 1 is mirroring: two disks hold identical copies, and the array survives the loss of one. RAID 5 stripes data with distributed parity across at least three disks and survives the loss of any one disk, with the parity blocks allowing the missing data to be recomputed. RAID 6 carries two parity sets across at least four disks and survives the loss of any two. RAID 10, mirroring then striping, combines two or more mirrored pairs and survives the loss of one disk from each pair; it is the common choice for performance-sensitive workloads because it gives both speed and fault tolerance without the parity-rebuild computation of RAID 5 and 6.

Three operational facts matter beyond the level table. First, a hot spare is a standby disk the array uses automatically to rebuild after a failure, shortening the window in which the array is degraded. Second, the rebuild itself is a risk: reading every remaining disk to reconstruct the lost data stresses drives that are often similar in age, which is one reason RAID 6 exists and why rebuild monitoring is a real operational duty. Third, RAID is not a backup, and the exam says this in a hundred ways. RAID survives disk death; it does not survive a file deleted by mistake, an application that corrupts its own database, a ransomware process that encrypts every volume, or a fire. RAID protects against hardware failure of a disk, nothing else. The storage array’s snapshots and replicas are the next rung, and the backup system is the one below them, and none of them substitutes for the others.

Storage replication comes in two flavors, and the difference is the recovery point. Synchronous replication writes to both the primary and the replica before acknowledging the write, so the replica is always current and the recovery point is essentially zero; the cost is latency, because every write waits for the far site, and distance erodes performance. Asynchronous replication batches or streams changes to the replica on a delay, so the replica may lag by seconds or minutes and the recovery point is that lag. The exam discriminator: synchronous buys near-zero RPO at latency cost; asynchronous buys most of the benefit without the latency. Both share a profound limitation, covered in the backups section, that replication copies every write faithfully, including the bad ones.

Snapshots are point-in-time views of data, usually implemented copy-on-write, so they are cheap and instant. They are the right tool for quick rollback before a risky change and for test environments. They are not backups, and the exam tests the distinction hard. A snapshot lives on the same storage as the data it pictures, so the array failure that kills the data kills the snapshot. A snapshot taken after corruption began contains the corruption; a week of nightly snapshots during a ransomware infection is a week of encrypted states. And snapshots are routinely deleted by the very mechanisms, like storage reclamation and replication, that manage the data itself. Treating snapshots as backups is how organizations discover, on the worst day, that their “backups” all share the fate of the array.

Compute and network

At the compute layer, clusters group servers so that a workload survives a node failure: the database cluster fails over to a replica, the application cluster rebalances sessions, and virtualization clusters restart virtual machines on surviving hosts. The subtlety the exam likes is state. Stateless workloads, web servers behind a load balancer, fail over by the load balancer simply routing elsewhere. Stateful workloads, a login session, an in-memory queue, a database transaction, need the state to survive the node, which is why sticky sessions, shared caches, replicated databases, and careful draining procedures exist. The load balancer itself is a favorite SPOF joke, and the professional version is real: load balancers come in redundant pairs with health checks between them, and the health check is what makes the pair honest.

Network resilience is diversity. Redundant switches, links, and carriers protect against the single device or cable failing, but only if the paths are actually diverse: two fibers that share a conduit, or two carriers that share a trench, are one path wearing a costume. NIST SP 800-53 Revision 5 control CP-8, telecommunications services, carries this idea, and its CP-8(2) enhancement explicitly targets single points of failure in the alternate services: the alternate path must not share the primary’s fate. The same logic applies to power feeds, cooling loops, and uplinks. Diversity is not measured in the data center; it is measured in the ground between buildings.

The room: data center tiers

The Uptime Institute’s tier classification, a commercial standard that has become the industry’s common language, grades data center infrastructure by what it survives, and the exam expects the four tiers and the planned-versus-unplanned distinction. Tier I is basic capacity: a single path for power and cooling with no redundant components, susceptible to disruption from both planned and unplanned activities. Tier II adds redundant capacity components but keeps a single path, so it tolerates some unplanned failures but not planned maintenance without shutdown. Tier III is concurrently maintainable: N+1 capacity with multiple independent distribution paths, so any planned activity, maintenance, testing, replacement, can be performed without disrupting the IT load, and most unplanned events are survivable. Tier IV is fault tolerant: 2N capacity with multiple active paths, so the site survives any planned activity and at least one unplanned failure, often a whole path, without interruption. The commonly quoted availability figures follow the ladder, roughly 99.671% for Tier I, 99.741% for Tier II, 99.982% for Tier III, and 99.995% for Tier IV, and the exam cares less about the decimals than about the structural fact: Tier III is where maintenance stops being an outage, and Tier IV is where a single failure stops being an outage.

The design patterns inside the tiers are the exam’s building blocks: redundant power distribution units and feeds labeled A and B, uninterruptible power supplies and generators sized and fed independently, chilled-water loops and cooling units in N+1 or 2N, hot aisle and cold aisle rack arrangements that keep equipment intake air cool, and the monitoring that tells operators the environment is inside its envelope. The resilience of the room is not a feature of the equipment list; it is a feature of the arrangement, the independence, and the maintenance discipline, which is why the physical operations section of this chapter is not an afterthought.

Backups: the machinery of last resort

Redundancy handles failure. Backups handle the events redundancy cannot: accidental deletion, application corruption, insider destruction, ransomware encryption, and the fire or flood that takes the room. The distinction is the heart of the section. Replication and RAID keep a current copy of whatever exists, including whatever is wrong with it. A backup is a point-in-time copy that can be wound back to a state before the trouble. This is why ransomware defense conversations end at the backup system: the encryption that spreads through production and its replicas cannot touch a backup that is offline, immutable, or isolated, and the restore from that backup is the recovery plan.

The classic backup types define the restore math, and the exam tests the math more than the definitions. A full backup copies everything selected. An incremental backup copies only what changed since the last backup of any type, full or incremental. A differential backup copies only what changed since the last full backup. Restoring from a full-plus-incremental scheme means laying down the full backup and then replaying every incremental in order, which grows as the week progresses and is the most fragile restore because any missing incremental breaks the chain. Restoring from a full-plus-differential scheme means laying down the full backup and then the single most recent differential, which is simpler and faster, at the cost of larger daily jobs. The trade is the same one from Chapter 5: cheaper backups mean dearer restores, and the restore is the part that actually matters.

Two windows govern backup design. The backup window is the time available for backup jobs to run, and it constrains how much can be backed up and how. The restore window is the time available to bring data back, and it constrains how the backup is stored and staged. The recovery point objective from the business impact analysis sets the backup frequency: the gap between backup points must never exceed the RPO. The recovery time objective sets the restore design: the restore path must be fast enough that the total recovery meets the RTO. A backup that completes but cannot be restored within the target has failed its only job, which is why testing, covered shortly, is not an optional extra.

The media and the scheme are chosen together. Magnetic tape remains the deep archive and offline-rotation medium, with the LTO family the current generation; LTO-9, introduced in 2020, holds 18 terabytes natively per cartridge. Disk-based backup, disk-to-disk, offers fast restores and deduplication, which stores only unique data blocks so that weekly fulls consume far less than their nominal size. Object storage, especially cloud object storage, gives cheap capacity, geographic spread, and native immutability features. The rotation schemes give the scheme its shape. Grandfather-father-son, GFS, keeps daily backups on a weekly cycle, weekly backups on a monthly cycle, and monthly backups on a yearly cycle, so a recent recovery point, a mid-range recovery point, and a long-term history all exist on a bounded number of media. Tower of Hanoi is a rotation that gives finer retention of recent history and coarser retention of older history by cycling sets on a schedule derived from powers of two; the exam is satisfied by knowing it exists and that both schemes are about balancing retention depth against media count. The widely used three-two-one practice, three copies of the data on two different media types with one copy offsite, is a rule of thumb rather than a formal standard, and its offsite clause is the one that matters for the building-level disaster, as Chapter 5 noted. Electronic vaulting automates the offsite piece by transferring backup data to a remote facility on schedule.

Protection of the backups themselves is the modern chapter of the discipline. Backups of sensitive data are encrypted at rest and in transit, and the encryption is only as good as the key management: keys stored beside the data they protect, or in the same vault as the backup media, are a paperwork exercise. Immutable backups, write-once-read-many storage enforced by object locks, tape WORM media, or appliance features, cannot be modified or deleted within the retention period, even by an administrator with the master credentials, which is precisely the property ransomware must not defeat. Air-gapped backups are isolated from the production network by design: offline tape in a vault, or cloud storage reached through separate credentials and networks that a production compromise cannot reach. NIST SP 800-53 Revision 5 control CP-9, system backup, collects the discipline: backups at a frequency consistent with recovery objectives, protection of the confidentiality, integrity, and availability of the backups themselves, and enhancements for testing reliability, CP-9(1), test restoration using sampling, CP-9(2), separate storage, CP-9(3), cryptographic protection, CP-9(4), transfer to an alternate site, CP-9(5), and dual authorization for deletion, CP-9(7). ISO/IEC 27001:2022 Annex A control 8.13, information backup, makes the same demand in ISO terms: backup copies of information, software, and system images taken and tested regularly, with a backup facility at a geographically different location when the business requires it.

Testing closes the loop, and the exam treats testing as the proof. A backup is not a backup until it restores. The practice is periodic restore validation: restore a sample of data and compare it, per CP-9(2)’s sampling approach; run full restore drills against a target RTO; verify that the media can be read, that the encryption keys still exist, that the restore software matches the backup format, and that the people know the procedure. Organizations that test discover, at the rehearsal instead of the incident, that the tape drive is dead, the key is lost, the vendor’s “backup” never included the database’s transaction logs, or the restore procedure has a step that references a retired tool. Every discovery is a bargain compared with the alternative.

Recovery operations

Recovery is a procedure that must be practiced, and its working documents are runbooks: step-by-step instructions that name the order of operations, the commands and tools, the people who perform each step, and the verification that ends each one. The exam rewards the sequence inside the runbook. Recovery follows dependencies: the network and authentication infrastructure must exist before the applications that use them; the data must be restored before the services that serve it; the database comes up before the application servers that connect to it. The order in a good runbook is the reverse of the order in which the stack was built, and the runbook exists to keep a stressed team from improvising the order under a clock.

Failover and failback are the two halves of the same operation. Failover moves the service to the alternate environment, whether a standby cluster, a warm site, or a cloud region, and is the dramatic half that exercises test. Failback is the return to the primary environment and is the half that is skipped, with predictable results: the primary that was failed away from is restored, validated, and only then trusted again, and the return is itself a controlled operation with its own runbook and verification. The manager’s discipline in recovery is verification: after the service is declared back, the recovery is not over until the data is validated, the users confirm function, and the incident timeline is recorded, because those checks are what turn a restoration into a recovered business.

Validation is the metric that keeps the promises honest. The RTO and RPO are targets, not guarantees, and the only way to know whether the machinery meets them is to attempt the recovery and measure the gap. Exercises that measure time-to-restore and data-loss-at-restore convert the plan’s promises into observed numbers, and the observed numbers, not the plan’s prose, are what management reports and what the auditor reads. When the measured RTO exceeds the target, the response is design work: faster restores, staged data, warm standby instead of cold, more rehearsal. When the measured RPO exceeds the target, the response is schedule work: more frequent backups, log shipping, replication.

Cloud recovery deserves its own note because the shared responsibility model moves the boundary. NIST SP 800-145 defines the three service models the exam uses: infrastructure as a service gives the customer control over the platform layers, so the customer configures its own resilience, multi-availability-zone deployments, load balancing, backups, and failover scripts; platform as a service shifts more of the platform management to the provider while the customer still owns its data and application behavior; software as a service hands the application to the provider, whose availability commitments live in the contract, while the customer remains responsible for the data it puts in, which is why backup of SaaS data, exports, snapshots, or third-party backup services, is a customer decision and a frequent exam scenario. The operational cautions are the same ones Chapter 5 listed for cloud alternate sites: the provider’s outage is the customer’s outage, the contract must actually commit to the RTO and RPO, data location and egress costs belong in the plan, and a cloud failure is not automatically a failover event unless the failover was designed and tested. Cloud object storage adds one genuinely new capability, native immutability, which has made it a standard answer for ransomware-resistant backup.

Physical operations: the building as a system

The facility is the layer under all the other layers, and its operations are availability controls. Power, cooling, and access decide whether the RAID array, the backups, and the runbooks ever get a chance to matter. This section covers the operational side of the physical controls Chapter 14 introduced at the design level: not what the controls are, but how an organization operates them day after day, and which standards govern the operations.

Access operations

Physical access control operations run on the same lifecycle logic as logical access, which is the point of convergence, and the operations are the same craft. Badges and credentials are issued on authorization, revoked on departure, and audited on a schedule. The exam’s favorite operational failures are the revocation gap: the badge that was never deactivated, the visitor badge that outlived the visit, the key that was never returned. The controls that close the gap are process, an automated feed from human resources that triggers revocation, and evidence, access logs that would show a terminated employee’s badge being used, which is why NIST SP 800-53 Revision 5 PE-6, monitoring physical access, and PE-8, visitor access records, are operations controls rather than hardware choices.

Visitor management is the operational heart of most facility security. A visitor signs in, is authenticated against an appointment or an approval, receives a clearly distinct badge, and is escorted or tracked through restricted areas. The log, who visited, when, whom they saw, is retained and reviewable, with the privacy consideration that visitor logs are personal data and their retention falls under the same rules as other personal data. Escort policy is the discriminator between zones: open areas permit unescorted access, controlled areas require escort or prior authorization, and the most sensitive areas enforce the two-person rule. The mantrap from Chapter 14 earns its operational paragraph here: the interlocked vestibule works because the process behind it, check the badge, wait for the first door to lock, verify, open the second, is operated consistently, and because someone audits the trap’s logs for tailgating attempts.

The audit discipline is the part of access operations that makes the rest honest. Access events, door opens, badge reads, alarm clears, are logged and reviewed, per PE-6, and anomalies get investigated the way login anomalies do. The convergence payoff: badge reads belong in the same monitoring pipeline as network logins, and a badge used at 3 a.m. by an account that just changed password deserves the same suspicion either event would earn alone.

Environmental and power operations

The environment has an envelope, and operations keep the equipment inside it. ASHRAE TC 9.9, the data center thermal guidelines, defines the recommended operating conditions for the classes of data center environments; for the common A1 class the envelope is a dry-bulb temperature between 18 and 27 degrees Celsius, a dew point between 5.5 and 15 degrees Celsius, and relative humidity at or below 60 percent. The exam fact is not the exact degrees but the existence of a defined envelope and the control that monitors it: NIST SP 800-53 Revision 5 PE-14, temperature and humidity, requires monitoring and control within the defined limits, and PE-15, water damage protection, requires leak detection and protection. The operations are continuous monitoring with alarms, hot aisle and cold aisle layout maintained so that cool air actually reaches intakes, and rapid response to the alarm that says a chilled-water pipe is weeping above the raised floor.

Fire protection operations are the least-tested, highest-stakes maintenance in the building. The design from Chapter 14, clean-agent suppression and pre-action sprinklers per NFPA 75, the Standard for the Fire Protection of Information Technology Equipment, is only as good as the operational routine: detectors tested, suppressant quantities verified, discharge paths clear, and the procedures for evacuation, shutdown, and re-entry documented and drilled. Emergency lighting, PE-12, must actually light the exits. The fire that most facilities experience is not a fire at all but the response to a false alarm, and the operations that matter are the ones that make detection trustworthy and response rehearsed.

Power operations are the availability controls with the longest lead times. The chain from Chapter 14 runs utility feed, uninterruptible power supply for conditioning and short interruptions, generator for extended outages, and the emergency power off switch as the life-safety override. Operations turn the chain into a guarantee: the UPS batteries are load-tested, the generator is tested under load on a schedule and its fuel supply is managed for duration, the transfer switch is exercised so the failover path is proven, and the automatic transfer is verified to bring the load across without a gap the UPS cannot bridge. NIST SP 800-53 Revision 5 carries the controls: PE-9 for power equipment and cabling, PE-10 for emergency shutoff, PE-11 for emergency power, PE-12 for emergency lighting. NFPA 110, the standard for emergency and standby power systems, governs the testing regimen for the generator side. The operational truth the exam rewards: power resilience is a testing discipline, not an equipment invoice.

Maintenance, media, and the flow of things

Equipment needs care, and care is an attack surface. Vendor access to the server room is authorized, scheduled, logged, and escorted or monitored, with the vendor’s credentials issued for the window and revoked after. Remote maintenance, in NIST SP 800-53 Revision 5 MA-4, nonlocal maintenance, is a control with its own requirements: documented approval for each instance, monitoring of the session, and protection of the session, encryption and access controls, because a vendor’s remote session is a privileged connection into the environment. MA-6, timely maintenance, requires that maintenance occur within defined timeframes so that degraded equipment, the redundant unit running on its spare, does not linger as a single point of failure. Maintenance windows belong to change management: the window is scheduled, the change is approved and tested, the rollback is rehearsed, and the window itself, as the availability arithmetic noted, is downtime until the tier design says otherwise.

Media and equipment have a lifecycle, and the operations controls run from receipt to disposal. NIST SP 800-53 Revision 5 PE-16, delivery and removal, controls the flow at the loading dock: deliveries are inspected, couriers are identified and authorized, and the removal of equipment or media requires authorization and a record, so that the server leaving the building is accounted for and the media leaving it is sanitized. Inside the building, media handling follows the rules Chapter 10 covered for storage and disposal: sensitive media labeled, stored in secure storage, transported under control, and sanitized according to NIST SP 800-88 before reuse or disposal, with destruction logged and witnessed where the data demands it. The exam’s media questions are almost always about the controls between the states: the backup tape that traveled home, the drive that was returned without sanitization, the decommissioned array that went to the recycler with data on it.

The standards close the section. NIST SP 800-53 Revision 5’s PE family is the anchor for the physical operations: PE-2 for physical access authorizations, PE-3 for physical access control, PE-6 for monitoring, PE-8 for visitor records, PE-9 through PE-15 for power, fire, temperature, and water, PE-16 for delivery and removal, PE-17 for alternate work sites. ISO/IEC 27001:2022 Annex A controls 7.1 through 7.14 carry the same discipline in ISO form, as Chapter 14 noted. PCI DSS v4.0 Requirement 9, restrict physical access to cardholder data, is the payment industry’s version: facility entry controls, badges and identification, visitor management, physical access to devices, and the protection of point-of-interaction devices against tampering and skimming, with the requirement that access to cardholder data environments is controlled and logged. A candidate who can map a physical operations scenario to the PE control or the Annex A control that governs it has answered the way the exam scores.

How the exam reads this material

Three patterns cover most of the questions this chapter feeds. The first is the match: the scenario names a failure, and the right answer names the control that survives it. Disk death maps to RAID, deletion and corruption map to backups, encryption and ransomware map to immutable or air-gapped backups, a dead switch or carrier maps to diversity and CP-8, a planned maintenance shutdown maps to Tier III concurrent maintainability, a single path outage maps to 2N or Tier IV. The exam’s design is that each failure type has one right tool, and the wrong answers are the tools for other failure types: RAID offered against ransomware, snapshots offered against array death, replication offered against logical corruption. The discriminator to keep in mind: redundancy survives failure, backups survive corruption and loss, and only backup with immutability and isolation survives an adversary who is actively destroying copies.

The second pattern is the arithmetic. Availability percentages convert to allowed downtime; MTBF and MTTR convert to an availability fraction; backup types convert to restore chains; RPO converts to backup frequency; RTO converts to restore speed. The exam does not expect you to be a reliability engineer, but it does expect the conversions to be second nature, because every scenario question in this chapter is secretly a conversion question wearing a story.

The third pattern is the operations discipline. The questions that are not about design are about maintenance: the generator that was never tested, the badge that was never revoked, the restore that was never rehearsed, the vendor session that was not monitored. The exam’s worldview is that controls decay without operations, and the answer that describes a control as an activity, tested, reviewed, audited, exercised, is usually the answer that survives. Physical and logical security are the same system, availability is a security property, and the facility is an availability control. Hold those three statements and the domain’s questions stop being a list of facts and become a set of decisions.

Practice questions

1. A component has a mean time between failures of 90 days and a mean time to repair of 1 hour. Using the standard availability formula, what is the approximate availability of this component?

   A. 99.0% B. 99.5% C. 99.95% D. 99.999%

2. A service is guaranteed to be available 99.99% of the time over a calendar year. What is the maximum allowed downtime, approximately?

   A. 8.8 hours B. 53 minutes C. 5 minutes D. 87 hours

3. A vendor SLA promises 99.9% availability measured over each calendar month. In a 30-day month, what is the allowed downtime?

   A. About 43 minutes B. About 8.8 hours C. About 5 minutes D. About 2.6 days

4. A database server needs to survive the loss of any two of its disks simultaneously. Which RAID level directly provides this?

   A. RAID 0 B. RAID 1 C. RAID 5 D. RAID 6

5. A high-transaction database needs both performance and fault tolerance. Four disks are available, and the team wants redundancy without parity computation overhead. Which configuration is the best fit?

   A. RAID 0 B. RAID 5 C. RAID 10 D. RAID 6

6. An organization takes a full backup every Saturday night and incremental backups on the other six nights. On Thursday morning the storage array fails. Which backup sets must be restored?

   A. Saturday’s full backup and Sunday’s, Monday’s, Tuesday’s, and Wednesday’s incrementals B. Saturday’s full backup and Wednesday’s incremental only C. Thursday’s incremental and Wednesday’s differential D. Saturday’s full backup and Thursday’s incremental

7. A storage team relies on nightly array snapshots as its only data protection. A ransomware infection encrypts production data over a period of days, and the team discovers that the snapshots do not help. What is the most accurate reason?

   A. Snapshots cannot be read by the restore software B. Snapshots taken during the infection contain the encrypted data, and snapshots share the fate of the array that holds them C. Snapshots only capture metadata, not file contents D. Snapshots are automatically deleted by the backup software

8. An organization synchronously replicates its primary database to a secondary data center and considers its data protected. Which threat does this replication NOT address?

   A. Failure of the primary storage array B. Loss of the primary data center C. A ransomware process that encrypts the database in place D. Failure of a single database server

9. An organization stores backup sets in object storage with compliance-mode object locks enforcing a 30-day retention. What is this control primarily designed to protect against?

   A. Theft of the backup media B. Deletion or modification of backups, including by ransomware or a compromised administrator, within the retention period C. Unauthorized reads of backup data D. Hardware failure of the backup repository

10. A backup administrator must restore a system protected by a full-plus-differential scheme. Which sets are required?

    A. The full backup and the most recent differential B. The full backup and every differential since the last full C. The most recent differential only D. The full backup and every incremental since the last full

11. An organization performs nightly backups to a disk array located in the same server room as its production systems. Which scenario is this arrangement least able to recover from?

    A. Accidental deletion of a database table B. Corruption of a file by an application C. A fire that destroys the server room D. Ransomware encryption of a single volume

12. A company encrypts all of its backup media. An auditor asks how the encryption keys are protected. Which response represents the correct control concern?

    A. Keys should be embedded in the backup application for speed B. Keys must be stored separately from the data they protect, with their own access controls and a recovery process C. Keys must be identical across all backup media to simplify restore D. Keys should be printed and stored with the media in the vault

13. A security auditor cites NIST SP 800-53 Revision 5 and asks which control requires backups at a frequency consistent with recovery objectives, plus periodic test restoration using sampling. Which control is being referenced?

    A. SI-13 B. CP-9 C. PE-14 D. MA-6

14. An ISO/IEC 27001:2022 auditor asks which Annex A control requires redundancy of information processing facilities sufficient to meet availability requirements. Which control is correct?

    A. Annex A 8.6 B. Annex A 8.13 C. Annex A 8.14 D. Annex A 8.15

15. A data center can perform any planned maintenance on power or cooling without disrupting the IT load, and it survives most single unplanned failures, but it does not claim to survive every single failure. Per the Uptime Institute classification, which tier does this describe?

    A. Tier I B. Tier II C. Tier III D. Tier IV

16. A terminated employee’s badge was not deactivated, and the person entered the server room the following week. Which control would most directly have prevented this entry?

    A. A longer passphrase on the badge system B. Automated revocation of the badge within hours of termination, tied to the HR offboarding process C. CCTV retention of 90 days D. A mantrap at the entrance

17. A vendor technician will perform remote maintenance on a production server during a change window. Per NIST SP 800-53 Revision 5 MA-4, what must be in place for this session?

    A. A verbal agreement with the vendor B. Documented authorization, monitoring of the session, and protection such as encryption of the connection C. A dedicated air-gapped workstation for the vendor D. Approval from the vendor’s own security team

18. A server room is running at 33 degrees Celsius with relative humidity near 75 percent. Per ASHRAE TC 9.9 guidance and the NIST SP 800-53 Revision 5 control family, what is the most direct concern?

    A. The room exceeds the recommended temperature and humidity envelope, risking equipment reliability, and PE-14 requires monitoring and control within defined limits B. The room is fine as long as fire suppression is clean-agent C. The temperature only matters for human comfort D. Humidity below 20 percent is the only requirement

19. A facility loses utility power. Which sequence of controls most correctly carries the IT load from the moment of loss through the extended outage?

    A. Generator immediately, then UPS B. UPS for the short interruption, then generator for the extended outage, with the transfer switch exercised so the transition is proven C. Emergency power off, then generator D. UPS only, until utility returns

20. A payment environment stores cardholder data in a server room. Per PCI DSS v4.0 Requirement 9, which set of controls is most directly required for that facility?

    A. Fire suppression and generator testing only B. Facility entry controls, identification and badges, visitor management, control of physical access to devices, and protection of point-of-interaction devices against tampering C. Encryption of all data in the room D. SIEM monitoring of network traffic only

Answers and rationales

1. C. Availability equals MTBF divided by the sum of MTBF and MTTR. MTBF of 90 days is 2,160 hours, so the fraction is 2,160 / 2,161, approximately 99.95%. Option A confuses the percentage with the MTBF share, option B is the common “two nines plus” guess, and option D is far beyond the arithmetic. The formula is A = MTBF / (MTBF + MTTR).

2. B. A year is about 8,760 hours, and 0.01% of 8,760 hours is about 52.6 minutes. Option A is the allowance for 99.9%, option C is the allowance for 99.999%, and option D is the allowance for roughly 99%. Each additional nine removes a decimal place from allowed downtime, not a percentage point.

3. A. A 30-day month is 43,200 minutes, and 0.1% of that is about 43 minutes. The availability percentage is a proportion of the measurement period, so the allowed downtime scales with the period: 99.9% allows 8.8 hours over a year but only about 43 minutes over a 30-day month. Options B and D belong to longer or looser periods, and option C is a five-nines allowance.

4. D. RAID 6 uses dual parity across at least four disks and survives the loss of any two disks. RAID 0 survives nothing, RAID 1 survives the loss of one disk of a mirrored pair, and RAID 5 survives one disk. RAID 10 can survive two losses only if they fall in different mirror pairs, so it is not the general answer for “any two disks”.

5. C. RAID 10, mirroring then striping, gives performance without parity computation and survives a disk failure in each mirror pair, which is the standard choice for high-transaction databases needing speed and fault tolerance on four disks. RAID 0 has no fault tolerance, and RAID 5 and RAID 6 pay a parity computation cost that RAID 10 avoids.

6. A. With a weekly full and daily incrementals, restore means the full backup plus every incremental in order since it: Saturday’s full, then Sunday through Wednesday. Incrementals capture changes since the previous backup of any type, so Thursday’s job, which never ran after the array failed, is irrelevant, and the chain must include all prior incrementals, not just the most recent one. Option B describes a differential restore pattern, which only works for a differential scheme.

7. B. Snapshots are point-in-time views on the same storage as the data, so an array failure takes them down with it, and a snapshot taken after encryption began contains the encrypted state. They are a rollback tool, not a backup. Option A confuses media types, option C misstates what snapshots capture, and option D is unrelated; the failure mode is shared fate plus preserved corruption.

8. C. Synchronous replication copies every write to the secondary, including the ransomware’s encryption of the data, so the secondary holds an identical, equally destroyed dataset. Replication addresses hardware and site failure of the primary, options A, B, and D, but not logical corruption, which is why backups with immutability and isolation are the ransomware answer.

9. B. Compliance-mode object locks enforce write-once-read-many immutability: neither ransomware nor a compromised administrator can delete or modify a backup within the retention window. Theft and unauthorized reads are addressed by encryption and access control, options A and C, and hardware failure by the storage platform’s own redundancy, option D. Immutability is a defense against destruction, which is the ransomware threat.

10. A. In a full-plus-differential scheme, the differential captures everything changed since the last full, so restore is the full backup plus the single most recent differential. Option B is the full-plus-every-differential pattern, which is not how differentials are defined, and option D describes the incremental restore chain, which requires every incremental in order.

11. C. A backup in the same room as the production systems it protects shares their fate: the fire that destroys production destroys the backup. Options A, B, and D are recoverable because the backup copies and their media survive. This is the rationale behind the offsite clause of the three-two-one practice and behind alternate-site storage in controls such as CP-9(5) and Annex A 8.13.

12. B. Backup encryption is only as strong as the key management: keys must be stored separately from the data they protect, with restricted access and a documented recovery process, or the encryption is theater. Options A and C weaken separation, and option D stores keys with the media, defeating the control. The same principle appears in the cryptography chapter’s key lifecycle material.

13. B. CP-9, system backup, requires backups at a frequency consistent with recovery objectives, and its CP-9(2) enhancement requires test restoration using sampling. SI-13 is predictable failure prevention, PE-14 is temperature and humidity, and MA-6 is timely maintenance. The backup frequency and restore testing language points squarely at CP-9.

14. C. Annex A 8.14, redundancy of information processing facilities, requires redundancy sufficient to meet availability requirements. Annex A 8.6 is capacity management, 8.13 is information backup, and 8.15 is logging. The pairing of “redundancy” and “availability” in the question names 8.14.

15. C. Concurrent maintainability, the ability to perform planned work without disrupting the IT load, with N+1 capacity and multiple distribution paths, is the defining property of Tier III. Tier I and Tier II are disrupted by planned work, and Tier IV additionally tolerates at least one full-path failure without disruption, a stronger claim than the question describes.

16. B. The entry happened because the badge still worked. Automated revocation tied to offboarding, within hours of termination, closes the window directly. A mantrap checks the badge but would have let the valid badge through, CCTV only documents the entry, and a passphrase is irrelevant to badge revocation. Revocation is the operational control that turns identity termination into physical access termination.

17. B. MA-4, nonlocal maintenance, requires documented approval, monitoring of the remote maintenance session, and protection of the session, typically encryption, plus the use of approved methods and tools. A verbal agreement is not a control, an air-gapped workstation is the wrong tool for a vendor that must reach production, and the vendor’s own approval does not govern the customer’s environment. Vendor remote access is a privileged connection and is controlled as one.

18. A. ASHRAE TC 9.9 defines the recommended envelope for data center environments, roughly 18 to 27 degrees Celsius dry bulb with a dew point between 5.5 and 15 degrees Celsius and relative humidity at or below 60 percent for the A1 class, and NIST SP 800-53 Revision 5 PE-14 requires temperature and humidity to be monitored and controlled within defined limits. The reading in the question is outside the envelope and is an equipment reliability and availability concern, not a comfort matter.

19. B. The UPS carries the load through the short interruption while the generator starts, and the generator carries the extended outage. The transfer switch must be exercised so the transition is proven, because the failure that kills this chain is a generator or transfer switch that has never been tested. Options A and C misorder the chain, and option D provides no extended coverage.

20. B. PCI DSS v4.0 Requirement 9 restricts physical access to cardholder data and covers facility entry controls, identification and badges, visitor management, physical access to devices, and protection of point-of-interaction devices against tampering and skimming. Fire suppression and generator testing are operational utilities, not the focus of Requirement 9, and encryption and network monitoring are governed by other requirements in the standard.

Resilience, availability, and physical operations on one page

Resilience is a stack, and the exam questions find the layer. The arithmetic comes first: availability is a fraction, each nine removes a decimal place of downtime, and A = MTBF / (MTBF + MTTR), which is why redundancy, adding capacity so no single failure matters, is the design answer and why maintenance counts as downtime until the design makes it concurrent. The tiers of the Uptime Institute encode the same idea: Tier III is where planned work stops being an outage, Tier IV is where a single failure stops being one.

At the data layer, RAID survives disk death and nothing else, replication survives site and hardware failure and nothing else, snapshots survive changes and nothing else, and only backups survive corruption, deletion, and the adversary who destroys copies. The backup discipline is the restore math, full and incrementals in order or full plus the latest differential; the rotation that keeps a bounded history, GFS and its cousins; the offsite copy that survives the building; the encryption with separated keys; and the immutability and isolation that answer ransomware. A backup is not a backup until it restores, and CP-9 with its sampling test is the control that says so.

Recovery is a practiced procedure, runbooks in dependency order, failover and failback both drilled, and the RTO and RPO verified by measurement rather than asserted in prose. In the cloud the shared responsibility model relocates the work: the customer configures resilience in IaaS, trusts the platform in PaaS, and still owns the data in SaaS. And under everything sits the building, operated as a system: badges revoked on departure, visitors escorted and logged, the environment inside the ASHRAE envelope under PE-14, the UPS and generator tested, the vendor session authorized and monitored under MA-4, the media handled from dock to disposal under PE-16, all of it audited per the PE family and, in payment environments, per PCI DSS v4.0 Requirement 9. Physical and logical security are one system, availability is a security property, and the facility is the bottom layer of the defense.