0. Purpose and Scope

RAID 60 is commonly marketed as an “ultra-resilient” architecture, combining RAID 6 dual-parity protection with RAID 0 striping. In real-world systems, RAID 60 introduces additional structural vulnerabilities that administrators are rarely warned about — especially during power events, staggered writes, or interrupted rebuilds.

This Technical Note documents the known failure surfaces, unsafe rebuild behaviors, and forensic triage protocols for RAID 60. It serves as the internal reference behind all RAID 60 triage pages in the ADR knowledge base.

1. RAID 60 Logical Structure Overview

• Architecture: RAID 60 nests two or more RAID 6 groups under a top-level RAID 0 stripe. Each RAID 6 group maintains its own P/Q parity; the RAID 0 layer stripes data across the groups.
• Stripe Groups: Every logical stripe contains multiple sub-stripes, one per RAID 6 group. All groups must agree on stripe order, parity layout, and write epoch for the array to be consistent.
• Parity Domains: Each RAID 6 group is a separate parity domain. The RAID 0 layer is blind to parity correctness and assumes each group is internally consistent.
• Controller Assumption: Controllers assume all groups write and flush in lockstep. In practice, cache, power, and firmware behavior frequently violate this assumption.

2. Known Vulnerabilities in RAID 60

RAID 60 survives multiple drive failures within limits, but it is highly vulnerable to cross-group divergence: inconsistencies between RAID 6 groups caused by timing, caching, and controller behavior.

2.1 Cross-Group Parity Divergence

• Definition: Stripe Group A and Stripe Group B no longer represent the same write epoch or parity state for a given logical block.
• Causes: Uneven cache flush timing, background parity operations applied to one group but not the other, or controller firmware resets that roll back one group further than the other.
• Symptoms: All drives test healthy, but the virtual disk is missing, will not mount, or mounts as empty/RAW.
• Controller Behavior: The controller hides or offlines the volume to avoid performing parity math across inconsistent groups.
• Recovery Implication: Recovery requires reconstructing parity domains per group, then realigning cross-group stripe mapping offline.

2.2 Staggered Stripe-Set Failure

• Scenario: One RAID 6 group successfully commits cached writes; another group loses power or resets mid-commit.
• Result: Group A reflects epoch N for certain stripes while Group B reflects epoch N−1.
• Trigger Events: Power loss, BBU failure, abrupt shutdowns during heavy write activity, or controller resets.
• Effect on RAID 60: The RAID 0 layer sees sub-stripes that no longer match; parity becomes unrecoverable without reconstructing the correct epoch alignment.
• Admin View: Virtual disk disappears or goes offline immediately after power events or firmware changes, even though SMART shows no disk failure.

2.3 Silent Parity Mismatch

• Definition: P/Q parity values no longer match underlying data for one or more stripes, but no drive shows a hard failure.
• Causes: Latent sector errors, unlogged write failures, inconsistently applied background consistency checks, or partial rebuilds.
• Visibility: Most controllers do not surface these mismatches until a read or verify operation touches the affected region.
• Risk: A “verify” or “consistency check” on a RAID 60 with silent parity mismatch can overwrite good data with bad parity.
• Recovery Implication: Safe triage requires cloning and offline parity analysis; live repair attempts may be destructive.

2.4 Foreign Config Asymmetry

• Definition: After a reboot or configuration change, one RAID 6 group presents its metadata as current while another appears stale or foreign.
• Controller View: Some members may appear as “Unconfigured Good” or part of a foreign array while others import normally.
• Trigger Conditions: Chassis migrations, slot re-ordering, mixed firmware levels, or partial configuration saves in NVRAM.
• Behavior: The controller cannot reconcile conflicting layouts and may repeatedly offer “Import Foreign” while keeping the array offline.
• Recovery Implication: Direct foreign import is unsafe; the correct approach is to capture metadata from all members and determine the correct generation and topology offline.

3. Rebuild Behavior in RAID 60

RAID 60 rebuild operations are often deceptive. Controllers may start rebuilds that appear correct but are mathematically destructive when stripe groups have already diverged.

• Expectation: Controllers assume a stable, consistent parity domain per group while rebuild is in progress.
• Reality: Cross-group drift, epoch mismatches, and silent parity errors can cause rebuild operations to overwrite the only valid parity or data.
• Common Symptom: Rebuild that stalls at 0% or a rebuilt volume that mounts but presents empty or corrupt directories.

3.2 Partial Parity Overwrite During Recovery

• Scenario: After a power event or drive replacement, the controller attempts an automatic rebuild or consistency check on one group.
• Problem: The group being “repaired” is already out of sync with the other group(s). Rebuild or verify operations write new parity based on incomplete or incorrect assumptions.
• Effect: Valid historical parity is destroyed, eliminating the mathematical basis for a full recovery.
• Admin View: Rebuild appears to progress, but the resulting volume is unreadable, missing data, or shows massive corruption.
• Recovery Implication: Once overwritten parity spreads across stripes, recovery complexity and risk increase dramatically; imaging before any rebuild is critical.

4. Stripe-Group Reconstruction Logic (Why Most Tools Fail)

Most RAID recovery utilities assume a single parity domain and a single, monotonic stripe layout. RAID 60 violates these assumptions.

• Single-Domain Assumption: Generic tools typically model one RAID 5/6 set, not multiple parity domains.
• Epoch Assumption: They assume all members share the same write epoch and journal state.
• Geometry Assumption: They expect symmetrical geometry and contiguous, predictable chunk numbering.
• RAID 60 Reality: Multiple RAID 6 groups can hold different generation counters, chunk offsets, and parity patterns for the same logical stripes.

4.1 Parity-Domain Verification & Group Alignment

• Step 1 — Per-Group Modeling: Each RAID 6 group must be modeled independently to determine its internal stripe size, parity rotation, and generation counters.
• Step 2 — Epoch Comparison: Groups must be compared to identify which holds the most complete and coherent parity state for each region.
• Step 3 — Cross-Group Alignment: Logical stripe numbers must be mapped across groups to ensure that sub-stripes correspond to the same write epoch.
• Step 4 — Validation: Candidate layouts are tested against real data and parity equations to confirm correctness before any reconstruction is attempted.
• Outcome: Correct group alignment enables safe virtual reconstruction of RAID 60; incorrect alignment yields plausible-looking but corrupt data.

5. Power Loss, Cache Events, and Epoch Drift

• BBU / Flash Cache Issues: Loss of battery-backed cache can cause pending writes to be committed to one group but lost in another.
• Half-Committed Metadata: Some disks update header or parity metadata while others do not, breaking parity agreement across groups.
• NVRAM Mismatch: Controller NVRAM may preserve a pre-event state while disks hold a post-event state, creating generation conflicts.
• Auto-Recovery Routines: Post-power-up auto-recovery may initiate parity verification or background rebuilds that further desynchronize groups.

5.3 Epoch Drift Signatures

• Metadata Timestamps: Divergent modification times on block headers or RAID metadata between groups.
• Generation Counters: Mismatched generation or epoch counters for corresponding stripes in different groups.
• Chunk Numbering: Non-monotonic or inconsistent chunk indices when decoded across groups.
• Parity Syndromes: Different parity syndromes for logical stripes that should be identical across groups.
• Implication: Epoch drift indicates that any live rebuild or verify operation is unsafe until group alignment is understood.

6. Safe Triage Protocol for RAID 60

The primary goal of RAID 60 triage is to preserve evidence and prevent destructive writes. The following actions are not safe prior to analysis:

• Do not rebuild.
• Do not import foreign configurations.
• Do not initialize or reconfigure RAID sets.
• Do not run filesystem repair tools (fsck, chkdsk, xfs_repair, etc.) on live members.
• Do not allow consistency checks or verify operations to run.

6.2 Safe Forensic Triage — Order of Operations

1. Freeze State: Disable any automated repair, rebuild, or verification tasks at the controller level.
2. Clone Members: Create bit-for-bit images of all disks, including those reported as “healthy,” and record slot → serial → WWN mappings.
3. Extract Metadata: Read and preserve RAID metadata from each member, including group IDs, parity patterns, generation counters, and layout descriptors.
4. Model Groups Independently: Reconstruct each RAID 6 group’s internal geometry and parity behavior on the cloned images.
5. Align Groups: Determine and validate the correct cross-group stripe alignment using parity and content checks.
6. Simulate Rebuild Virtually: Perform virtual reconstruction of the RAID 60 array on cloned data to verify correctness before any on-controller repair is attempted.

7. Summary and Use in ADR Triage Pages

Technical Note TN-R60-001 formalizes how RAID 60 fails at the logic level: cross-group parity divergence, staggered stripe-set failures, silent parity mismatches, foreign-config asymmetry, and destructive rebuild behavior. It also defines ADR’s safe triage protocol and reconstruction methodology.

ADR RAID 60 triage pages may cite this note as: “ADR Technical Note TN-R60-001” using the section anchors listed below.