RAID Triage Center — Real Help When RAID Goes Dark » ADR Technical Note TN-R5-001

0. Purpose and Scope

This Technical Note documents real-world RAID 5 failure behaviors observed across enterprise controllers (Broadcom/LSI MegaRAID, Dell PERC, HPE Smart Array, Adaptec/Areca) and explains why rebuilds stall, arrays disappear after a reboot, or volumes mount with missing or corrupted files. It also prescribes safe forensic procedures that prevent parity overwrite, metadata loss, or controller-induced damage during recovery.

1. RAID 5 Primer — How Single Parity Works

• Layout: Striped data plus a single distributed parity block (P) per stripe.
• Design Reality: RAID 5 tolerates exactly one physical member failure.
• Rebuild Mechanics: Missing blocks are reconstructed from parity + survivors.
• Fragility Window: Any unreadable sector on a survivor during rebuild causes immediate stall or silent corruption.
• Write Penalty: Read-modify-write cycles stress older disks, exposing latent errors.

2. Failure Surfaces That Break Single-Parity Arrays

• Silent Sector Errors (LSEs): The most common cause of rebuild stalls and partial reconstruction.
• Read-Modify-Write Drift: Cache or timing irregularities produce parity mismatches.
• Cache/NVRAM Corruption: Partial writes or cache loss desynchronize stripe ordering.
• Stale or Foreign Metadata: Controller imports outdated or partially written metadata blocks.
• Thermal/Power Instability: Momentary drops during rebuild push survivors offline.

3. Rebuild Failure Modes

• Stall at X%: Survivor has unreadable sectors; controller loops retries.
• “Rebuild Completed” but Files Corrupt: Parity reconstructed incorrectly due to silent errors.
• Replacement Disk Dropped: Controller misidentifies survivor as defective.
• Consistency Check Damage: Controller overwrites data during parity “fix-up.”

4. Metadata Desynchronization After Power Events

• Trigger Conditions: Reboot mid-write, aborted rebuild, wrong disk insertion, stale cache writes.
• Effects: Volume disappears, mounts empty, or displays corrupted file structures.
• Triage Indicators: Stripe size mismatch, unexpected parity rotation, stale “last known good” headers.

5. Single-Drive Failure Hazards

• False Stability: RAID 5 appears healthy until survivor failures occur during heavy I/O.
• Second Drive Drop: Background processes cause additional drives to panic offline.
• File Loss During Degraded Operation: Missing stripes create silent corruption.

6. Why RAID 5 Fails After a Routine Reboot

• Mechanism: Stale cache or partial stripe writes invalidate parity alignment.
• Symptoms: “Foreign Config Detected,” “Virtual Disk Incomplete,” “Array Offline — Missing Member.”
• Results: Wrong logical order imported; volume mounts empty or damaged.

7. Controller Behaviors That Matter

• Foreign Config Handling: Controllers often import stale metadata automatically.
• Cache Policy Impact: BBU failure forces write-through → disk timeouts → drive drops.
• Patrol Read / Background Verify: On degraded RAID 5, these operations cause additional corruption.

8. Correct Order of Operations — ADR SOP

1. Stop all writes immediately.
2. Clone all surviving members sector-by-sector.
3. Capture controller config (XML/NVRAM).
4. Validate logical order, block size, and stripe geometry.
5. Map parity rotation and verify stripe boundaries.
6. Identify unreadable sectors; isolate damaged zones.
7. Reconstruct virtual RAID in software (never rebuild on controller).
8. Extract recovered data to a separate device.