RAID 60 Data Corruption After Drive Replacement or Rebuild
If files became corrupt after a drive replacement or rebuild, the replacement did not necessarily cause the original failure. The rebuild may have read from an unstable source, used the wrong member state, stopped partway, or exposed filesystem damage that was already present.
Stop writing to the array. Keep the removed member and the rebuild target, and do not run another rebuild or filesystem repair on the only copy.
What this condition means
A rebuild creates a new member from the sources selected by the controller. The removed drive and the target may now contain different useful portions of the same lower group.
File corruption can originate inside a lower RAID 6 group, in the stripe above the groups, or in the filesystem. Each layer must be checked in that order.
What to record
- Which member was replaced and its lower group
- Original drive, replacement target, bay, serial number, and size
- Rebuild percentage, completion status, and errors
- When corruption was first observed and which files are affected
- Any repair, restore, consistency check, or later rebuild
What not to do
- Discard the removed member
- Restart or repeat the rebuild
- Run filesystem repair against the production array
- Overwrite affected data with a restore
- Assume a completed rebuild proves the data is correct
How ADR evaluates it
ADR protects the original member and rebuild target, then determines the most defensible sources for the affected lower group.
Both RAID layers and the filesystem are validated. Priority data is recovered to separate storage rather than written back to the rebuilt array.
What happens next
Keep both the old and new member. ADR may be able to use information from either state to reconstruct the group and recover uncorrupted data.