mdadm RAID6 causing SATA errors

netthier

I have 4 HDDs connected via the included SATA cables:

1. # lsblk
   
2. NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   
3. sda            8:0    0  10.9T  0 disk
   
4. sdb            8:16   0  10.9T  0 disk
   
5. sdc            8:32   0  10.9T  0 disk
   
6. sdd            8:48   0  10.9T  0 disk

Copy the code

I ran the long SMART test on all drives and no errors were detected.
After creating a RAID array using

1. mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

Copy the code

The initial resync begins, but gets stuck after a few seconds.
dmesg logs are available here: https://paste.debian.net/hidden/b161164f/

Is this a problem with your SATA cables or the on-board SATA controller? Or maybe a software issue?
I will attempt switching out the cables later today to see if anything changes

Both the board and the drives are powered using an external ATX PSU, so I doubt its a power issue.

Note: last commit in the kernel directory is a95cf55eb5292c66c62fef90bd8d4abb5d776d17
Latest bundle I applied is v1.0.4a

netthier

Adding a PCIe SATA controller and connecting the drives to it worked, seems like the one on-board is not suited for RAID.

nodisknofun

Hey there,

i did the same with BTRFS-Raid and got similar errors like netthier. Would be nice if someone dig into the kernel source to fix this.


Greeting,
NoDiskNoFun

emcginnis

I can recreate this issue by writing to multiple sata drives at the same time, which means this isn't raid or btrfs-raid related and is instead is likely an issue with the sata port multiplier.
Example using dcfldd https://linux.die.net/man/1/dcfldd

1. sudo dcfldd if=/dev/urandom of=/dev/sda1 of=/dev/sdb1 count=100000

Copy the code

Which freezes after transferring ~1.5GB of data.
The kernel shows the following errors

1. [  230.178075] ata1.00: failed to read SCR 1 (Emask=0x40)
   
2. [  230.178204] ata1.01: failed to read SCR 1 (Emask=0x40)
   
3. [  230.178251] ata1.02: failed to read SCR 1 (Emask=0x40)
   
4. [  230.178289] ata1.03: failed to read SCR 1 (Emask=0x40)
   
5. [  230.178335] ata1.04: failed to read SCR 1 (Emask=0x40)
   
6. [  230.178370] ata1.05: failed to read SCR 1 (Emask=0x40)
   
7. [  230.178406] ata1.06: failed to read SCR 1 (Emask=0x40)
   
8. [  230.178441] ata1.07: failed to read SCR 1 (Emask=0x40)
   
9. [  230.178476] ata1.08: failed to read SCR 1 (Emask=0x40)
   
10. [  230.178510] ata1.09: failed to read SCR 1 (Emask=0x40)
    
11. [  230.178544] ata1.10: failed to read SCR 1 (Emask=0x40)
    
12. [  230.178576] ata1.11: failed to read SCR 1 (Emask=0x40)
    
13. [  230.178609] ata1.12: failed to read SCR 1 (Emask=0x40)
    
14. [  230.178642] ata1.13: failed to read SCR 1 (Emask=0x40)
    
15. [  230.178675] ata1.14: failed to read SCR 1 (Emask=0x40)
    
16. [  230.178731] ata1.01: exception Emask 0x100 SAct 0x1100000 SErr 0x0 action 0x6 frozen
    
17. [  230.178774] ata1.01: failed command: WRITE FPDMA QUEUED
    
18. [  230.178838] ata1.01: cmd 61/40:a0:00:08:00/05:00:00:00:00/40 tag 20 ncq dma 688128 out
    
19. [  230.178838]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
    
20. [  230.178878] ata1.01: status: { DRDY }
    
21. [  230.178910] ata1.01: failed command: WRITE FPDMA QUEUED
    
22. [  230.178969] ata1.01: cmd 61/40:c0:40:0d:00/05:00:00:00:00/40 tag 24 ncq dma 688128 out
    
23. [  230.178969]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
    
24. [  230.179002] ata1.01: status: { DRDY }
    
25. [  230.179039] ata1.03: exception Emask 0x100 SAct 0xfeefffff SErr 0x0 action 0x6 frozen

Copy the code

emcginnis

Disabling FBS/FIS fixes this issue by instead defaulting to CBS
But this is not a great workaround since CBS is super slow. CBS means the host can only do a transaction to one device at a time, whereas FBS lets the host interleave transactions between each device on the port multiplier.  

To disable FBS, and get functional sata ports, comment out the following in drivers/ata/ahci_platform.c

1.         if (of_device_is_compatible(dev->of_node, "rockchip,rk-ahci"))
   
2.                 hpriv->flags |= AHCI_HFLAG_YES_FBS;
   

Copy the code