Ethernet Network locks up

crashoverride

When transferring a large file over the on board ethernet, the driver will lock up.  The following is seen in the log:

1. [ 3067.819477] NETDEV WATCHDOG: eth0 (rk_gmac-dwmac): transmit queue 0 timed out
   
2. [ 3067.819715] ------------[ cut here ]------------
   
3. [ 3067.819739] WARNING: at net/sched/sch_generic.c:306
   
4. [ 3067.819752] Modules linked in:
   
5. 
   
6. [ 3067.819790] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W       4.4.55 #5
   
7. [ 3067.819804] Hardware name: Rockchip RK3399 Firefly Board (Linux Opensource) (DT)
   
8. [ 3067.819822] task: ffffffc079648c40 ti: ffffffc079650000 task.ti: ffffffc079650000
   
9. [ 3067.819852] PC is at dev_watchdog+0x1bc/0x22c
   
10. [ 3067.819870] LR is at dev_watchdog+0x1bc/0x22c
    
11. [ 3067.819885] pc : [<ffffff8008972ba0>] lr : [<ffffff8008972ba0>] pstate: 20000145
    
12. [ 3067.819897] sp : ffffffc07ff2adf0
    
13. [ 3067.819933] x29: ffffffc07ff2adf0 x28: ffffffc07ff2aef8
    
14. [ 3067.820001] x27: ffffffc07ff2c388 x26: 00000000ffffffff
    
15. [ 3067.820021] x25: 0000000000000002 x24: ffffffc0786a5080
    
16. [ 3067.820041] x23: 0000000000000001 x22: ffffffc078f743e0
    
17. [ 3067.820060] x21: ffffff80091eb000 x20: 0000000000000000
    
18. [ 3067.820080] x19: ffffffc078f74000 x18: 00000000000c8115
    
19. [ 3067.820099] x17: 0000000000000090 x16: 000000000000000e
    
20. [ 3067.820119] x15: 000000000000000a x14: 756f2064656d6974
    
21. [ 3067.820139] x13: 2030206575657571 x12: 2074696d736e6172
    
22. [ 3067.820201] x11: 74203a2963616d77 x10: 642d63616d675f6b
    
23. [ 3067.820220] x9 : 7228203068746520 x8 : 3a474f4448435441
    
24. [ 3067.820240] x7 : 572056454454454e x6 : ffffff80091fa838
    
25. [ 3067.820259] x5 : 0000000000000004 x4 : 0000000000000000
    
26. [ 3067.820278] x3 : 0000000000000000 x2 : 0000000000000040
    
27. [ 3067.820297] x1 : 0000000000000006 x0 : 0000000000000041
    
28. [ 3067.820318]
    
29.                PC: 0xffffff8008972b20:
    
30. [ 3067.820333] 2b20  d65f03c0 f9402661 3717fc21 52800014 f0003984 6b02029f 540004a0 f9405001
    
31. [ 3067.820460] 2b40  b5000041 f9420661 f9405803 f240047f 54000300 f940e085 b983fa63 cb050063
    
32. [ 3067.820509] 2b60  ab01007f 54000265 f9405401 b00043d5 91000421 f9005401 3950daa0 35000200
    
33. [ 3067.820557] 2b80  aa1303e0 97ff859b aa0003e2 b0002c40 2a1403e3 aa1303e1 9122b800 97dfa039
    
34. [ 3067.820606] 2ba0  d4210000 52800020 3910daa0 14000004 11000694 91060000 17ffffdf f9410260
    
35. [ 3067.820654] 2bc0  f9403801 aa1303e0 d63f0020 f0003980 b983fa61 f940e000 8b000020 97de1868
    
36. [ 3067.820741] 2be0  aa0003e1 91104260 97de2116 35fff600 d538d081 f9423260 8b010000 885f7c03
    
37. [ 3067.820790] 2c00  11000463 88027c03 35ffffa2 17ffffa8 a9bd7bfd 910003fd a90153f3 f90013f5
    
38. [ 3067.820856]
    
39.                LR: 0xffffff8008972b20:
    
40. [ 3067.820892] 2b20  d65f03c0 f9402661 3717fc21 52800014 f0003984 6b02029f 540004a0 f9405001
    
41. [ 3067.820941] 2b40  b5000041 f9420661 f9405803 f240047f 54000300 f940e085 b983fa63 cb050063
    
42. [ 3067.820999] 2b60  ab01007f 54000265 f9405401 b00043d5 91000421 f9005401 3950daa0 35000200
    
43. [ 3067.821048] 2b80  aa1303e0 97ff859b aa0003e2 b0002c40 2a1403e3 aa1303e1 9122b800 97dfa039
    
44. [ 3067.821096] 2ba0  d4210000 52800020 3910daa0 14000004 11000694 91060000 17ffffdf f9410260
    
45. [ 3067.821145] 2bc0  f9403801 aa1303e0 d63f0020 f0003980 b983fa61 f940e000 8b000020 97de1868
    
46. [ 3067.821193] 2be0  aa0003e1 91104260 97de2116 35fff600 d538d081 f9423260 8b010000 885f7c03
    
47. [ 3067.821241] 2c00  11000463 88027c03 35ffffa2 17ffffa8 a9bd7bfd 910003fd a90153f3 f90013f5
    
48. [ 3067.821293]
    
49.                SP: 0xffffffc07ff2ad70:
    
50. [ 3067.821308] ad70  78f743e0 ffffffc0 00000001 00000000 786a5080 ffffffc0 00000002 00000000
    
51. [ 3067.821356] ad90  ffffffff 00000000 7ff2c388 ffffffc0 7ff2aef8 ffffffc0 7ff2adf0 ffffffc0
    
52. [ 3067.821404] adb0  08972ba0 ffffff80 7ff2adf0 ffffffc0 08972ba0 ffffff80 20000145 00000000
    
53. [ 3067.821452] add0  090a5000 ffffff80 002a3b60 00000001 00000000 00000080 ffffffff 00000000
    
54. [ 3067.821500] adf0  7ff2ae40 ffffffc0 080fa450 ffffff80 090a5000 ffffff80 00000100 00000000
    
55. [ 3067.821548] ae10  78f74410 ffffffc0 089729e4 ffffff80 091eb000 ffffff80 78f74000 ffffffc0
    
56. [ 3067.821596] ae30  78f74410 ffffffc0 08b97000 ffffff80 7ff2ae80 ffffffc0 080fa754 ffffff80
    
57. [ 3067.821645] ae50  090a5000 ffffff80 091ebe10 ffffff80 089729e4 ffffff80 78f74000 ffffffc0
    
58. [ 3067.821702]
    
59.                X6: 0xffffff80091fa7b8:
    
60. [ 3067.821716] a7b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
61. [ 3067.821764] a7d8  00000000 00000000 00000001 00000000 30303020 32303330 36373130 61205d34
    
62. [ 3067.821812] a7f8  20386637 382e3736 30373132 58205d32 30203a36 66666678 38666666 31393030
    
63. [ 3067.821860] a818  62376166 300a3a38 37393830 30616232 70205d3e 74617473 32203a65 30303030
    
64. [ 3067.821908] a838  0a353431 4c5b200a 7469676f 20686365 69676f4c 68636574 42535520 79654b20
    
65. [ 3067.821956] a858  72616f62 6f205d64 7375206e 65662d62 30306333 752e3030 312d6273 692f322e
    
66. [ 3067.822004] a878  7475706e 4c0a0a30 412b2053 2b204c43 2d205a58 20345a4c 4345532b 504d4f43
    
67. [ 3067.822053] a898  4c422b20 2044494b 464c452b 4c495455 4b2b2053 20444f4d 4e44492d 47200a29
    
68. [ 3067.822117]
    
69.                X19: 0xffffffc078f73f80:
    
70. [ 3067.822131] 3f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
71. [ 3067.822178] 3fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
72. [ 3067.822225] 3fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
73. [ 3067.822272] 3fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
74. [ 3067.822319] 4000  30687465 00000000 00000000 00000000 00000000 00000000 78e01788 ffffffc0
    
75. [ 3067.822367] 4020  00000000 00000000 00000000 00000000 00000000 00000000 094e0000 ffffff80
    
76. [ 3067.822414] 4040  00000018 00000002 00000003 00000000 780eb050 ffffffc0 78e30050 ffffffc0
    
77. [ 3067.822462] 4060  78f74a58 ffffffc0 78f74a58 ffffffc0 78f74070 ffffffc0 78f74070 ffffffc0
    
78. [ 3067.822515]
    
79.                X21: 0xffffff80091eaf80:
    
80. [ 3067.822529] af80  00000000 00000000 78ee59c0 ffffffc0 78ee5990 ffffffc0 081418b0 ffffff80
    
81. [ 3067.822578] afa0  08d04d28 ffffff80 08b5c6b0 ffffff80 08b5ae58 ffffff80 08141b50 ffffff80
    
82. [ 3067.822626] afc0  08fdcb74 ffffff80 00000000 00000000 78ee5a20 ffffffc0 78ee59f0 ffffffc0
    
83. [ 3067.822675] afe0  081418b0 ffffff80 08d04d28 ffffff80 08b5c618 ffffff80 08b5ad88 ffffff80
    
84. [ 3067.822724] b000  08141b50 ffffff80 08fdcb08 ffffff80 00000000 00000000 78ee5a80 ffffffc0
    
85. [ 3067.822772] b020  78ee5a50 ffffffc0 081418b0 ffffff80 08d04d28 ffffff80 08b5c580 ffffff80
    
86. [ 3067.822821] b040  08b5acb8 ffffff80 08141b50 ffffff80 08fdca9c ffffff80 00000000 00000000
    
87. [ 3067.822869] b060  78ee5ae0 ffffffc0 78ee5ab0 ffffffc0 081418b0 ffffff80 08d04d28 ffffff80
    
88. [ 3067.822921]
    
89.                X22: 0xffffffc078f74360:
    
90. [ 3067.822936] 4360  78f74358 ffffffc0 ffffffff 0000ffff 00000000 00000000 00000000 00000000
    
91. [ 3067.822984] 4380  00000000 00000000 00000000 00000000 00000000 00000000 78e01810 ffffffc0
    
92. [ 3067.823032] 43a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
93. [ 3067.823079] 43c0  786a5000 ffffffc0 00000001 00000001 74810c00 ffffffc0 000003e8 00000000
    
94. [ 3067.823127] 43e0  00540053 dead4ead 00000002 00000000 79648c40 ffffffc0 00001388 00000000
    
95. [ 3067.823175] 4400  00000000 00000000 0023e3fc 00000001 00000200 dead0000 00000000 00000000
    
96. [ 3067.823222] 4420  002a3b60 00000001 089729e4 ffffff80 78f74000 ffffffc0 00000002 ffffffff
    
97. [ 3067.823271] 4440  ffffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    
98. [ 3067.823322]
    
99.                X24: 0xffffffc0786a5000:
    
100. [ 3067.823337] 5000  78f74000 ffffffc0 74810c00 ffffffc0 74810c00 ffffffc0 786fdac0 ffffffc0
     
101. [ 3067.823385] 5020  78713180 ffffffc0 7868c918 ffffffc0 787131a8 ffffffc0 78713180 ffffffc0
     
102. [ 3067.823434] 5040  091b99e8 ffffff80 78715d98 ffffffc0 00000001 00000007 00000000 00000000
     
103. [ 3067.823482] 5060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
104. [ 3067.823529] 5080  12001200 dead4ead ffffffff 00000000 ffffffff ffffffff ffffffff 00000000
     
105. [ 3067.823578] 50a0  002a1fa2 00000001 00000001 00000000 00000006 00000000 00000000 00000000
     
106. [ 3067.823625] 50c0  00072dbc 00072d97 0000002a 00000000 00000000 00000000 00000000 00000000
     
107. [ 3067.823672] 50e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
108. [ 3067.823725]
     
109.                X27: 0xffffffc07ff2c308:
     
110. [ 3067.823739] c308  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
111. [ 3067.823786] c328  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
112. [ 3067.823833] c348  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
113. [ 3067.823880] c368  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
114. [ 3067.823927] c388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
115. [ 3067.823973] c3a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
116. [ 3067.824020] c3c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
117. [ 3067.824067] c3e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
     
118. [ 3067.824117]
     
119.                X28: 0xffffffc07ff2ae78:
     
120. [ 3067.824132] ae78  79650000 ffffffc0 7ff2af00 ffffffc0 0809faf0 ffffff80 090a5108 ffffff80
     
121. [ 3067.824181] ae98  00000001 00000000 00000082 00000000 00000002 00000000 091f7000 ffffff80
     
122. [ 3067.824228] aeb8  091eb668 ffffff80 00000100 00000000 0000000a 00000000 00000001 00000000
     
123. [ 3067.824276] aed8  091eb000 ffffff80 7ff2af10 ffffffc0 08b869f0 00000000 794ea6a0 ffffffc0
     
124. [ 3067.824325] aef8  00000000 00000000 7ff2af90 ffffffc0 0809fe98 ffffff80 00000000 00000000
     
125. [ 3067.824372] af18  0907d000 ffffff80 00000000 00000000 00206400 ffffffc0 00000001 00000000
     
126. [ 3067.824420] af38  79653da0 ffffffc0 00000071 00000000 7ff2b050 ffffffc0 00d88048 00000000
     
127. [ 3067.824468] af58  79650000 ffffffc0 7ff2af90 ffffffc0 080e8504 ffffff80 090a5000 ffffff80
     
128. [ 3067.824520]
     
129.                X29: 0xffffffc07ff2ad70:
     
130. [ 3067.824534] ad70  78f743e0 ffffffc0 00000001 00000000 786a5080 ffffffc0 00000002 00000000
     
131. [ 3067.824582] ad90  ffffffff 00000000 7ff2c388 ffffffc0 7ff2aef8 ffffffc0 7ff2adf0 ffffffc0
     
132. [ 3067.824630] adb0  08972ba0 ffffff80 7ff2adf0 ffffffc0 08972ba0 ffffff80 20000145 00000000
     
133. [ 3067.824679] add0  090a5000 ffffff80 002a3b60 00000001 00000000 00000080 ffffffff 00000000
     
134. [ 3067.824726] adf0  7ff2ae40 ffffffc0 080fa450 ffffff80 090a5000 ffffff80 00000100 00000000
     
135. [ 3067.824775] ae10  78f74410 ffffffc0 089729e4 ffffff80 091eb000 ffffff80 78f74000 ffffffc0
     
136. [ 3067.824823] ae30  78f74410 ffffffc0 08b97000 ffffff80 7ff2ae80 ffffffc0 080fa754 ffffff80
     
137. [ 3067.824871] ae50  090a5000 ffffff80 091ebe10 ffffff80 089729e4 ffffff80 78f74000 ffffffc0
     
138. 
     
139. [ 3067.824933] ---[ end trace 7f05d76b5c73f942 ]---
     
140. [ 3067.824946] Call trace:
     
141. [ 3067.824964] Exception stack(0xffffffc07ff2ac20 to 0xffffffc07ff2ad50)
     
142. [ 3067.824985] ac20: ffffffc078f74000 0000008000000000 ffffffc07ff2adf0 ffffff8008972ba0
     
143. [ 3067.825004] ac40: 0000000000000000 ffffff800923d04a 0000000000000041 0000000000000000
     
144. [ 3067.825024] ac60: 000000060001c844 000000000000053b 0000000000000002 ffffff80090c55c0
     
145. [ 3067.825044] ac80: ffffffc07ff2ad10 ffffff80080e7984 ffffffc07ff2ad70 ffffff8008efb8ae
     
146. [ 3067.825064] aca0: ffffff80091eb000 ffffffc078f743e0 0000000000000001 ffffffc0786a5080
     
147. [ 3067.825083] acc0: 0000000000000041 0000000000000006 0000000000000040 0000000000000000
     
148. [ 3067.825103] ace0: 0000000000000000 0000000000000004 ffffff80091fa838 572056454454454e
     
149. [ 3067.825123] ad00: 3a474f4448435441 7228203068746520 642d63616d675f6b 74203a2963616d77
     
150. [ 3067.825143] ad20: 2074696d736e6172 2030206575657571 756f2064656d6974 000000000000000a
     
151. [ 3067.825159] ad40: 000000000000000e 0000000000000090
     
152. [ 3067.825181] [<ffffff8008972ba0>] dev_watchdog+0x1bc/0x22c
     
153. [ 3067.825206] [<ffffff80080fa450>] call_timer_fn+0x7c/0x150
     
154. [ 3067.825228] [<ffffff80080fa754>] run_timer_softirq+0x204/0x25c
     
155. [ 3067.825249] [<ffffff800809faf0>] __do_softirq+0x14c/0x2c8
     
156. [ 3067.825266] [<ffffff800809fe98>] irq_exit+0x7c/0xbc
     
157. [ 3067.825286] [<ffffff80080e8868>] __handle_domain_irq+0x94/0xb8
     
158. [ 3067.825305] [<ffffff8008080dd4>] gic_handle_irq+0xbc/0x180
     
159. [ 3067.825321] Exception stack(0xffffffc07ff2b000 to 0xffffffc07ff2b130)
     
160. [ 3067.825341] b000: ffffffc079653da0 0000008000000000 ffffffc079653ed0 ffffff80088660cc
     
161. [ 3067.825361] b020: 0000000060000145 0000000000000004 ffffffc07ff27060 ffffffc07ff2b050
     
162. [ 3067.825380] b040: ffffffc079653ed0 ffffffc079653da0 0000000000000000 0000000000000000
     
163. [ 3067.825399] b060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
     
164. [ 3067.825417] b080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
     
165. [ 3067.825436] b0a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
     
166. [ 3067.825454] b0c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
     
167. [ 3067.825473] b0e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
     
168. [ 3067.825492] b100: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
     
169. [ 3067.825507] b120: 0000000000000000 0000000000000000
     
170. [ 3067.825526] [<ffffff8008081fb4>] el1_irq+0xb4/0x140
     
171. [ 3067.825550] [<ffffff80088660cc>] cpuidle_enter_state+0x1a4/0x244
     
172. [ 3067.825571] [<ffffff80088661e0>] cpuidle_enter+0x34/0x44
     
173. [ 3067.825591] [<ffffff80080db1e8>] call_cpuidle+0x60/0x70
     
174. [ 3067.825610] [<ffffff80080db424>] cpu_startup_entry+0x22c/0x2b8
     
175. [ 3067.825630] [<ffffff800808cc1c>] secondary_start_kernel+0x184/0x190
     
176. [ 3067.825647] [<00000000002828e8>] 0x2828e8
     

Copy the code

This does not happen when using the built in WiFi.

isle

please check this in your kernel:
commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5
Author: T-Firefly <account@t-firefly.com>
Date:   Thu Jun 22 10:47:13 2017 +0800

    ARM64: DTS: update gmac TX/RX delay
   
    Change-Id: I080b31faa3ba427d787b995858323eeb09d3f617

crashoverride

The patch is present in my kernel:

1. 
   
2. $ git log
   
3. commit 9f76a6185a72bb87f567551f5bc664ed317f999d
   
4. Author: T-Firefly <account@t-firefly.com>
   
5. Date:   Thu Jun 22 14:16:11 2017 +0800
   
6. 
   
7.     ARM64: DTS: Add 3g-pcie power enable control
   
8.    
   
9.     Change-Id: Ie4ef7bbe9c927af08ba3ef15acad38badba151a1
   
10. 
    
11. commit 0e1adde7e6a307cbd7e5125ffe4bda5e224f6eb2
    
12. Author: T-Firefly <account@t-firefly.com>
    
13. Date:   Thu Jun 22 10:57:10 2017 +0800
    
14. 
    
15.     Revert "arm: rockchip: remove setting for rk timer"
    
16.    
    
17.     This reverts commit 54b2fd94a4730667d489ec72b77b5fdcfdb16ba1.
    
18. 
    
19. commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5
    
20. Author: T-Firefly <account@t-firefly.com>
    
21. Date:   Thu Jun 22 10:47:13 2017 +0800
    
22. 
    
23.     ARM64: DTS: update gmac TX/RX delay
    
24.    
    
25.     Change-Id: I080b31faa3ba427d787b995858323eeb09d3f617
    

Copy the code

crashoverride

Reverting the commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5 makes it work again in testing so far.

1. 
   
2.     Revert "ARM64: DTS: update gmac TX/RX delay"
   
3. 
   
4.     This reverts commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5.
   

Copy the code

crashoverride

After further testing, even with the above commit reverted the ethernet locks up.  This happens consistently with large transfers.

isle

could you please test this in the latest android7.1 ?

crashoverride

I only have one Firefly-RK3999 for testing.  Since the eMMC is not removable, this makes it impracticable for me to re-flash my device with Android since it wipes out all my Ubuntu development work.  Additionally, there are no backup tools provided for me to make a copy of my current eMMC.

Therefore, since the Firefly team presumably has access to more than one device, it is expected they would maintain one with an Ubuntu image and one with an Android image.  The issue I posted is trivially reproducible using SFTP or smbclient.  I am sure someone on the Firefly team can test the Android equivalent.

Roboticfloorcle

I faced a similar Ethernet network lockup issue on my system. In my case, updating the network driver and changing the cable solved the problem. While troubleshooting, I usually let my Robotic Floor Cleaner handle cleaning, which helps me focus without distractions.