Firefly Open Source Community

Title: Ethernet Network locks up [Print This Page]

Author: crashoverride    Time: 6/28/2017 01:44
Title: Ethernet Network locks up
When transferring a large file over the on board ethernet, the driver will lock up.  The following is seen in the log:
  1. [ 3067.819477] NETDEV WATCHDOG: eth0 (rk_gmac-dwmac): transmit queue 0 timed out
  2. [ 3067.819715] ------------[ cut here ]------------
  3. [ 3067.819739] WARNING: at net/sched/sch_generic.c:306
  4. [ 3067.819752] Modules linked in:

  5. [ 3067.819790] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W       4.4.55 #5
  6. [ 3067.819804] Hardware name: Rockchip RK3399 Firefly Board (Linux Opensource) (DT)
  7. [ 3067.819822] task: ffffffc079648c40 ti: ffffffc079650000 task.ti: ffffffc079650000
  8. [ 3067.819852] PC is at dev_watchdog+0x1bc/0x22c
  9. [ 3067.819870] LR is at dev_watchdog+0x1bc/0x22c
  10. [ 3067.819885] pc : [<ffffff8008972ba0>] lr : [<ffffff8008972ba0>] pstate: 20000145
  11. [ 3067.819897] sp : ffffffc07ff2adf0
  12. [ 3067.819933] x29: ffffffc07ff2adf0 x28: ffffffc07ff2aef8
  13. [ 3067.820001] x27: ffffffc07ff2c388 x26: 00000000ffffffff
  14. [ 3067.820021] x25: 0000000000000002 x24: ffffffc0786a5080
  15. [ 3067.820041] x23: 0000000000000001 x22: ffffffc078f743e0
  16. [ 3067.820060] x21: ffffff80091eb000 x20: 0000000000000000
  17. [ 3067.820080] x19: ffffffc078f74000 x18: 00000000000c8115
  18. [ 3067.820099] x17: 0000000000000090 x16: 000000000000000e
  19. [ 3067.820119] x15: 000000000000000a x14: 756f2064656d6974
  20. [ 3067.820139] x13: 2030206575657571 x12: 2074696d736e6172
  21. [ 3067.820201] x11: 74203a2963616d77 x10: 642d63616d675f6b
  22. [ 3067.820220] x9 : 7228203068746520 x8 : 3a474f4448435441
  23. [ 3067.820240] x7 : 572056454454454e x6 : ffffff80091fa838
  24. [ 3067.820259] x5 : 0000000000000004 x4 : 0000000000000000
  25. [ 3067.820278] x3 : 0000000000000000 x2 : 0000000000000040
  26. [ 3067.820297] x1 : 0000000000000006 x0 : 0000000000000041
  27. [ 3067.820318]
  28.                PC: 0xffffff8008972b20:
  29. [ 3067.820333] 2b20  d65f03c0 f9402661 3717fc21 52800014 f0003984 6b02029f 540004a0 f9405001
  30. [ 3067.820460] 2b40  b5000041 f9420661 f9405803 f240047f 54000300 f940e085 b983fa63 cb050063
  31. [ 3067.820509] 2b60  ab01007f 54000265 f9405401 b00043d5 91000421 f9005401 3950daa0 35000200
  32. [ 3067.820557] 2b80  aa1303e0 97ff859b aa0003e2 b0002c40 2a1403e3 aa1303e1 9122b800 97dfa039
  33. [ 3067.820606] 2ba0  d4210000 52800020 3910daa0 14000004 11000694 91060000 17ffffdf f9410260
  34. [ 3067.820654] 2bc0  f9403801 aa1303e0 d63f0020 f0003980 b983fa61 f940e000 8b000020 97de1868
  35. [ 3067.820741] 2be0  aa0003e1 91104260 97de2116 35fff600 d538d081 f9423260 8b010000 885f7c03
  36. [ 3067.820790] 2c00  11000463 88027c03 35ffffa2 17ffffa8 a9bd7bfd 910003fd a90153f3 f90013f5
  37. [ 3067.820856]
  38.                LR: 0xffffff8008972b20:
  39. [ 3067.820892] 2b20  d65f03c0 f9402661 3717fc21 52800014 f0003984 6b02029f 540004a0 f9405001
  40. [ 3067.820941] 2b40  b5000041 f9420661 f9405803 f240047f 54000300 f940e085 b983fa63 cb050063
  41. [ 3067.820999] 2b60  ab01007f 54000265 f9405401 b00043d5 91000421 f9005401 3950daa0 35000200
  42. [ 3067.821048] 2b80  aa1303e0 97ff859b aa0003e2 b0002c40 2a1403e3 aa1303e1 9122b800 97dfa039
  43. [ 3067.821096] 2ba0  d4210000 52800020 3910daa0 14000004 11000694 91060000 17ffffdf f9410260
  44. [ 3067.821145] 2bc0  f9403801 aa1303e0 d63f0020 f0003980 b983fa61 f940e000 8b000020 97de1868
  45. [ 3067.821193] 2be0  aa0003e1 91104260 97de2116 35fff600 d538d081 f9423260 8b010000 885f7c03
  46. [ 3067.821241] 2c00  11000463 88027c03 35ffffa2 17ffffa8 a9bd7bfd 910003fd a90153f3 f90013f5
  47. [ 3067.821293]
  48.                SP: 0xffffffc07ff2ad70:
  49. [ 3067.821308] ad70  78f743e0 ffffffc0 00000001 00000000 786a5080 ffffffc0 00000002 00000000
  50. [ 3067.821356] ad90  ffffffff 00000000 7ff2c388 ffffffc0 7ff2aef8 ffffffc0 7ff2adf0 ffffffc0
  51. [ 3067.821404] adb0  08972ba0 ffffff80 7ff2adf0 ffffffc0 08972ba0 ffffff80 20000145 00000000
  52. [ 3067.821452] add0  090a5000 ffffff80 002a3b60 00000001 00000000 00000080 ffffffff 00000000
  53. [ 3067.821500] adf0  7ff2ae40 ffffffc0 080fa450 ffffff80 090a5000 ffffff80 00000100 00000000
  54. [ 3067.821548] ae10  78f74410 ffffffc0 089729e4 ffffff80 091eb000 ffffff80 78f74000 ffffffc0
  55. [ 3067.821596] ae30  78f74410 ffffffc0 08b97000 ffffff80 7ff2ae80 ffffffc0 080fa754 ffffff80
  56. [ 3067.821645] ae50  090a5000 ffffff80 091ebe10 ffffff80 089729e4 ffffff80 78f74000 ffffffc0
  57. [ 3067.821702]
  58.                X6: 0xffffff80091fa7b8:
  59. [ 3067.821716] a7b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  60. [ 3067.821764] a7d8  00000000 00000000 00000001 00000000 30303020 32303330 36373130 61205d34
  61. [ 3067.821812] a7f8  20386637 382e3736 30373132 58205d32 30203a36 66666678 38666666 31393030
  62. [ 3067.821860] a818  62376166 300a3a38 37393830 30616232 70205d3e 74617473 32203a65 30303030
  63. [ 3067.821908] a838  0a353431 4c5b200a 7469676f 20686365 69676f4c 68636574 42535520 79654b20
  64. [ 3067.821956] a858  72616f62 6f205d64 7375206e 65662d62 30306333 752e3030 312d6273 692f322e
  65. [ 3067.822004] a878  7475706e 4c0a0a30 412b2053 2b204c43 2d205a58 20345a4c 4345532b 504d4f43
  66. [ 3067.822053] a898  4c422b20 2044494b 464c452b 4c495455 4b2b2053 20444f4d 4e44492d 47200a29
  67. [ 3067.822117]
  68.                X19: 0xffffffc078f73f80:
  69. [ 3067.822131] 3f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  70. [ 3067.822178] 3fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  71. [ 3067.822225] 3fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  72. [ 3067.822272] 3fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  73. [ 3067.822319] 4000  30687465 00000000 00000000 00000000 00000000 00000000 78e01788 ffffffc0
  74. [ 3067.822367] 4020  00000000 00000000 00000000 00000000 00000000 00000000 094e0000 ffffff80
  75. [ 3067.822414] 4040  00000018 00000002 00000003 00000000 780eb050 ffffffc0 78e30050 ffffffc0
  76. [ 3067.822462] 4060  78f74a58 ffffffc0 78f74a58 ffffffc0 78f74070 ffffffc0 78f74070 ffffffc0
  77. [ 3067.822515]
  78.                X21: 0xffffff80091eaf80:
  79. [ 3067.822529] af80  00000000 00000000 78ee59c0 ffffffc0 78ee5990 ffffffc0 081418b0 ffffff80
  80. [ 3067.822578] afa0  08d04d28 ffffff80 08b5c6b0 ffffff80 08b5ae58 ffffff80 08141b50 ffffff80
  81. [ 3067.822626] afc0  08fdcb74 ffffff80 00000000 00000000 78ee5a20 ffffffc0 78ee59f0 ffffffc0
  82. [ 3067.822675] afe0  081418b0 ffffff80 08d04d28 ffffff80 08b5c618 ffffff80 08b5ad88 ffffff80
  83. [ 3067.822724] b000  08141b50 ffffff80 08fdcb08 ffffff80 00000000 00000000 78ee5a80 ffffffc0
  84. [ 3067.822772] b020  78ee5a50 ffffffc0 081418b0 ffffff80 08d04d28 ffffff80 08b5c580 ffffff80
  85. [ 3067.822821] b040  08b5acb8 ffffff80 08141b50 ffffff80 08fdca9c ffffff80 00000000 00000000
  86. [ 3067.822869] b060  78ee5ae0 ffffffc0 78ee5ab0 ffffffc0 081418b0 ffffff80 08d04d28 ffffff80
  87. [ 3067.822921]
  88.                X22: 0xffffffc078f74360:
  89. [ 3067.822936] 4360  78f74358 ffffffc0 ffffffff 0000ffff 00000000 00000000 00000000 00000000
  90. [ 3067.822984] 4380  00000000 00000000 00000000 00000000 00000000 00000000 78e01810 ffffffc0
  91. [ 3067.823032] 43a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  92. [ 3067.823079] 43c0  786a5000 ffffffc0 00000001 00000001 74810c00 ffffffc0 000003e8 00000000
  93. [ 3067.823127] 43e0  00540053 dead4ead 00000002 00000000 79648c40 ffffffc0 00001388 00000000
  94. [ 3067.823175] 4400  00000000 00000000 0023e3fc 00000001 00000200 dead0000 00000000 00000000
  95. [ 3067.823222] 4420  002a3b60 00000001 089729e4 ffffff80 78f74000 ffffffc0 00000002 ffffffff
  96. [ 3067.823271] 4440  ffffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  97. [ 3067.823322]
  98.                X24: 0xffffffc0786a5000:
  99. [ 3067.823337] 5000  78f74000 ffffffc0 74810c00 ffffffc0 74810c00 ffffffc0 786fdac0 ffffffc0
  100. [ 3067.823385] 5020  78713180 ffffffc0 7868c918 ffffffc0 787131a8 ffffffc0 78713180 ffffffc0
  101. [ 3067.823434] 5040  091b99e8 ffffff80 78715d98 ffffffc0 00000001 00000007 00000000 00000000
  102. [ 3067.823482] 5060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  103. [ 3067.823529] 5080  12001200 dead4ead ffffffff 00000000 ffffffff ffffffff ffffffff 00000000
  104. [ 3067.823578] 50a0  002a1fa2 00000001 00000001 00000000 00000006 00000000 00000000 00000000
  105. [ 3067.823625] 50c0  00072dbc 00072d97 0000002a 00000000 00000000 00000000 00000000 00000000
  106. [ 3067.823672] 50e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  107. [ 3067.823725]
  108.                X27: 0xffffffc07ff2c308:
  109. [ 3067.823739] c308  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  110. [ 3067.823786] c328  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  111. [ 3067.823833] c348  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  112. [ 3067.823880] c368  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  113. [ 3067.823927] c388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  114. [ 3067.823973] c3a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  115. [ 3067.824020] c3c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  116. [ 3067.824067] c3e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  117. [ 3067.824117]
  118.                X28: 0xffffffc07ff2ae78:
  119. [ 3067.824132] ae78  79650000 ffffffc0 7ff2af00 ffffffc0 0809faf0 ffffff80 090a5108 ffffff80
  120. [ 3067.824181] ae98  00000001 00000000 00000082 00000000 00000002 00000000 091f7000 ffffff80
  121. [ 3067.824228] aeb8  091eb668 ffffff80 00000100 00000000 0000000a 00000000 00000001 00000000
  122. [ 3067.824276] aed8  091eb000 ffffff80 7ff2af10 ffffffc0 08b869f0 00000000 794ea6a0 ffffffc0
  123. [ 3067.824325] aef8  00000000 00000000 7ff2af90 ffffffc0 0809fe98 ffffff80 00000000 00000000
  124. [ 3067.824372] af18  0907d000 ffffff80 00000000 00000000 00206400 ffffffc0 00000001 00000000
  125. [ 3067.824420] af38  79653da0 ffffffc0 00000071 00000000 7ff2b050 ffffffc0 00d88048 00000000
  126. [ 3067.824468] af58  79650000 ffffffc0 7ff2af90 ffffffc0 080e8504 ffffff80 090a5000 ffffff80
  127. [ 3067.824520]
  128.                X29: 0xffffffc07ff2ad70:
  129. [ 3067.824534] ad70  78f743e0 ffffffc0 00000001 00000000 786a5080 ffffffc0 00000002 00000000
  130. [ 3067.824582] ad90  ffffffff 00000000 7ff2c388 ffffffc0 7ff2aef8 ffffffc0 7ff2adf0 ffffffc0
  131. [ 3067.824630] adb0  08972ba0 ffffff80 7ff2adf0 ffffffc0 08972ba0 ffffff80 20000145 00000000
  132. [ 3067.824679] add0  090a5000 ffffff80 002a3b60 00000001 00000000 00000080 ffffffff 00000000
  133. [ 3067.824726] adf0  7ff2ae40 ffffffc0 080fa450 ffffff80 090a5000 ffffff80 00000100 00000000
  134. [ 3067.824775] ae10  78f74410 ffffffc0 089729e4 ffffff80 091eb000 ffffff80 78f74000 ffffffc0
  135. [ 3067.824823] ae30  78f74410 ffffffc0 08b97000 ffffff80 7ff2ae80 ffffffc0 080fa754 ffffff80
  136. [ 3067.824871] ae50  090a5000 ffffff80 091ebe10 ffffff80 089729e4 ffffff80 78f74000 ffffffc0

  137. [ 3067.824933] ---[ end trace 7f05d76b5c73f942 ]---
  138. [ 3067.824946] Call trace:
  139. [ 3067.824964] Exception stack(0xffffffc07ff2ac20 to 0xffffffc07ff2ad50)
  140. [ 3067.824985] ac20: ffffffc078f74000 0000008000000000 ffffffc07ff2adf0 ffffff8008972ba0
  141. [ 3067.825004] ac40: 0000000000000000 ffffff800923d04a 0000000000000041 0000000000000000
  142. [ 3067.825024] ac60: 000000060001c844 000000000000053b 0000000000000002 ffffff80090c55c0
  143. [ 3067.825044] ac80: ffffffc07ff2ad10 ffffff80080e7984 ffffffc07ff2ad70 ffffff8008efb8ae
  144. [ 3067.825064] aca0: ffffff80091eb000 ffffffc078f743e0 0000000000000001 ffffffc0786a5080
  145. [ 3067.825083] acc0: 0000000000000041 0000000000000006 0000000000000040 0000000000000000
  146. [ 3067.825103] ace0: 0000000000000000 0000000000000004 ffffff80091fa838 572056454454454e
  147. [ 3067.825123] ad00: 3a474f4448435441 7228203068746520 642d63616d675f6b 74203a2963616d77
  148. [ 3067.825143] ad20: 2074696d736e6172 2030206575657571 756f2064656d6974 000000000000000a
  149. [ 3067.825159] ad40: 000000000000000e 0000000000000090
  150. [ 3067.825181] [<ffffff8008972ba0>] dev_watchdog+0x1bc/0x22c
  151. [ 3067.825206] [<ffffff80080fa450>] call_timer_fn+0x7c/0x150
  152. [ 3067.825228] [<ffffff80080fa754>] run_timer_softirq+0x204/0x25c
  153. [ 3067.825249] [<ffffff800809faf0>] __do_softirq+0x14c/0x2c8
  154. [ 3067.825266] [<ffffff800809fe98>] irq_exit+0x7c/0xbc
  155. [ 3067.825286] [<ffffff80080e8868>] __handle_domain_irq+0x94/0xb8
  156. [ 3067.825305] [<ffffff8008080dd4>] gic_handle_irq+0xbc/0x180
  157. [ 3067.825321] Exception stack(0xffffffc07ff2b000 to 0xffffffc07ff2b130)
  158. [ 3067.825341] b000: ffffffc079653da0 0000008000000000 ffffffc079653ed0 ffffff80088660cc
  159. [ 3067.825361] b020: 0000000060000145 0000000000000004 ffffffc07ff27060 ffffffc07ff2b050
  160. [ 3067.825380] b040: ffffffc079653ed0 ffffffc079653da0 0000000000000000 0000000000000000
  161. [ 3067.825399] b060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  162. [ 3067.825417] b080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  163. [ 3067.825436] b0a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  164. [ 3067.825454] b0c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  165. [ 3067.825473] b0e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  166. [ 3067.825492] b100: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  167. [ 3067.825507] b120: 0000000000000000 0000000000000000
  168. [ 3067.825526] [<ffffff8008081fb4>] el1_irq+0xb4/0x140
  169. [ 3067.825550] [<ffffff80088660cc>] cpuidle_enter_state+0x1a4/0x244
  170. [ 3067.825571] [<ffffff80088661e0>] cpuidle_enter+0x34/0x44
  171. [ 3067.825591] [<ffffff80080db1e8>] call_cpuidle+0x60/0x70
  172. [ 3067.825610] [<ffffff80080db424>] cpu_startup_entry+0x22c/0x2b8
  173. [ 3067.825630] [<ffffff800808cc1c>] secondary_start_kernel+0x184/0x190
  174. [ 3067.825647] [<00000000002828e8>] 0x2828e8
Copy the code


This does not happen when using the built in WiFi.
Author: isle    Time: 6/28/2017 10:58
please check this in your kernel:
commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5
Author: T-Firefly <account@t-firefly.com>
Date:   Thu Jun 22 10:47:13 2017 +0800

    ARM64: DTS: update gmac TX/RX delay
   
    Change-Id: I080b31faa3ba427d787b995858323eeb09d3f617


Author: crashoverride    Time: 6/28/2017 20:46
The patch is present in my kernel:

  1. $ git log
  2. commit 9f76a6185a72bb87f567551f5bc664ed317f999d
  3. Author: T-Firefly <account@t-firefly.com>
  4. Date:   Thu Jun 22 14:16:11 2017 +0800

  5.     ARM64: DTS: Add 3g-pcie power enable control
  6.    
  7.     Change-Id: Ie4ef7bbe9c927af08ba3ef15acad38badba151a1

  8. commit 0e1adde7e6a307cbd7e5125ffe4bda5e224f6eb2
  9. Author: T-Firefly <account@t-firefly.com>
  10. Date:   Thu Jun 22 10:57:10 2017 +0800

  11.     Revert "arm: rockchip: remove setting for rk timer"
  12.    
  13.     This reverts commit 54b2fd94a4730667d489ec72b77b5fdcfdb16ba1.

  14. commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5
  15. Author: T-Firefly <account@t-firefly.com>
  16. Date:   Thu Jun 22 10:47:13 2017 +0800

  17.     ARM64: DTS: update gmac TX/RX delay
  18.    
  19.     Change-Id: I080b31faa3ba427d787b995858323eeb09d3f617
Copy the code


Author: crashoverride    Time: 6/30/2017 04:46
Reverting the commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5 makes it work again in testing so far.

  1.     Revert "ARM64: DTS: update gmac TX/RX delay"

  2.     This reverts commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5.
Copy the code


Author: crashoverride    Time: 7/2/2017 10:31
After further testing, even with the above commit reverted the ethernet locks up.  This happens consistently with large transfers.
Author: isle    Time: 7/3/2017 10:03
could you please test this in the latest android7.1 ?
Author: crashoverride    Time: 7/4/2017 04:22
I only have one Firefly-RK3999 for testing.  Since the eMMC is not removable, this makes it impracticable for me to re-flash my device with Android since it wipes out all my Ubuntu development work.  Additionally, there are no backup tools provided for me to make a copy of my current eMMC.

Therefore, since the Firefly team presumably has access to more than one device, it is expected they would maintain one with an Ubuntu image and one with an Android image.  The issue I posted is trivially reproducible using SFTP or smbclient.  I am sure someone on the Firefly team can test the Android equivalent.




Welcome Firefly Open Source Community (https://bbs.t-firefly.com/) Powered by Discuz! X3.1