Firefly Open Source Community

Title: Ethernet Network locks up [Print This Page]
Author: crashoverride    Time: 6/28/2017 01:44
Title: Ethernet Network locks up
When transferring a large file over the on board ethernet, the driver will lock up.  The following is seen in the log:
  1. [ 3067.819477] NETDEV WATCHDOG: eth0 (rk_gmac-dwmac): transmit queue 0 timed out
  2. [ 3067.819715] ------------[ cut here ]------------
  3. [ 3067.819739] WARNING: at net/sched/sch_generic.c:306
  4. [ 3067.819752] Modules linked in:

  6. [ 3067.819790] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W       4.4.55 #5
  7. [ 3067.819804] Hardware name: Rockchip RK3399 Firefly Board (Linux Opensource) (DT)
  8. [ 3067.819822] task: ffffffc079648c40 ti: ffffffc079650000 task.ti: ffffffc079650000
  9. [ 3067.819852] PC is at dev_watchdog+0x1bc/0x22c
  10. [ 3067.819870] LR is at dev_watchdog+0x1bc/0x22c
  11. [ 3067.819885] pc : [<ffffff8008972ba0>] lr : [<ffffff8008972ba0>] pstate: 20000145
  12. [ 3067.819897] sp : ffffffc07ff2adf0
  13. [ 3067.819933] x29: ffffffc07ff2adf0 x28: ffffffc07ff2aef8
  14. [ 3067.820001] x27: ffffffc07ff2c388 x26: 00000000ffffffff
  15. [ 3067.820021] x25: 0000000000000002 x24: ffffffc0786a5080
  16. [ 3067.820041] x23: 0000000000000001 x22: ffffffc078f743e0
  17. [ 3067.820060] x21: ffffff80091eb000 x20: 0000000000000000
  18. [ 3067.820080] x19: ffffffc078f74000 x18: 00000000000c8115
  19. [ 3067.820099] x17: 0000000000000090 x16: 000000000000000e
  20. [ 3067.820119] x15: 000000000000000a x14: 756f2064656d6974
  21. [ 3067.820139] x13: 2030206575657571 x12: 2074696d736e6172
  22. [ 3067.820201] x11: 74203a2963616d77 x10: 642d63616d675f6b
  23. [ 3067.820220] x9 : 7228203068746520 x8 : 3a474f4448435441
  24. [ 3067.820240] x7 : 572056454454454e x6 : ffffff80091fa838
  25. [ 3067.820259] x5 : 0000000000000004 x4 : 0000000000000000
  26. [ 3067.820278] x3 : 0000000000000000 x2 : 0000000000000040
  27. [ 3067.820297] x1 : 0000000000000006 x0 : 0000000000000041
  28. [ 3067.820318]
  29.                PC: 0xffffff8008972b20:
  30. [ 3067.820333] 2b20  d65f03c0 f9402661 3717fc21 52800014 f0003984 6b02029f 540004a0 f9405001
  31. [ 3067.820460] 2b40  b5000041 f9420661 f9405803 f240047f 54000300 f940e085 b983fa63 cb050063
  32. [ 3067.820509] 2b60  ab01007f 54000265 f9405401 b00043d5 91000421 f9005401 3950daa0 35000200
  33. [ 3067.820557] 2b80  aa1303e0 97ff859b aa0003e2 b0002c40 2a1403e3 aa1303e1 9122b800 97dfa039
  34. [ 3067.820606] 2ba0  d4210000 52800020 3910daa0 14000004 11000694 91060000 17ffffdf f9410260
  35. [ 3067.820654] 2bc0  f9403801 aa1303e0 d63f0020 f0003980 b983fa61 f940e000 8b000020 97de1868
  36. [ 3067.820741] 2be0  aa0003e1 91104260 97de2116 35fff600 d538d081 f9423260 8b010000 885f7c03
  37. [ 3067.820790] 2c00  11000463 88027c03 35ffffa2 17ffffa8 a9bd7bfd 910003fd a90153f3 f90013f5
  38. [ 3067.820856]
  39.                LR: 0xffffff8008972b20:
  40. [ 3067.820892] 2b20  d65f03c0 f9402661 3717fc21 52800014 f0003984 6b02029f 540004a0 f9405001
  41. [ 3067.820941] 2b40  b5000041 f9420661 f9405803 f240047f 54000300 f940e085 b983fa63 cb050063
  42. [ 3067.820999] 2b60  ab01007f 54000265 f9405401 b00043d5 91000421 f9005401 3950daa0 35000200
  43. [ 3067.821048] 2b80  aa1303e0 97ff859b aa0003e2 b0002c40 2a1403e3 aa1303e1 9122b800 97dfa039
  44. [ 3067.821096] 2ba0  d4210000 52800020 3910daa0 14000004 11000694 91060000 17ffffdf f9410260
  45. [ 3067.821145] 2bc0  f9403801 aa1303e0 d63f0020 f0003980 b983fa61 f940e000 8b000020 97de1868
  46. [ 3067.821193] 2be0  aa0003e1 91104260 97de2116 35fff600 d538d081 f9423260 8b010000 885f7c03
  47. [ 3067.821241] 2c00  11000463 88027c03 35ffffa2 17ffffa8 a9bd7bfd 910003fd a90153f3 f90013f5
  48. [ 3067.821293]
  49.                SP: 0xffffffc07ff2ad70:
  50. [ 3067.821308] ad70  78f743e0 ffffffc0 00000001 00000000 786a5080 ffffffc0 00000002 00000000
  51. [ 3067.821356] ad90  ffffffff 00000000 7ff2c388 ffffffc0 7ff2aef8 ffffffc0 7ff2adf0 ffffffc0
  52. [ 3067.821404] adb0  08972ba0 ffffff80 7ff2adf0 ffffffc0 08972ba0 ffffff80 20000145 00000000
  53. [ 3067.821452] add0  090a5000 ffffff80 002a3b60 00000001 00000000 00000080 ffffffff 00000000
  54. [ 3067.821500] adf0  7ff2ae40 ffffffc0 080fa450 ffffff80 090a5000 ffffff80 00000100 00000000
  55. [ 3067.821548] ae10  78f74410 ffffffc0 089729e4 ffffff80 091eb000 ffffff80 78f74000 ffffffc0
  56. [ 3067.821596] ae30  78f74410 ffffffc0 08b97000 ffffff80 7ff2ae80 ffffffc0 080fa754 ffffff80
  57. [ 3067.821645] ae50  090a5000 ffffff80 091ebe10 ffffff80 089729e4 ffffff80 78f74000 ffffffc0
  58. [ 3067.821702]
  59.                X6: 0xffffff80091fa7b8:
  60. [ 3067.821716] a7b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  61. [ 3067.821764] a7d8  00000000 00000000 00000001 00000000 30303020 32303330 36373130 61205d34
  62. [ 3067.821812] a7f8  20386637 382e3736 30373132 58205d32 30203a36 66666678 38666666 31393030
  63. [ 3067.821860] a818  62376166 300a3a38 37393830 30616232 70205d3e 74617473 32203a65 30303030
  64. [ 3067.821908] a838  0a353431 4c5b200a 7469676f 20686365 69676f4c 68636574 42535520 79654b20
  65. [ 3067.821956] a858  72616f62 6f205d64 7375206e 65662d62 30306333 752e3030 312d6273 692f322e
  66. [ 3067.822004] a878  7475706e 4c0a0a30 412b2053 2b204c43 2d205a58 20345a4c 4345532b 504d4f43
  67. [ 3067.822053] a898  4c422b20 2044494b 464c452b 4c495455 4b2b2053 20444f4d 4e44492d 47200a29
  68. [ 3067.822117]
  69.                X19: 0xffffffc078f73f80:
  70. [ 3067.822131] 3f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  71. [ 3067.822178] 3fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  72. [ 3067.822225] 3fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  73. [ 3067.822272] 3fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  74. [ 3067.822319] 4000  30687465 00000000 00000000 00000000 00000000 00000000 78e01788 ffffffc0
  75. [ 3067.822367] 4020  00000000 00000000 00000000 00000000 00000000 00000000 094e0000 ffffff80
  76. [ 3067.822414] 4040  00000018 00000002 00000003 00000000 780eb050 ffffffc0 78e30050 ffffffc0
  77. [ 3067.822462] 4060  78f74a58 ffffffc0 78f74a58 ffffffc0 78f74070 ffffffc0 78f74070 ffffffc0
  78. [ 3067.822515]
  79.                X21: 0xffffff80091eaf80:
  80. [ 3067.822529] af80  00000000 00000000 78ee59c0 ffffffc0 78ee5990 ffffffc0 081418b0 ffffff80
  81. [ 3067.822578] afa0  08d04d28 ffffff80 08b5c6b0 ffffff80 08b5ae58 ffffff80 08141b50 ffffff80
  82. [ 3067.822626] afc0  08fdcb74 ffffff80 00000000 00000000 78ee5a20 ffffffc0 78ee59f0 ffffffc0
  83. [ 3067.822675] afe0  081418b0 ffffff80 08d04d28 ffffff80 08b5c618 ffffff80 08b5ad88 ffffff80
  84. [ 3067.822724] b000  08141b50 ffffff80 08fdcb08 ffffff80 00000000 00000000 78ee5a80 ffffffc0
  85. [ 3067.822772] b020  78ee5a50 ffffffc0 081418b0 ffffff80 08d04d28 ffffff80 08b5c580 ffffff80
  86. [ 3067.822821] b040  08b5acb8 ffffff80 08141b50 ffffff80 08fdca9c ffffff80 00000000 00000000
  87. [ 3067.822869] b060  78ee5ae0 ffffffc0 78ee5ab0 ffffffc0 081418b0 ffffff80 08d04d28 ffffff80
  88. [ 3067.822921]
  89.                X22: 0xffffffc078f74360:
  90. [ 3067.822936] 4360  78f74358 ffffffc0 ffffffff 0000ffff 00000000 00000000 00000000 00000000
  91. [ 3067.822984] 4380  00000000 00000000 00000000 00000000 00000000 00000000 78e01810 ffffffc0
  92. [ 3067.823032] 43a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  93. [ 3067.823079] 43c0  786a5000 ffffffc0 00000001 00000001 74810c00 ffffffc0 000003e8 00000000
  94. [ 3067.823127] 43e0  00540053 dead4ead 00000002 00000000 79648c40 ffffffc0 00001388 00000000
  95. [ 3067.823175] 4400  00000000 00000000 0023e3fc 00000001 00000200 dead0000 00000000 00000000
  96. [ 3067.823222] 4420  002a3b60 00000001 089729e4 ffffff80 78f74000 ffffffc0 00000002 ffffffff
  97. [ 3067.823271] 4440  ffffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  98. [ 3067.823322]
  99.                X24: 0xffffffc0786a5000:
  100. [ 3067.823337] 5000  78f74000 ffffffc0 74810c00 ffffffc0 74810c00 ffffffc0 786fdac0 ffffffc0
  101. [ 3067.823385] 5020  78713180 ffffffc0 7868c918 ffffffc0 787131a8 ffffffc0 78713180 ffffffc0
  102. [ 3067.823434] 5040  091b99e8 ffffff80 78715d98 ffffffc0 00000001 00000007 00000000 00000000
  103. [ 3067.823482] 5060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  104. [ 3067.823529] 5080  12001200 dead4ead ffffffff 00000000 ffffffff ffffffff ffffffff 00000000
  105. [ 3067.823578] 50a0  002a1fa2 00000001 00000001 00000000 00000006 00000000 00000000 00000000
  106. [ 3067.823625] 50c0  00072dbc 00072d97 0000002a 00000000 00000000 00000000 00000000 00000000
  107. [ 3067.823672] 50e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  108. [ 3067.823725]
  109.                X27: 0xffffffc07ff2c308:
  110. [ 3067.823739] c308  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  111. [ 3067.823786] c328  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  112. [ 3067.823833] c348  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  113. [ 3067.823880] c368  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  114. [ 3067.823927] c388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  115. [ 3067.823973] c3a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  116. [ 3067.824020] c3c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  117. [ 3067.824067] c3e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  118. [ 3067.824117]
  119.                X28: 0xffffffc07ff2ae78:
  120. [ 3067.824132] ae78  79650000 ffffffc0 7ff2af00 ffffffc0 0809faf0 ffffff80 090a5108 ffffff80
  121. [ 3067.824181] ae98  00000001 00000000 00000082 00000000 00000002 00000000 091f7000 ffffff80
  122. [ 3067.824228] aeb8  091eb668 ffffff80 00000100 00000000 0000000a 00000000 00000001 00000000
  123. [ 3067.824276] aed8  091eb000 ffffff80 7ff2af10 ffffffc0 08b869f0 00000000 794ea6a0 ffffffc0
  124. [ 3067.824325] aef8  00000000 00000000 7ff2af90 ffffffc0 0809fe98 ffffff80 00000000 00000000
  125. [ 3067.824372] af18  0907d000 ffffff80 00000000 00000000 00206400 ffffffc0 00000001 00000000
  126. [ 3067.824420] af38  79653da0 ffffffc0 00000071 00000000 7ff2b050 ffffffc0 00d88048 00000000
  127. [ 3067.824468] af58  79650000 ffffffc0 7ff2af90 ffffffc0 080e8504 ffffff80 090a5000 ffffff80
  128. [ 3067.824520]
  129.                X29: 0xffffffc07ff2ad70:
  130. [ 3067.824534] ad70  78f743e0 ffffffc0 00000001 00000000 786a5080 ffffffc0 00000002 00000000
  131. [ 3067.824582] ad90  ffffffff 00000000 7ff2c388 ffffffc0 7ff2aef8 ffffffc0 7ff2adf0 ffffffc0
  132. [ 3067.824630] adb0  08972ba0 ffffff80 7ff2adf0 ffffffc0 08972ba0 ffffff80 20000145 00000000
  133. [ 3067.824679] add0  090a5000 ffffff80 002a3b60 00000001 00000000 00000080 ffffffff 00000000
  134. [ 3067.824726] adf0  7ff2ae40 ffffffc0 080fa450 ffffff80 090a5000 ffffff80 00000100 00000000
  135. [ 3067.824775] ae10  78f74410 ffffffc0 089729e4 ffffff80 091eb000 ffffff80 78f74000 ffffffc0
  136. [ 3067.824823] ae30  78f74410 ffffffc0 08b97000 ffffff80 7ff2ae80 ffffffc0 080fa754 ffffff80
  137. [ 3067.824871] ae50  090a5000 ffffff80 091ebe10 ffffff80 089729e4 ffffff80 78f74000 ffffffc0

  139. [ 3067.824933] ---[ end trace 7f05d76b5c73f942 ]---
  140. [ 3067.824946] Call trace:
  141. [ 3067.824964] Exception stack(0xffffffc07ff2ac20 to 0xffffffc07ff2ad50)
  142. [ 3067.824985] ac20: ffffffc078f74000 0000008000000000 ffffffc07ff2adf0 ffffff8008972ba0
  143. [ 3067.825004] ac40: 0000000000000000 ffffff800923d04a 0000000000000041 0000000000000000
  144. [ 3067.825024] ac60: 000000060001c844 000000000000053b 0000000000000002 ffffff80090c55c0
  145. [ 3067.825044] ac80: ffffffc07ff2ad10 ffffff80080e7984 ffffffc07ff2ad70 ffffff8008efb8ae
  146. [ 3067.825064] aca0: ffffff80091eb000 ffffffc078f743e0 0000000000000001 ffffffc0786a5080
  147. [ 3067.825083] acc0: 0000000000000041 0000000000000006 0000000000000040 0000000000000000
  148. [ 3067.825103] ace0: 0000000000000000 0000000000000004 ffffff80091fa838 572056454454454e
  149. [ 3067.825123] ad00: 3a474f4448435441 7228203068746520 642d63616d675f6b 74203a2963616d77
  150. [ 3067.825143] ad20: 2074696d736e6172 2030206575657571 756f2064656d6974 000000000000000a
  151. [ 3067.825159] ad40: 000000000000000e 0000000000000090
  152. [ 3067.825181] [<ffffff8008972ba0>] dev_watchdog+0x1bc/0x22c
  153. [ 3067.825206] [<ffffff80080fa450>] call_timer_fn+0x7c/0x150
  154. [ 3067.825228] [<ffffff80080fa754>] run_timer_softirq+0x204/0x25c
  155. [ 3067.825249] [<ffffff800809faf0>] __do_softirq+0x14c/0x2c8
  156. [ 3067.825266] [<ffffff800809fe98>] irq_exit+0x7c/0xbc
  157. [ 3067.825286] [<ffffff80080e8868>] __handle_domain_irq+0x94/0xb8
  158. [ 3067.825305] [<ffffff8008080dd4>] gic_handle_irq+0xbc/0x180
  159. [ 3067.825321] Exception stack(0xffffffc07ff2b000 to 0xffffffc07ff2b130)
  160. [ 3067.825341] b000: ffffffc079653da0 0000008000000000 ffffffc079653ed0 ffffff80088660cc
  161. [ 3067.825361] b020: 0000000060000145 0000000000000004 ffffffc07ff27060 ffffffc07ff2b050
  162. [ 3067.825380] b040: ffffffc079653ed0 ffffffc079653da0 0000000000000000 0000000000000000
  163. [ 3067.825399] b060: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  164. [ 3067.825417] b080: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  165. [ 3067.825436] b0a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  166. [ 3067.825454] b0c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  167. [ 3067.825473] b0e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  168. [ 3067.825492] b100: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  169. [ 3067.825507] b120: 0000000000000000 0000000000000000
  170. [ 3067.825526] [<ffffff8008081fb4>] el1_irq+0xb4/0x140
  171. [ 3067.825550] [<ffffff80088660cc>] cpuidle_enter_state+0x1a4/0x244
  172. [ 3067.825571] [<ffffff80088661e0>] cpuidle_enter+0x34/0x44
  173. [ 3067.825591] [<ffffff80080db1e8>] call_cpuidle+0x60/0x70
  174. [ 3067.825610] [<ffffff80080db424>] cpu_startup_entry+0x22c/0x2b8
  175. [ 3067.825630] [<ffffff800808cc1c>] secondary_start_kernel+0x184/0x190
  176. [ 3067.825647] [<00000000002828e8>] 0x2828e8
Copy the code


This does not happen when using the built in WiFi.
Author: isle    Time: 6/28/2017 10:58
please check this in your kernel:
commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5
Author: T-Firefly <account@t-firefly.com>
Date:   Thu Jun 22 10:47:13 2017 +0800

    ARM64: DTS: update gmac TX/RX delay
   
    Change-Id: I080b31faa3ba427d787b995858323eeb09d3f617

Author: crashoverride    Time: 6/28/2017 20:46
The patch is present in my kernel:

  2. $ git log
  3. commit 9f76a6185a72bb87f567551f5bc664ed317f999d
  4. Author: T-Firefly <account@t-firefly.com>
  5. Date:   Thu Jun 22 14:16:11 2017 +0800

  7.     ARM64: DTS: Add 3g-pcie power enable control
  8.    
  9.     Change-Id: Ie4ef7bbe9c927af08ba3ef15acad38badba151a1

  11. commit 0e1adde7e6a307cbd7e5125ffe4bda5e224f6eb2
  12. Author: T-Firefly <account@t-firefly.com>
  13. Date:   Thu Jun 22 10:57:10 2017 +0800

  15.     Revert "arm: rockchip: remove setting for rk timer"
  16.    
  17.     This reverts commit 54b2fd94a4730667d489ec72b77b5fdcfdb16ba1.

  19. commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5
  20. Author: T-Firefly <account@t-firefly.com>
  21. Date:   Thu Jun 22 10:47:13 2017 +0800

  23.     ARM64: DTS: update gmac TX/RX delay
  24.    
  25.     Change-Id: I080b31faa3ba427d787b995858323eeb09d3f617
Copy the code

Author: crashoverride    Time: 6/30/2017 04:46
Reverting the commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5 makes it work again in testing so far.

  2.     Revert "ARM64: DTS: update gmac TX/RX delay"

  4.     This reverts commit d0ad8bcdd913ef2b0222fb790c9f88dfc08fc1e5.
Copy the code

Author: crashoverride    Time: 7/2/2017 10:31
After further testing, even with the above commit reverted the ethernet locks up.  This happens consistently with large transfers.
Author: isle    Time: 7/3/2017 10:03
could you please test this in the latest android7.1 ?
Author: crashoverride    Time: 7/4/2017 04:22
I only have one Firefly-RK3999 for testing.  Since the eMMC is not removable, this makes it impracticable for me to re-flash my device with Android since it wipes out all my Ubuntu development work.  Additionally, there are no backup tools provided for me to make a copy of my current eMMC.

Therefore, since the Firefly team presumably has access to more than one device, it is expected they would maintain one with an Ubuntu image and one with an Android image.  The issue I posted is trivially reproducible using SFTP or smbclient.  I am sure someone on the Firefly team can test the Android equivalent.



Welcome Firefly Open Source Community (https://bbs.t-firefly.com/) Powered by Discuz! X3.1