genet MTU for upstream - #7617
Conversation
|
39dfaf2 contains a brutal approach to make MTU 9000 work (without any offloading). Performs quite ok, but needs more testing |
39dfaf2 to
c702d70
Compare
c702d70 to
66f6684
Compare
|
I'm still waiting on documentation from Broadcom to read the official word on how jumbo frames with offload was meant to work (if it was). Seeing as it was the offloading headers that seemed to cause issues, I did wonder if disabling offloading would allow it to work with bigger buffers. I only had a very quick read through the patches, but wonder if we can "dynamically" disable offload when the mtu is increased above the magic threshold. Possibly not based on the comment of losing the queues as well. I had considered VLAN headers, but didn't know the answer off the top of my head, and wasn't in a position to set up VLANs to test. Thanks for taking care of it. I'm not fussed over |
|
@ffainelli and @Ryceancurry if you can spare some time, your thoughts on this would be really appreciated (as always). Thanks! |
|
https://lore.kernel.org/netdev/20260406-devel-autonomous-eee-v1-1-b335e7143711@tipi-net.de/t/
https://datasheets.raspberrypi.com/cm4/cm4-datasheet.pdf#page=7
https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54210
https://magazine.raspberrypi.com/articles/raspberry-pi-4-in-detail
https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54213
While we do understand the importance of taking one step at a time, it would be great to test whether |
Can you give me more color on the failure? Do we see fragmented packets? Is the packet corrupted? Or do we not receive a RX descriptor at all? Full disclosure, I threw AI at the RTL(I'm a SW guy), it suggests a RTL bug where the RSB is reserved at every packet ready threshold. So I wonder if we are seeing a 64B hole between each 3820B chunk within the jumbo packet. At least that is the running theory right now. I will continue to dig. |
|
Status blocks off, what's the true max frame length the MAC can handle ? |
|
Thanks for looking into this too!
With status block enabled and threshold at 0xf0 I get a descriptor (one per oversized frame): 3904 = 64 (RSB) + 2 (align) + 3838. Payload is fine and matches my test pattern. I also don't see any holes, just a hart cut off. I've also tested with 3840, 5000 and 9014 B frames and all of them produce the same descriptor. Some things I've noticed and might be worth mentioning:
For contrast, with |
I've tested further and came up with a solution which allows to switch to higher MTU on a live interface (tested 1514, 4096, 8192, 9014 B frames with threshold at 0xf0). Anyway, blocks needs to be disabled for anything higher. If I keep the TSB to preserve TX checksum offload, RX is still fine at 986 Mbit/s but TX drops to 0. A while ago I proposed to get rid of the TX queues in genet [1]. Florian and Justin reviewed and tested it, but the reasoning was not good enough. Even though the queues are not absolutely blocking it,the TSB has no queue selection role anymore and it would simplify the jumbo patch. Might be worth a v2. [1] https://lore.kernel.org/netdev/20260612205915.3156127-1-nb@tipi-net.de/ |
bcmgenet_hfb_init() runs INIT_LIST_HEAD() on priv->rxnfc_list, which drops every rule off the list, and bcmgenet_open() calls it on each ifup. Every rule the user configured is silently lost: # ethtool -N eth0 flow-type ether dst $MAC action 0 Added rule with ID 0 # ethtool -n eth0 | grep -c Filter: 1 # ip link set eth0 down && ip link set eth0 up # ethtool -n eth0 | grep -c Filter: 0 Initialise the lists once at probe and restore the rules on open, as bcmgenet_resume() already does. Fixes: 3e37095 ("net: bcmgenet: add support for ethtool rxnfc flows") Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
bcmgenet_netif_stop() disables the Tx queues first and only stops Tx NAPI several steps later. A completion already in flight calls netif_tx_wake_queue() in between, so a queue runs again while bcmgenet_dma_teardown() and bcmgenet_fini_dma() free the rings, and a transmit entering that window touches freed control blocks. The close path is not affected because dev_deactivate_many() stops the qdisc before ndo_stop() runs. bcmgenet_suspend() and the MTU change added later in this series leave the qdisc running, so both can hit it. Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
bcmgenet_netif_stop() already takes stop_phy. Give the start side the same choice so a caller that left the PHY running can bring the datapath back without tripping the phy_start() state check. No functional change. Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
ENET_MAX_MTU_SIZE holds a frame length, not an MTU. Both users program it into hardware that wants a frame length, so the name will mislead once the MTU is no longer fixed at ETH_DATA_LEN. Name the receive offset too, which was open coded as 66. No functional change. Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
The driver never sets dev->max_mtu, so the MTU is stuck at ETH_DATA_LEN. Raising it alone does not work: the RBUF and TBUF packet ready thresholds cut a frame off at their 2048 byte reset default. Receive then finds no end of packet marker and drops the frame as fragmented, and transmit takes the frame but never puts it on the wire. Program both thresholds from the configured MTU and size the DMA buffers to hold exactly what the threshold permits. The registers are 8 bit in units of 16 bytes and want a multiple of the 256 byte burst size, so 0xf0 is the largest usable value. That leaves an MTU of 3820 once the alignment bytes, the Ethernet header and a VLAN tag are taken off. Changing the MTU only has to resize the buffers and rewrite those registers, so the PHY keeps running and the link stays up. Link: raspberrypi#5561 Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
The packet ready threshold is where the receive buffer writes its status block, not a limit on the frame, so anything past it arrives without an end of packet marker. That is what caps the MTU at 3820. Turn the status blocks off above ENET_MAX_MTU and the limit goes away. The receive length then comes from the descriptor. Both checksum offloads live in the blocks, so ndo_fix_features masks them out while jumbo is in use and restores them on the way back down. Measured on a CM4 at MTU 9000 with the threshold still at 0xf0: 986 Mbit/s receive, 990 transmit, payloads byte exact at 1514, 4096, 8192 and 9014 byte frames. Switching between 1500 and 9000 on a live interface works in both directions. Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
66f6684 to
36e861a
Compare
5 participants
Based on @6by9's #7614, with the goal of upstreaming the MTU support.
I've tested the original patch on CM4 and discovered some issues. So I've created a slightly different patch (series) which I intend to send to netdev. It also contains some fixes Sashiko would have flagged any way...
TBUF_PKT_RDY_THLD (TBUF + 0x10)is never programmed and it stays at0x80. At MTU3824TX iperf3 is stuck at 0.00 Mbit/s while ping works and the link is up.tx_pkts rises, buttx_good_pktsdoesn't. Kudos to @wtschueller who discovered this Jumbo frame support on Pi4 ethernet (Genet) #5561UMAC_MAX_FRAME_LENgets the MTU value, but it's a frame length and counts the FCS. Frames from3824up result inrx_length_errors(at least in my testing), so the real limit seems to be MTU3806.RX_BUF_LENGTH 10240costs no throughput (936/941 at MTU 1500, same as unpatched) but is above KMALLOC_MAX_CACHE_SIZE on arm64, thus it cant hurt to derive it from the MTU instead.0xf0seems to be the real limit:0xfbreceives fine but resulted in TX hard-hung on my setup.Happy to add @6by9 as Co-developed-by since it's based on your findings. But this requires a Signed-off, which I wouldn't add without consent.