Avalanche client port ignores incoming packet with tcp segment data size (MSS) greater than expected.
[link text][1][link text][2]**SR 1-1602658104** When vlan tag is used with VirtIO driver on a KVM system, the incomning packets the Avalanche received is 1522. This exceeds the MSS values that were exchanged in the SYN/SYN ACK between Av client and a apache server; 1460 MSS was advertised by both ends. This seems to be related to basic TCP. When the Avalanche client receives a packet with TCP segment size greater than the expected MSS, it completely ignores it. See Cap3_AvaVAcap_on_AvaCommander_vertio.pcap packet #16 is received with a TCP segment data of 1502. I am not sure if this is a violation and open for discussion. Is this a violation? Based on the maximum segement size, a receiver is not expected to read a packet that's greater. Because the server is sending frames bigger than 1460, Eable jumbo frame should ack to the bigger packets. [1]: /storage/temp/4544-cap3_avavacap_on_avacommander_vertio.zip [2]: /storage/temp/4530-re++update+sr-+1-1602658104+-+strange+behavior+of+avav+anywhere+with+vertio+on+kvm.zip
The answer by definition of MSS. RFC 879 November 1983 TCP Maximum Segment Size 13. The Truth The rule relating the maximum IP datagram size and the maximum TCP segment size is: TCP Maximum Segment Size = IP Maximum Datagram Size - 40 The rule must match the default case. If the TCP Maximum Segment Size option is not transmitted then the data sender is allowed to send IP datagrams of maximum size (576) with a minimum IP header (20) and a minimum TCP header (20) and thereby be able to stuff 536 octets of data into each TCP segment. The definition of the MSS option can be stated: The maximum number of data octets that may be received by the sender of this TCP option in TCP segments with no TCP header options transmitted in IP datagrams with no IP header options. 14. The Consequences When TCP is used in a situation when either the IP or TCP headers are not minimum and yet the maximum IP datagram that can be received remains 576 octets then the TCP Maximum Segment Size option must be used to reduce the limit on data octets allowed in a TCP segment. For example, if the IP Security option (11 octets) were in use and the IP maximum datagram size remained at 576 octets, then the TCP should send the MSS with a value of 525 (536-11).
Thank you so much for providing beneficial information about Linux NIC driver setting. I and customer confirmed that NIC driver's generic-receive-offload is caused of this problem. Seeing several Web Site, KVM/VirtIO NW driver seems may troubles.... Thank you so much and please close this SR. Regards,
Updates: Investigation on VertIO Driver Setting. The VertIO setting is different from normal linux NIC driver setting. Customer tried to make VertIO disabled with some way but it did not improve AvaVA TCP MSS issue. - VertIO Driver Setting - > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp-segmentation-offload: on > udp-fragmentation-offload: on > generic-segmentation-offload: on > generic-receive-offload: off > large-receive-offload: off > ntuple-filters: off receive-hashing: off - Default NIC Driver Setting- > rx-checksumming: off > tx-checksumming: off > scatter-gather: off > tcp-segmentation-offload: off > udp-fragmentation-offload: off > generic-segmentation-offload: off > generic-receive-offload: off > large-receive-offload: off > ntuple-filters: off receive-hashing: off
The real problem I noticed is the TCP segment data size in "Cap3_AvaVAcap_on_AvaCommander_vertio.pcap". Take a look at the packet #16 that the Avalanche received a TCP segment data of 1502 which exceeded the default MSS value of 1460. Therefore the packet was dropped. The subsequent packet #17 had the correct MSS and it was ack'd by the Avalanche. The pattern then repeated a few more times. It indicated that it was not an Avalanche problem. The problem is who was sending the extra TCP segment data. If "Cap1_ClearSight_Between_KVMandCatalyst.pcap" was taken from the same test then it could not be the server where it was sending MSS 1460. If the pcaps are indeed from the same test, then it points to the VirtIO that was sending the bad packets. Please confirm these pcaps to isolate the issue. As it indicates the issue is not the Avalanche I will suggest the followings. For example, check the VirtIO drivers are updated. Check the driver properties such as the Generic Segmentation Offload and the TCP Segmentation Offload settings. Try disabling them. Use the command 'ethtool -k eth0' to show the settings and compare them between default driver and virtIO driver. You can try to mask the problem by enabling Jumbo Frame on the Avalanche so that it expects the 1502 MSS. This is not a solution and I only suggest for testing/debugging to see if it would ack to all the packets. Let me know how it goes. Thank you.
16 People are following this question.
