Netflix tcp_sack vulnerability hosing things

[BatteryKing]

No. I spent time scanning through logs captured by the daily cron job and through /var/log/messages on the receiving side, but did not see it. Is this something you think will show up on the server side, and if so, where?

[1fallen]

Depends I guess, not knowing how the Server side is set-up, but I meant on your >> or the D-Loader side. However worth the Site's Admins view also.
I have learned that cli is the best method when able, is always the best to see in real time. BTW seen with "transmission-cli"
Forgot to add: http://man7.org/linux/man-pages/man1/watch.1.html

[BatteryKing]

Ah yes, using a torrent client. However BitTorrent runs over UDP and so any lost packets are handled in user space. I am focused on https, which in this case is going over TCP while UDP based protocols such as QUIC are not available. This https server actually provides simplified 'views' to complex datasets with a dynamic server (think more in terms of accessing data views in a database), so I cannot use another method to get the requisite 'files'. Also TCP has the area of fault managed by the kernel. I am thinking a way to see what you are talking about using tools available would be iPerf, however there would need to be strong justification to try to get an iPerf test setup and I don't think I have sufficient justification to get this to happen right now.

I am thinking the best way to test is to setup a local test environment with traffic control rules setup to introduce a lot of delay and a small amount of packet loss in order to simulate the real world connection in question.

[1fallen]

I am thinking the best way to test is to setup a local test environment with traffic control rules setup to introduce a lot of delay and a small amount of packet loss in order to simulate the real world connection in question.

+1
I'm very impressed with your mannerism, with all that's going on here (Suggested Advice equates to noise at times), and at the work place.
All I can do now is offer stabs in the dark, and not really wanted/or needed here. I had actually put in my 2 week notice when we were first struck with this mess, but they wouldn't let me, and have since backed off.
This kind of stuff can wind anyone around the axle. This Thread has very interested me to point of fixation.
Here's a fun read: https://thenewstack.io/http-3-replac...d-reliability/

[BatteryKing]

I am curious as to where you are finding this mess happening as one thing I am trying to determine is how widespread the problem I see really is? I have seen so many different things go wrong to make networking an unreliable experience over the years, so for me to confine it to a kernel patch this time around is big. Especially with my 25 years using Linux this is the first time I pinned the problem to the Linux kernel and hopefully the last time.

When HTTP/3 (QUIC) gets widespread adoption, it will be interesting to see how often problems get pinned to somebody's (as in userspace) implementation causing the problem. I use SFQ (stochastic fair queuing) on my firewall specifically because older versions of Windows did not play nice with their lackluster TCP implementations, but also because UDP has no inherent traffic control and I don't want UDP based programs to go buck wild and hog the whole connection, but at the same time I prioritize certain UDP frames such as DNS. TCP is pretty dated though, so something like QUIC is really needed. WiFi and TCP is really not optimal at all both because of packet loss including the 3 way handshake and current 802.11ac is unidirectional at a time, leading to lots of slowdown as the transmission switches back and forth and doing this in true mesh WiFi networks slows things down dramatically more. Even though I am heavily wired in at home as in CAT6 and fiber to different parts of the house, my ISP loses lots of packets with their antiquated coax tech and cheaped out network in general and so I see random glitches from things like 3 way handshakes failing when they shouldn't.

[1fallen]

It was pretty wide spread, I do network monitoring (For some pretty big players in one centralized location.), so I can scan and do test's that might not go over well with your higher-ups.
I've mentioned before that I can not mention steps taken on our end. (I know how well this goes over but it is what it is.)
At first much like yourself I played around a ton with all the things you've already done on your end, but we have an Unix proprietary system with one of the best engineer's I've had the pleasure of working with, so a lot of home-brewed software is in play here on my end. Plus I still see a lot of packet loss on the client side, and it's usually on the client side where the lost packets reside.
Internet Providers (And the right person on the other end of the phone) are also a good place to ask, as they see it all.
EDIT: I used:

tcpdump -i enp0s25 -n 'ip[8]<65 and tcp[13]&0x2f=2'  | grep 'sackOK'

[oksanareynuk]

Netflix has identified several critical vulnerabilities in the Linux and FreeBSD TCP stacks that allow you to remotely initiate a kernel crash or cause excessive resource consumption when processing specially crafted TCP packets (packet-of-death). Problems are caused by errors in the handlers of the maximum size of the data block in the TCP packet (MSS, Maximum segment size) and the mechanism for selective acknowledgment of connections (SACK, TCP Selective Acknowledgment).