Great thread so far. On the hybrid transport question, the thing most people underweight is which packets get which treatment. The clean rule I apply: anything that must eventually be true but is useless if late goes on TCP (matchmaking state, inventory, account data, patch verification), and anything time-sensitive that we can interpolate or discard goes on UDP (input, movement, hit registration). The failure mode that wrecks tournaments is treating them as binary—real clients use both simultaneously, and the tricky part is the interaction, not the individual protocols. For example, a game server that multiplexes movement updates over UDP but keeps chat/voice signaling on TCP will silently stall if a single lost TCP segment triggers head-of-line blocking; everyone hears the voice packet drop but the visible symptom is “the menu froze,” which is maddening to reproduce. I’ve seen orgs waste hours chasing a phantom client bug when the real culprit was a 1400-byte MSS mismatch on the WAN hop causing TCP to wedge the whole session.
For actual root-cause debugging, stop guessing and capture at the right layer. tcpdump or Wireshark with the UDP dissectors will show you retransmissions, out-of-order arrivals, and whether your application is accidentally piggybacking unreliable data on a reliable path. The tell for a real retransmission storm vs. a legit high-ping situation is the delta between send and ack timestamps—if you’re seeing SACK blocks that reference already-acknowledged sequence numbers, that’s often a client-side receive-buffer overflow or a full-duplex negotiation failure
