Skip to content

quinn-proto: Fix RTT measurements with max_ack_delay timer resolution - #2681

Draft
conradludgate wants to merge 4 commits into
quinn-rs:mainfrom
conradludgate:fix-rtt-ack-delay
Draft

conradludgate wants to merge 4 commits into
quinn-rs:mainfrom
conradludgate:fix-rtt-ack-delay

Conversation

@conradludgate

Copy link
Copy Markdown

Because tokio timers are roughly 0-2ms late (uniform distribution in my testing), the max_ack_delay can sometimes be 2ms late (1ms on average). I found the following scenario would play out:

  1. sender sends a packet to receiver
  2. receiver receives a packet, queues an ACK
  3. receiver sleeps for 25ms (max_ack_delay)
  4. receiver wakes up at 26.5s (1.5ms late)
  5. ACK is sent with delay=26500 us
  6. sender clamps the delay back down to 25ms (max_ack_delay)
  7. sender RTT ends up inflated by 1.5ms

I saw the following guidance in https://www.rfc-editor.org/rfc/rfc9000.html#section-18.2-4.28.1

The maximum acknowledgment delay is an integer value indicating the maximum amount of time in milliseconds by which the endpoint will delay sending acknowledgments. This value SHOULD include the receiver's expected delays in alarms firing. For example, if a receiver sets a timer for 5ms and alarms commonly fire up to 1ms late, then it should send a max_ack_delay of 6ms.

So, if we expect a 1ms delay, we should add that to the the reported max_ack_delay.

However, I (well, Claude Fable 5) found a potential issue when testing this. From the following https://www.rfc-editor.org/rfc/rfc9002.html#section-5.3-7.4

MUST NOT subtract the acknowledgment delay from the RTT sample if the resulting value is smaller than the min_rtt. This limits the underestimation of the smoothed_rtt due to a misreporting peer.

Now that the delay is closer to what we actually expect, when performing latest_rtt - delay it has a natural tendency to fall around the min_rtt value. In the current code, if this occurs we always take the latest_rtt value to update the smoothed_rtt, which in this case would be ~27ms. So by accounting for the timing delay in the max_ack_delay, we end up making the smoothed_rtt value explode higher. This is more likely to manifest if the first handshake is slower than the network latency (eg, public-key cryptography compared to loopback network).

Right now, the simple "fix" was to clamp the adjusted_rtt to min_rtt in this event, rather than latest_rtt. But I'm not confident that it's really the correct thing to do, according to the spec.


I'll add, this problem only occurred with light load with a unidirectional stream. Once we introduced additional bidirectional streams, the problem went away. I imagine in this scenario the min_rtt will also be updated accordingly to avoid the issue described above.

@Ralith

Ralith commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR! Your compromise makes intuitive sense to me, and this isn't an interop hazard, so I'm inclined to accept that as written.

I'm ready to approve this, except for one reservation: why are we both setting the timer early and advertising extra time? Isn't that double-counting the slop?

@conradludgate

Copy link
Copy Markdown
Author

why are we both setting the timer early and advertising extra time? Isn't that double-counting the slop?

Good question, might be overcompensating. In my testing, the tokio timers would fire uniformly in the range of 0..2ms late, never early. I was accounting for the max expected delay (2 * TIMER_GRANULARITY), but the P50 delay is <=1ms.

Happy to drop down to just the 1ms slop, or happy to fix the code to only go in one direction (either under-sleep or over-report, not both)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants