Summary:
Currently the `packetsWritten` observer event is only triggered when ACK-eliciting packets are written to the network. This change makes it so that the event is triggered even when non ACK-eliciting packets are written. In addition, this change increases the information provided in the observer event to reduce the processing that must be performed by observers.
More specifically, this diff makes the following changes:
- It creates two new structs — `WriteEvent` and `PacketsWrittenEvent` — and makes `PacketsWrittenEvent` and `AppLimitedEvent` inherit from `WriteEvent`, which contains the base set of fields that appear in both derived events. The base set of fields includes `writeCount` and the reference to the list of `OutstandingPackets`.
- A `PacketsWrittenEvent` is generated after each write loop during which packets are written.
- The `PacketsWrittenEvent` records the total number of packets written (the sum of ACK-eliciting and non ACK-eliciting packets written) and the total number of ACK-eliciting packets written during the last loop. By exposing this information, observers can determine whether there is value in inspecting the list of `OutstandingPackets` before doing so.
- In the future, I'll extend this event to also report the number of stream bytes written.
- It adopts a new builder pattern for observer events to make it easier to create instances of these structures:
- We use this approach instead of aggregate initialization (specifically designated initializers via aggregate initialization), because designated initializers (C++20 feature, although supported by some compilers long before) is not yet supported on all platforms for which we need this code to compile.
- I intend to change the other observer events to use this pattern as the number of arguments in some of their constructors makes them increasingly difficult to deal with.
Reviewed By: mjoras
Differential Revision: D31216724
fbshipit-source-id: 42ceb922d46ab0e0dc356f8726c2d0467a219587
Summary: Provide observers with visibility into stream events triggered by local and peer (e.g., new stream opened locally or by peer).
Reviewed By: mjoras
Differential Revision: D31886978
fbshipit-source-id: 7556fef0f336bd0f190b4474f1a7b0120aae6ef1
Summary:
The StreamManager monitors can now have a reference to a new observer interface QuicStreamPrioritiesObserver. If the reference is set, the observer is notified whenever streams are created, removed, or change priorities
QuicTransportBase implements this observer interface. It uses the event notifications to control a connection's background mode based upon the new background mode parameters: priority level threshold, and target utilization factor. When these parameters are set, and all active streams have a priority lower than the threshold, the connection's congestion controller is set to use only the defined utilization factor of the available bandwidth.
Reviewed By: bschlinker
Differential Revision: D31562505
fbshipit-source-id: 9c74fa834301f745d97e851741b911795f756023
Summary:
This diff introduces split callbacks to QUIC.
Old: callback:
* `ConnectionCallback`
Two new callbacks:
* `ConnectionSetupCallback`
* `ConnectionStreamsCallback`
For now the idea is that both old and new callbacks will live side by side and mvfst will decide which callbacks scheme to use via the `useSplitConnectionCallbacks` flag supplied on both QUIC client and server creation.
Reviewed By: mjoras
Differential Revision: D30407129
fbshipit-source-id: 7b726c2652ebe00b47f44bf291cffb9d9f6c884b
Summary:
This is another diff preparing mvfst for the new split callbacks.
This change introduces a callback dispatcher. Right now, it simply calls the existing monolithic callback everywhere. In future diffs, this dispatcher will be using the `useSplitCallbacks` flag to decide which callback object to call.
Reviewed By: mjoras
Differential Revision: D30404458
fbshipit-source-id: 5bb919185b845acc3e8ce7c596c0769b541ff69d
Summary:
For the split QUIC callbacks, onConnectionError() will be present in both connection setup and normal connection callbacks.
This diff has no functional changes rather just separates the code that will change (in diffs higher up the stack) into a few functions for easiness.
Reviewed By: mjoras, lnicco
Differential Revision: D30400469
fbshipit-source-id: df259f201b74c51704e4284cbf94ad485f80a1ab
Summary:
This change just adds a (currently no-op) flag that will be used in diffs up the stack.
The idea here is that I'll add split QUIC connection callback interfaces that will live side by side with existing single monolithic callback for now. We will experiment with split callbacks on small scale to see that there is no regressions and then will phase out the old callback gradually.
This flag is to control which callback(s) to use.
Reviewed By: mjoras
Differential Revision: D30399667
fbshipit-source-id: 8fc4e4a005e93cf6d48a987f49edee33b90dbbf1
Summary:
For each `QuicServerTransport` created by `QuicServerWorker`, we are able to set the `cmsg`s to be sent whenever there's a write.
With this we can set things like TOS/SO_MARK per connection
Reviewed By: bschlinker
Differential Revision: D29313608
fbshipit-source-id: d60c97f65681086ae1079b6b95beade95158ec59
Summary:
Add a new QuicSocket API that sets the maximum pacing rate in Bytes per second to be used if pacing is enabled. This setting is passed down to the Pacer which enforces it.
The change also includes:
- Refactoring the setPacingRate function of the Pacer to make it more consistent with the other refreshPacingRate function.
- A new flag for testing the MAX_PACING_RATE using tperf.
Reviewed By: mjoras
Differential Revision: D29276789
fbshipit-source-id: 818d86707084b2697f7417b4a47e62cbbce65c73
Summary: DeliveryCallback is a ByteEventCallback, so callers shouldn't need to change.
Reviewed By: bschlinker, mjoras
Differential Revision: D28562237
fbshipit-source-id: d7d41bbc3cbc708f7ecda5086fcba46b6a7847b0
Summary:
Previously we were using a single flag appDataSentEvents to enable 3 callbacks
(startWritingFromAppLimited, packetsWritten and appRateLimited). This commit
creates separate flags to enable each of these callbacks, we have use-cases
where Observers might want only packetsWritten events and not the others.
Also, this commit refactors the logic to deliver these 3 callbacks to all
observers into a separate method so they can be unit tested easily.
Reviewed By: bschlinker
Differential Revision: D27925011
fbshipit-source-id: 7f7436dfc3d50c3abcb8ec121b221d20e30b0c4b
Summary:
This assigns a DSRPacketizationRequestSender to a QUIC stream, and let
it own it.
Reviewed By: mjoras
Differential Revision: D27668523
fbshipit-source-id: 4beba6ddf247801368c3e1c24a0a4956490d45cd
Summary:
Instead of writing real data into the transport, we want to support a
use case where only its metadata is written to the transport. Sending of the
real data is delegated to another entity in such setup.
Reviewed By: mjoras
Differential Revision: D26131772
fbshipit-source-id: 4fcfa3a1626203f63c61898e6de089a3079d043d
Summary:
Before the change, there's no good way to recreate Cubic CC instance with custom CC factory, because Cubic is created by default.
On client side this requires calling setCongestionControl() or setTransportSettings() after calling setCongestionControllerFactory(), which is normally the case.
Reviewed By: yangchi
Differential Revision: D26401996
fbshipit-source-id: dfda39be835c67b9db42f726b3ac64c7b3d37c2f
Summary: Allows CongestionControllerFactor to access Pacer and update it's state if needed
Reviewed By: yangchi
Differential Revision: D26382984
fbshipit-source-id: fd73fbb6f0989c52a1a2061670fb9e6b188049d6
Summary: Adding QuiConnectionStats to client transport. Moving getConnectionsStats() logic from server worker into transport base class.
Reviewed By: lnicco
Differential Revision: D26316635
fbshipit-source-id: a384eee5d1bc7b23d908e0b03fafcc4ee962b0b9
Summary: This function is useful for when an application wants to explicitly reset all its non-control streams in one go.
Reviewed By: yangchi
Differential Revision: D26490209
fbshipit-source-id: 747449938e4dcd58bee028835f34776e5314cdf9
Summary: this param is passed to transport then ignored
Reviewed By: avasylev
Differential Revision: D26133327
fbshipit-source-id: 459dd0132185513215ba034f213d4137d7b56ba1
Summary:
We were using the LifecycleObserver and InstrumentationObserver classes
separately, to generate and receive callbacks.
This change migrates both these to use the unified Observer callback class and
adjusts the unit tests.
Reviewed By: bschlinker
Differential Revision: D25845845
fbshipit-source-id: c489400f5d70bccadbcc1d957136c5ade36b65ff
Summary: Adds a top level API to set stream priorities, mirroring what is currently proposed in httpbis. For now, default all streams to the highest urgency, round-robin, which mirrors the current behavior in mvfst.
Reviewed By: mjoras
Differential Revision: D20318260
fbshipit-source-id: eec625e2ab641f7fa6266517776a2ca9798e8f89
Summary:
Knob Frames are supported only by peers who are on the QUIC::MVFST version.
Expose this information to the outside, so producers of Knob frames can check,
before constructing frames to exchange with the peer.
Reviewed By: bschlinker
Differential Revision: D24608694
fbshipit-source-id: 88a8ff892efc061ca755d3eeabaed83d27d8271f
Summary: Temporarily, we use a simple json-format for transport knob. Currently it's limited to a single packet, but that should be enough for current usage.
Reviewed By: mjoras
Differential Revision: D24584261
fbshipit-source-id: f51ed0fb7560cda2ca447a49d9ad6575c1b2a59f
Summary:
After some experiments where probes are obviously causing congestion, it now
makes sense to pose some delay before sending the next probe. This is not
mentioned in the d6d spec, but in a related spec rfc4821 iirc, where consecutive
probes should have 1sec delay.
Reviewed By: mjoras
Differential Revision: D23910766
fbshipit-source-id: dcf5d05c4590489be503563c98144e12c3987cff
Summary:
This glues together the d6d lifecycle via probe timeout and raise timeout.
Had to put these two timeouts in the base transport because it has all the
necessary accountings (e.g. check close state, process callbacks) that should
happen before scheduling timeouts.
Other notable changes (included here because code is simple):
- Keep track of d6d probes in loss state. Upon second thought, it makes more
sense because we are reducing the available bandwidth as a result of sending
probes anyway. And not tracking them imposes a delay on congestion controller.
I think this does not violate the d6d spec's point of not penalizing congestion
window for d6d probes, because
- 1. we don't put losses of d6d probes in loss event. Therefore from the POV of
congestion controller, d6d probes never get lost.
- there will be at most kDefaultD6DMaxOutstandingProbes losses (i.e. 2)
that we don't tell congestion controller about. Even if those are actually
caused by congestion, it should have minimal impact because 2 is small and if there's really a congestion, the loss of normal packets should provide the signal.
- Pacing d6d probes
- Kick off d6d after a delay of 1s. This should filter out short-lived connections where probing is relatively expensive and less useful.
Reviewed By: mjoras
Differential Revision: D23736656
fbshipit-source-id: 8121fa8bcebab10a56a4e046c32c4e99ed6c3013
Summary:
This introduces a new extension frame, the KnobFrame, and an implementation.
The idea with Knobs is that they are arbitrary key-values within a namespace that can be transmitted to the peer at any time. They provide an alternative to transport settings, by eschewing the standard transport setting space entirely.
The idea is simple, each knob has a "knobspace", "id", and value. The knobspace is a namespace in which the knob has semantic meaning (e.g. 0xfaceb00). The id and value are just that, arbitrary identifiers and values.
On receiving a knob it is the application's reponsibility to deal with it.
Reviewed By: mjoras
Differential Revision: D23601468
fbshipit-source-id: 63e5e4a7bdb76e11e8c952f1234f512a629ef348
Summary: ^ This might increase readability. e.g. "cbs->offset" is more readable than "cbs->first".
Reviewed By: bschlinker, yangchi
Differential Revision: D23361655
fbshipit-source-id: 811eb4439c2c717059ec5f0d15255ec5ede9a2b8
Summary: We have an API behavior where setReadCalback will issue a StopSending on behalf of the app. This is useful but has confusing semantics as it always defaults to GenericApplicationError::NO_ERROR. Instead let the error be specified as part of the API.
Reviewed By: yangchi, lnicco
Differential Revision: D23055196
fbshipit-source-id: 755f4122bf445016c9b5adb23c3090fc23173eb9
Summary:
Adds support for timestamping on TX (TX byte events). This allows the application to determine when a byte that it previously wrote to the transport was put onto the wire.
Callbacks are processed within a new function `QuicTransportBase::processCallbacksAfterWriteData`, which is invoked by `writeSocketDataAndCatch`.
Reviewed By: mjoras
Differential Revision: D22008855
fbshipit-source-id: 99c1697cb74bb2387dbad231611be58f9392c99f
Summary:
Introduces framework for handling callbacks when a `ByteEvent` occurs.
- Adds `ByteEventCallback`, a new class that can be used to notify a callback implementation about delivery (ACK) of a byte and is used in the subsequent diff for notifications about TX of a byte.
- Adds `ByteEvent` and `ByteEventCancellation`, both of which are used to communicate ancillary information when a ByteEvent happens to the callback implementation. This makes it easier to add additional fields (such as kernel timestamps, information about ACK delays, etc) in the future, without requiring the function signatures of all callback implementations to be updated.
- For the moment, `DeliveryCallback` inherits from `ByteEventCallback` so that existing implementations do not need to be adjusted; `DeliveryCallback` implements `ByteEventCallback`'s methods and transforms the received `ByteEvents` into the expected callback. I will deprecate `DeliveryCallback` in a subsequent diff.
- Refactors existing logic for handling delivery callbacks to be generic and capable of handling different types of byte events. This allows the same logic for registering and cancelling byte events to be reused for ACK and TX timestamps; custom logic is only required to decide when to trigger the callback.
Reviewed By: mjoras
Differential Revision: D21877803
fbshipit-source-id: 74f344aa9f83ddee2a3d0056c97c62da4a581580
Summary:
Adds `QuicSocket::InstrumentationObserver`, an observer that can be registered to receive various transport events.
The ultimate goal of this class is to provide an interface similar to what we have through TCP tracepoints. This means we need to be able to register multiple callbacks.
- Initially, the first event exposed through the callback is app rate limited. In the future, we will expose retransmissions (which are loss + TLP), loss events (confirmed), spurious retransmits, RTT measurements, and raw ACK / send operations to enable throughput and goodput measurements.
- Multiple callbacks can be registered, but a `folly::small_vector` is used to minimize memory overhead in the common case of between 0 and 2 callbacks registered.
- We currently have a few different callback classes to support instrumentation, including `QuicTransportStatsCallback` and `QLogger`. However, neither of these meet our needs:
- We only support installing a single transport stats callback and QLogger callback, and they're both specialized to specific use cases. TransportStats is about understanding in aggregation how often an event (like CWND limited) is occurring, and QLogger is about logging a specific event, instead of notifying a callback about an event and allowing it to decide how to proceed.
- Ideally, we can find a way to create a callback class that handles all three cases; we can start strategizing around that as we extend `InstrumentationObserver` and identify overlap.
Differential Revision: D21923745
fbshipit-source-id: 9fb4337d55ba3e96a89dccf035f2f6978761583e
Summary:
Adds `QuicSocket::LifecycleObserver`, an observer that is notified when a socket is closed and destroyed.
- Can be used by instrumentation that ties its lifetime to that of the socket.
- Multiple observer can be registered, but a `folly::small_vector` is used to minimize memory overhead in the common case of 0 - 2 being registered.
- `folly::AsyncSocket` has a matching interface being added (D21613750), so instrumentation can follow the same paradigm for both QUIC and TCP.
- In the future, will extend to also be triggered when a transport becomes connected, similar to what we have for `AsyncSocket`.
Reviewed By: mjoras
Differential Revision: D21653651
fbshipit-source-id: 0aa374cc0dd9b9c11d81b49597326bd53264531d
Summary:
Callbacks notifying that max stream limit has been increased and new streams can be created.
This get triggered every time limit gets bumped, doesn't track if limit was previously exhausted or not.
Reviewed By: mjoras
Differential Revision: D22339814
fbshipit-source-id: 6bcd0b52adb61e1ac440d11559dc8544ad7aa1ac
Summary: It's not a great practice to leak the excpetion string via a conn close, but it is useful for the app to be able to report what the exception string was.
Reviewed By: yangchi
Differential Revision: D20628591
fbshipit-source-id: bf6eb5f33f516cec0034caed53da998643fcc120
Summary:
This ensures they are available to the whole stack rather than the transport only. The validator needs it in the server case, and will soon need it in the client case, so that seems appropriate to make it available.
Pull Request resolved: https://github.com/facebookincubator/mvfst/pull/117
Reviewed By: yangchi
Differential Revision: D20536366
Pulled By: mjoras
fbshipit-source-id: a76d369c0a82b9be1f985aed1f33f7a6b338a2ae
Summary:
Currnetly there isn't a way for apps to unregister a pending
WriteCallbacks for a stream. resetStream() does that if the transport isn't in
Closed state. This diff adds such support even if transport is already in
Closed state. This solves the problem where app has a class that's both stream
ReadCallback and stream WriteCallback and readError would kill the callback
object itself. The new API gives such class a chance to remove itself from the
transport.
Reviewed By: mjoras
Differential Revision: D20545067
fbshipit-source-id: 81d9f025310769aadef062711a49adc47a0639d0
Summary:
Currently, before server generate the destination CID, we route packets with client's address, port and client's source connection ID. But now that client can use 0-len source connection ID, the different connections from the same client address and port will be routed to the same server connections.
This diff changes it to use client's initial destination connection ID as part of the routing key.
Reviewed By: udippant
Differential Revision: D19268354
fbshipit-source-id: 837f5bd2f1e3a74957afacf7aabad922b1719219
Summary:
In QuicExceptions, in the case where the toString method was able to statically determine the response strings, we simply return the string literals in a folly::StringPiece instead of unnecessarily copying them into std::string.
Some toString methods had some dynamically generated responses and thus could not be updated. Added a TODO explaining the fact.
Reviewed By: mjoras
Differential Revision: D19192117
fbshipit-source-id: d9e5f202f9bf240009e8b8fd16f337b0506fbeb0
Summary:
In the current client code we read one packet, go back to epoll, and then read
another packet. This is not very efficient.
This changes it so that we can read multiple packets in one go from an epoll
callback.
This only performs changes on the client
Reviewed By: mjoras
Differential Revision: D18797962
fbshipit-source-id: 81be82111064ade4fe3a07b1d9d3d01e180f29f5
Summary:
F14 should be faster and have lower memory urilization for near-empty sets and maps. For most H3 connections these are mosotly going to be near-empty, so CPU wins will likely be minimal.
For usecases that have extremely high numbers of streams, there are likely going to be CPU wins.
Reviewed By: yangchi
Differential Revision: D18484047
fbshipit-source-id: 7f5616d6d6c8651ca5b03468d7d8895d1f51cb53
