The FORGIVE protocol

Distributed ML training exchanges gradient updates on every step. With packet trimming on and congestion control off, my most congested ASTRA-sim fabric hit a 25 percent retransmission rate. With DCQCN on, the trim rate was sevenfold lower and the training run took 24 percent longer to complete. Each step incurs one of those costs.

Gradient descent tolerates some lost gradient bytes, but not on every step and not without limit. I call the protocol Fabric-Overload Relief: Gradients under Iteration-Varying Exemption. It forgives only gradient payload and only after a trim: neither tensor-parallel nor pipeline-parallel traffic is eligible. The loss budget tightens during the critical learning regime, so a critical step has a smaller loss budget than a non-critical one.

A trim reports a missing range

When a queue fills, a switch with packet trimming trims the packet to its header and forwards that header on a high-priority queue. The receiver learns which bytes are missing at once, not from a timeout a millisecond later. After a drop the receiver has nothing to decide. After a trim it knows the missing range and can decide what to do about the range.

NDP introduced packet trimming in 2017. Ultra Ethernet made packet trimming optional switch behaviour in specification 1.0, June 2025, alongside a default bulk mode that sprays packets across paths.

Gradient tolerance varies by step

Accordion reads changes in gradient norms to find critical learning regimes, the periods where a model is especially sensitive to compression. Accordion compresses lightly there and hard everywhere else, for up to 5.5 times better compression at accuracy comparable to uncompressed training. DBLP applies Accordion’s schedule to network transport. A hash draw at the sender sheds whole gradient messages: few during the critical learning regime, more after it.

A published tolerance depends on what the study gave up for it. MLT profiled 16 convolutional and recurrent models: under 3 percent of gradient bytes droppable at fixed rounds and accuracy, 10 percent when more rounds were allowed. Weintraub and colleagues dropped a tenth of Llama 2 7B’s gradient bytes uniformly at random for 1.17 percent worse perplexity, and 40 percent for 6.65 percent worse.

In FORGIVE a tolerance is a per-step fraction of a rank’s gradient bytes, and it applies only to the gradient all-reduce. FORGIVE adapts Accordion’s two-level compression schedule as two such loss probabilities: $P_{\text{low}}$ covers critical steps and $P_{\text{high}}$ the rest. ASTRA-sim simulates communication and compute times, not the model, so Accordion has no gradient norms to read. I treat steps 1, 2, 3 and 20 of the 20 as critical. The literature puts a model’s sensitivity to lost gradients early in training, and three of those four are early.

The loss budget

A per-model bound holds for the whole run and cannot change between phases. FORGIVE gives each receiving rank one loss budget per training step, the unit at which a step is classified critical or not. The collective schedule already says how many gradient bytes a step will bring a rank, so the receiver can size that loss budget before the first byte arrives: the step’s tolerance $p$ times those bytes. Forgiven bytes never exceed the loss budget.

A trimmed header carries the packet’s original sequence number and length, so the receiver can compute how many of those bytes are still missing. A re-segmented retransmission can straddle the cumulative acknowledgement, and its full length would charge the loss budget for bytes already delivered. The receiver then either forgives the range and acknowledges past the hole, or requests its retransmission, as the transport already does.

FORGIVE vests the loss budget: it becomes available in proportion to the bytes that arrive. A loss budget available in full at step start is exhausted by the step’s first burst. With $\text{received}$ the distinct gradient bytes a rank has received this step and $\text{forgiven}$ the bytes it has forgiven, a missing range of length $\text{len}$ may be forgiven only if $\text{forgiven} + \text{len} \le p \times (\text{received} + \text{forgiven} + \text{len})$ , equivalently $\text{forgiven} + \text{len} \le \frac{p}{1 - p} \times \text{received}$ . Forgiven bytes are at most a fraction $p$ of the bytes received or forgiven so far. At step completion, $\text{received} + \text{forgiven}$ equals the step’s expected bytes, $\text{expected}$ , so $\text{forgiven} \le p \times \text{expected}$ still holds; vesting adds a bound at every instant before step completion. A refused range is requested for retransmission and reconsidered if it is trimmed again.

A range that passes the vesting rule is then forgiven with probability $P$ , drawn fresh for each trimmed header, with $0 \le P \le 1$ and $P = 1$ by default. A range the draw refuses is requested for retransmission and charges nothing. It stays outstanding until it arrives or a later trim of it is forgiven. $P$ changes how much of the loss budget is charged. $P$ leaves the total bound and the vesting rule as they are.

Selective repeat removes the loss-recovery saving

I built the loss schedule first, as sender-side shedding at admission. It appeared to cut training time by 4 to 11 percent. The transport underneath was go-back-N: a control run with selective repeat cut the same burst’s completion time from 1.8 s to 29 ms. Go-back-N put up to 79 bytes on the wire for every byte the policy removed. I had measured the transport’s amplification, not the policy.

So I went looking for the time the burst was supposed to cost. I swept eight fabrics at 64 ranks and 400 Gbps. The fabrics varied congestion control, incast ratio and oversubscription between 2:1 and 4:1. In every fabric the burst cost under 1 percent of the 20-step training time. Skipping a round of loss recovery saves at most a round trip per flow, under 0.2 percent of an all-reduce.

The remaining cost came from congestion control. With DCQCN on, millions of rate cuts per run reduced the trim rate by a factor of eight to ten and added 18 to 25 percent to the 20-step training time. A loss budget can cover the trims those rate cuts avoid.

Congestion-control exemption

Forgiving the trims does not stop the rate cuts. So an eligible flow’s sender ignores congestion signals too, for as long as the receiver reports that the loss budget is not exhausted.

The exemption covers ECN marks as well as trims. Switches here begin ECN-marking at 800 KB of queue occupancy and trim only when the 4 MiB data queue is full, so marks arrive long before trims: at least 74 percent of rate cuts in the worst fabric came from marks, which forgiveness does not affect. Ignoring trims alone would leave those rate cuts in place, so eligible senders ignore every congestion notification packet.

Each receiving rank owns its loss budget, and all flows to that rank share it. Senders cannot see how much remains. A retransmission request does not tell them either, since vesting and the draw refuse ranges the loss budget could still cover. So the receiver’s acknowledgements and retransmission requests carry two flags. exemption-eligible is fixed when the flow is created: gradient all-reduce payload on a step with $p > 0$ . budget-exhausted is set while $\text{forgiven} + \text{outstanding} + \text{one packet} > p \times \text{expected}$ , where $\text{outstanding}$ counts distinct trimmed bytes neither arrived nor forgiven, and one packet is 4,096 bytes, the largest range the next trim can report. A sender ignores congestion signals while its flow is eligible and the loss budget is not exhausted. The sender obeys congestion control again when a report sets budget-exhausted, and ignores congestion signals again if a later report clears the flag. The latest report wins.

State and decision

FORGIVE assumes a fabric and an interface. From the fabric:

Packet trimming, on a lossy RDMA network.
A lossless high-priority queue for the trimmed headers.
Selective repeat, and a receiver that already reassembles out of order because the fabric sprays packets across paths.
A rate-based congestion control underneath.

From the software above the transport, four facts that no packet contains:

Which step is beginning, and what its gradient norms say.
How many gradient bytes the step will bring this rank.
Which flows carry them.
Which step a given flow belongs to.

Those facts cross a host-local interface between NCCL and the network interface card, so FORGIVE adds no packet type. Its only wire change is the two flags on the receiver’s feedback.

The switch trims, reports the missing range, and decides nothing about it. The scoreboard and the forgiveness decision live at the receiver, the mode bit at the sender.

A detector classifies each step, and it must run before the step’s gradient traffic starts. Accordion’s criterion, the rate of change in gradient norms, puts the step inside or outside the critical learning regime, and that classification picks the loss budget. The protocol has no list of critical steps.

The parameters are the two loss probabilities and the forgiveness probability:

P_low: float. The loss probability for the eligible gradient bytes a rank expects on a critical step. 0.005 here.
P_high: float. The same probability on every other step. 0.1 here.
P: float. The probability of forgiving a range that passes the vesting rule, drawn fresh per trimmed header. 1 by default.

State per flow, at the receiver:

rcv_nxt: Seq. The lowest sequence number not yet settled, by arrival or by forgiveness.
scoreboard: dict[Range, State]. The ranges above rcv_nxt, each one Requested, Forgiven or Received.
exemption_eligible: bool. Fixed when the flow is created: True for gradient all-reduce payload on a step with p > 0.

State per flow, at the sender:

cc_mode: Mode. Obeying when the flow starts. After that the mode follows the two flags on the receiver’s latest report.

State per training step, at each receiving rank. No rank reads another’s.

p: float. P_low or P_high, by the detector’s classification of this step.
budget: int. The loss budget in bytes for the whole step. It vests as bytes arrive.
received: int. Distinct gradient bytes this rank has received this step. The data path counts a byte when it first becomes Received.
forgiven: int. Bytes the receiver acknowledged without receiving.
outstanding: int. Distinct trimmed bytes neither received nor forgiven. A trim adds its new bytes; an arrival or a forgiveness takes them off.
open: bool. True until this rank’s all-reduce for the step completes.

At all times, forgiven is at most budget. A step the detector never classified has no loss budget and must go through ordinary loss recovery.

Handlers below are named for where they run. Nothing runs at the switch. Four calls get the four facts above from outside the transport:

InCriticalRegime(gradients). Accordion’s criterion, at the receiving rank, over the gradients that rank holds. The network does not carry this input.
ExpectedGradientBytes(step). What this step will deliver to this rank. A receiving net plugin is already handed the size of every message posted to it, so the plugin can keep the step’s total as a sum.
StepOf(flow). Which step a flow belongs to. Nothing carries a flow’s step today.
IsGradientAllReduce(flow). Whether a flow is gradient traffic. A communicator’s traffic class already becomes an IP type of service on RoCE, so the DSCP carries it.

Everything else is the transport’s own. Mark and Advance write the scoreboard, SendAck and SendRetransmissionRequest are packets the transport already sends, and ReduceRate and Retransmit are what congestion control and loss recovery already do. Draw(P) is true with probability P.

Receiver::OnStepBegin(step, gradients):
  critical = InCriticalRegime(gradients)
  entry = steps[step]
  // A critical step can afford less loss.
  entry.p = critical ? P_low : P_high
  entry.budget = Floor(entry.p * ExpectedGradientBytes(step))
  entry.received = 0
  entry.forgiven = 0
  entry.outstanding = 0
  entry.open = true

Receiver::OnAllReduceComplete(step):
  // The step is over. Any range trimmed later is requested for
  // retransmission.
  steps[step].open = false

A range that reaches Forgiven or Received never leaves that state, so none is charged twice. A range that arrives without ever being trimmed becomes Received directly.

scoreboard value	trimmed header reports the range	data packet arrives
absent	run `OnTrimmedHeader` below	`Received`, ACK
`Requested`	run `OnTrimmedHeader` below	`Received`, ACK
`Forgiven`	ACK, no charge	discard the payload, no credit back
`Received`	duplicate ACK, no charge	`Received`, ACK

Receiver::Unsettled(flow, range):
  // The result holds only the bytes the receiver still lacks.
  return the bytes of range at or above flow.rcv_nxt that are marked
    neither Received nor Forgiven on flow.scoreboard

Receiver::Flags(flow):
  // Both flags are sent with every acknowledgement and retransmission request.
  entry = steps[StepOf(flow)]
  exhausted = flow.exemption_eligible &&
    entry.forgiven + entry.outstanding + kPacketBytes > entry.budget
  return (flow.exemption_eligible, exhausted)

Receiver::Repair(flow, range):
  // Every refusal takes this path. Mark only the unsettled subranges, so
  // a range that is partly Received keeps its Received bytes.
  Mark(flow.scoreboard, Unsettled(flow, range), Requested)
  SendRetransmissionRequest(range, kNormal, Flags(flow))

Receiver::OnTrimmedHeader(flow, range):
  missing = Unsettled(flow, range)
  // Nothing is missing after all, so acknowledge and charge nothing.
  if (missing is empty):
    SendAck(flow.rcv_nxt, Flags(flow))
    return

  // Only gradients are forgivable. Everything else is requested for
  // retransmission.
  if (!IsGradientAllReduce(flow)):
    Repair(flow, range)
    return

  entry = steps[StepOf(flow)]
  // No loss budget is open for this step, so there is nothing to charge.
  if (entry == null || !entry.open):
    Repair(flow, range)
    return

  // A range trimmed again is already outstanding. Count only its new bytes.
  entry.outstanding += Count(bytes of missing not marked Requested)
  len = Count(missing)

  // The vesting rule keeps forgiven bytes within a fraction p of the bytes
  // received or forgiven so far.
  if (entry.forgiven + len >
      entry.p * (entry.received + entry.forgiven + len)):
    Repair(flow, range)
    return

  // A range the draw refuses charges nothing and stays outstanding.
  if (!Draw(P)):
    Repair(flow, range)
    return

  // Forgiving charges the loss budget once.
  entry.forgiven += len
  entry.outstanding -= len
  Mark(flow.scoreboard, missing, Forgiven)
  flow.rcv_nxt = Advance(flow.rcv_nxt, flow.scoreboard)
  // The acknowledgement carries the ECN echo whether or not this flow is
  // exempt, so the signal is still there when this flow obeys congestion
  // control again.
  SendAck(flow.rcv_nxt, ecn_echo, Flags(flow))

Every path either sends the retransmission request the transport already sends, or charges the loss budget once.

Sender::OnFlowStart(flow):
  // Every flow obeys congestion control until the first report arrives,
  // one round trip after the flow starts.
  flow.cc_mode = Obeying

Sender::OnReport(flow, exemption_eligible, budget_exhausted):
  // Every acknowledgement and retransmission request is a report. The
  // latest one wins, so the exemption can end and begin again within a
  // step.
  flow.cc_mode =
    (exemption_eligible && !budget_exhausted) ? Exempt : Obeying

Sender::OnCongestionNotification(flow):
  // An exempt sender ignores ECN marks as well as trims. Most of
  // DCQCN's slowdown comes from marks.
  if (flow.cc_mode == Exempt):
    return
  ReduceRate(flow)

Sender::OnRetransmissionRequest(flow, range):
  // OnReport has already applied this request's flags. A refusal alone
  // does not end the exemption.
  if (flow.cc_mode == Obeying):
    ReduceRate(flow)
  Retransmit(range)

Only OnTrimmedHeader charges the loss budget, neither forgiven nor received ever falls, and a closed loss budget never reopens. The bound therefore holds at every instant, not only when a step ends: forgiven never exceeds $\frac{p}{1 - p} \times \text{received}$ .

Results

I ran the worst fabric with three seeds for each policy variant. All variants drew from one random stream, so the sender-side baseline shed the same messages the receiver-side policy could forgive.

Against a tight baseline, $P_{\text{low}} = P_{\text{high}} = 0.005$ , forgiveness with exemption at $P_{\text{high}} = 0.1$ cut the 20-step training time by 16.1 to 16.7 percent and gave up 7.55 to 7.57 percent of gradient bytes. The median all-reduce completion time on non-critical steps fell from 32.3 ms to 17.4 ms; on critical steps it stayed at the baseline’s. Plain DCQCN with no loss at all stayed within -1.1 to +0.8 percent of that tight baseline over five seeds, so every figure here is measured against DCQCN with no loss tolerance.

With the loss budget available in full at step start, the same policy cut training time by only 9.1 to 10.2 percent, for 7.9 to 8.1 percent of gradient bytes. The first burst of each step exhausted the loss budget. In one seed’s trim-level accounting, switches trimmed seven times more bytes in the first fifth of a step than in the last fifth, 12.3 GB against 1.8 GB. The vesting rule forgave 18 percent of the trimmed bytes that arrived in the first fifth and 60 percent of those in the last. Its bound grows in proportion to the bytes received, so it allows more forgiveness late in the step. Whether late forgiveness costs anything is open.

Those runs forgave every range the vesting rule allowed, $P = 1$ . At $P = 0.25$ , training time still fell 15.8 to 16.9 percent and the loss fell to 5.5 percent. At $P = 0$ the receiver forgives nothing and the exemption stays: training time fell 15.0 to 15.7 percent at zero loss, with 9.5 to 10.5 percent of bytes retransmitted against 5.2 to 5.9 percent under $P = 1$ . The exemption accounted for about 15 of the 16 points. Forgiveness accounted for about one point and halved the retransmission load. $P$ sets how much loss the run accepts and barely changes how much time it saves.

Ignoring congestion signals did not destabilize the fabric. Retransmission timeouts fell by more than half, senders received a third as many congestion notification packets, and tensor-parallel completion times stayed within 4.4 percent of the baseline either way, inside that metric’s seed spread.

Exempt flows sustained a higher sending rate, so the trim rate more than doubled, from about 3 percent to 6.7 to 7.4 percent. The budget-exhausted flag was set or cleared about 0.5 times per exempt flow. Of the exempt flows, 19 percent obeyed congestion control again after their exemption began, for 0.17 ms each on average. The loss-budget invariant held on every step.

The exemption also shortened training on a fabric that rarely trimmed. With eight spines per leaf the same fabric is not oversubscribed, and the tight baseline trimmed 0.02 to 0.04 percent of bytes. Seven senders still converge on every receiving link, so DCQCN cut rates there on every step, on signals the exemption ignores. Training time fell 4.5 to 7.5 percent for 1.0 to 1.3 percent of gradient bytes, and switches trimmed ten times as many bytes from exempt flows as under the baseline.

Forgiveness charges the loss budget only under congestion

Forgiveness and DBLP’s sender-side shedding run the same schedule at the same $P_{\text{low}}$ and $P_{\text{high}}$ . They differ in where the loss budget goes.

Shedding charges it whether or not the network is congested. Forgiveness charges it only for bytes a switch trimmed. On the worst fabric at the same loss budget, sender-side shedding gave up 8.1 percent of its gradient bytes, slightly more than forgiveness, and cut training time by only 2.4 to 3.3 percent.

A loose baseline, $P_{\text{low}} = P_{\text{high}} = 0.1$ , sheds on the critical steps too, which conflicts with the critical learning regime. The loose baseline gave up 10.0 to 10.2 percent of gradient bytes and cut training time by 2.6 to 3.9 percent, no more than shedding under the schedule.

Limits

Nothing here tests whether a current model tolerates losing up to 10 percent of its gradient bytes on non-critical steps. DBLP tested EfficientNet and ResNet; Weintraub tested 10 percent uniform loss on Llama 2 7B without phase dependence. My search turned up no published work on phase-gated gradient loss at Transformer scale, and none measuring loss that is bursty and correlated, which packet trimming produces.

A deployment has to run a real detector. A detector that classifies a critical step as non-critical lets 10 percent of its gradient bytes be forgiven where the schedule allows 0.5 percent. One that classifies a non-critical step as critical only loses the time saving. Nothing here measures either, and the cheapest check is offline: replay a detector over the gradient norms of a real training run and count the steps it misses.

The two host facts no component carries today need no wire change. The decision does: forgiving a range writes the transport’s own reliability state, which on an RDMA fabric lives in the network interface card rather than in a plugin above it. MLT could not do this on an RDMA card and used UDP in user space instead, and FORGIVE asks more of the card than MLT did. Ultra Ethernet is putting trimming, the retransmission request for a trimmed header and selective repeat into hardware, so a card that can do this would most likely be an Ultra Ethernet card.

The congestion control is DCQCN because that is what the simulator models. Meta runs its 400 Gbps ML training networks with DCQCN off, and there the exemption has nothing to act on. Ultra Ethernet’s default is Network Signal-based Congestion Control, which I have not modelled, so nothing here says whether the idea applies to it.

The simulation has one training job. Exempt flows shared the fabric with their own job’s tensor-parallel traffic and one background burst, never with another job obeying congestion control. Such a job would lose bandwidth to exempt flows, and only a run with a second job would show whether it can tolerate that.

The simulation has 64 ranks with tensor parallelism on the network, so eligible gradient traffic is only 24 percent of the bytes. With tensor parallelism kept on NVLink, a fabric carrying only data-parallel and pipeline-parallel traffic would offer four times as much.

Prior work

Four systems give up gradient bytes on purpose, and each picks which bytes in a different way.

MLT has the sender and receiver agree on a tolerated fraction per tensor before training. Once enough of a tensor arrives the receiver stops requesting retransmissions, so the bytes it gives up are whichever arrive last. MLT’s bound is per model and constant over training, it weakens congestion control for every flow for the whole run, and its transport is UDP in user space, which the authors say RDMA network interface cards cannot host.
LTP closes a round early on network conditions.
OptiReduce bounds each round by an adaptive timeout and spreads the resulting loss over the whole gradient with a randomized Hadamard transform.
Trimmable gradients lay out each packet so its trimmed prefix is already a quantized gradient, which removes retransmission entirely and any bound with it: whatever the switch trims is accepted.

That last paper’s future work asks for a congestion control that deliberately over-sends and lets the switch trim the excess. The exemption does that, within the loss budget.

FORGIVE decides per missing range from the switch’s trim report, so the bytes it gives up are the ones a switch trimmed rather than the ones that arrived last. Its loss budget is per receiving rank and training step, not per model. Its exemption is per flow, stays within that loss budget, and follows the receiver’s latest report on the budget.

On a fabric that trims and runs selective repeat, the long loss-recovery tail MLT, LTP and OptiReduce were built to reduce does not exist: those systems ran on TCP and UDP with millisecond timeouts. Forgiveness alone does not reduce the rate cuts there, and the exemption accounts for most of the result. The selective-repeat result, the lightly congested result and the $P = 0$ run showed that.

The tolerance numbers in the literature come from loss that is uniform and independent. Packet trimming produces loss that is bursty, correlated across ranks, and concentrated on the busiest steps. Does a model care about the difference?