Exponential Backoff With Jitter for Retrying Fallible Async Operations in Rust
use std::time::Duration;
use rand::Rng;
#[derive(Clone, Debug)]
pub struct BackoffPolicy {
pub base_delay: Duration,
pub max_delay: Duration,
pub factor: f64,
pub max_retries: u32,
}
impl Default for BackoffPolicy {
fn default() -> Self {
BackoffPolicy {
base_delay: Duration::from_millis(100),
max_delay: Duration::from_secs(10),
factor: 2.0,
max_retries: 5,
}
}
}
impl BackoffPolicy {
pub fn delay_for(&self, attempt: u32) -> Duration {
let base = self.base_delay.as_secs_f64();
let capped = self.max_delay.as_secs_f64();
let raw = base * self.factor.powi(attempt as i32);
let bounded = raw.min(capped);
// full jitter: pick uniformly in [0, bounded]
let jittered = rand::thread_rng().gen_range(0.0..=bounded);
Duration::from_secs_f64(jittered)
}
}
#[derive(Debug)]
pub enum RetryError<E> {
Exhausted { last_error: E, attempts: u32 },
NonRetryable(E),
}
use std::future::Future;
use crate::backoff::{BackoffPolicy, RetryError};
enum Outcome<T, E> {
Done(T),
Retry(E),
}
pub async fn retry<T, E, F, Fut, C>(
policy: &BackoffPolicy,
mut operation: F,
should_retry: C,
) -> Result<T, RetryError<E>>
where
F: FnMut() -> Fut,
Fut: Future<Output = Result<T, E>>,
C: Fn(&E) -> bool,
{
let mut attempt: u32 = 0;
loop {
let outcome = match operation().await {
Ok(value) => Outcome::Done(value),
Err(err) => {
if !should_retry(&err) {
return Err(RetryError::NonRetryable(err));
}
Outcome::Retry(err)
}
};
match outcome {
Outcome::Done(value) => return Ok(value),
Outcome::Retry(err) => {
if attempt >= policy.max_retries {
return Err(RetryError::Exhausted { last_error: err, attempts: attempt });
}
tokio::time::sleep(policy.delay_for(attempt)).await;
attempt += 1;
}
}
}
}
use serde::Deserialize;
use crate::backoff::{BackoffPolicy, RetryError};
use crate::retry::retry;
#[derive(Debug, Deserialize)]
pub struct User {
pub id: u64,
pub name: String,
}
fn is_transient(err: &reqwest::Error) -> bool {
if err.is_timeout() || err.is_connect() {
return true;
}
matches!(err.status().map(|s| s.as_u16()), Some(429) | Some(502) | Some(503) | Some(504))
}
pub async fn fetch_user(
client: &reqwest::Client,
base_url: &str,
id: u64,
) -> Result<User, RetryError<reqwest::Error>> {
let policy = BackoffPolicy::default();
let url = format!("{}/users/{}", base_url, id);
let user = retry(
&policy,
|| {
let url = url.clone();
let client = client.clone();
async move {
let resp = client.get(&url).send().await?.error_for_status()?;
resp.json::<User>().await
}
},
is_transient,
)
.await?;
Ok(user)
}
A retry loop with exponential backoff is a standard resilience pattern for operations that fail transiently — network blips, rate limits, or a service that is briefly unavailable. Rather than hammering a struggling dependency, each attempt waits progressively longer, giving the remote side room to recover while still eventually surfacing a hard failure to the caller. This snippet builds the pattern as a small, reusable primitive in three files: a policy type that owns the timing math, a generic retry driver, and a real caller that wires it to an HTTP request.
In backoff.rs, BackoffPolicy holds the tunables: base_delay, max_delay, a factor, and max_retries. The delay_for method computes base_delay * factor^attempt, clamps it to max_delay, and then applies full jitter — a random value between zero and the computed delay. Jitter matters because without it, many clients that failed at the same instant retry in lockstep, producing synchronized thundering-herd spikes that keep the dependency down. The f64 arithmetic guards against overflow by saturating at max_delay before converting back to a Duration. The RetryError enum distinguishes an operation that Exhausted its retries from one that failed with a NonRetryable error, so the caller can react differently.
In retry.rs, the retry function is generic over an async operation and a classifier. The operation is a closure returning a future, called fresh on every attempt — futures are single-use in Rust, so the loop cannot simply poll the same future twice; it must reconstruct it. The should_retry predicate lets the caller decide which errors are worth retrying: a 429 or 503 is transient, but a 400 or an auth failure is not, and retrying those only wastes time. On a retryable failure the loop sleeps for policy.delay_for(attempt) via tokio::time::sleep, then increments the attempt counter. Once attempt exceeds max_retries, it returns RetryError::Exhausted carrying the last error, so no context is lost.
The Attempt bookkeeping keeps the control flow explicit: Outcome::Retry re-enters the loop, Outcome::Done returns immediately. Structuring it this way keeps the timing, the classification, and the invocation cleanly separated, which makes the policy trivially unit-testable without any real I/O.
In client.rs, fetch_user shows the primitive in practice against a reqwest client. The closure it passes rebuilds and sends the request each attempt, and the classifier inspects the error to decide retryability — timeouts and connect errors are retried, while a deserialization error is not. The ? operator flows the final RetryError up to the caller. A subtle pitfall worth noting: the closure must be FnMut and must not move non-Copy state out of itself, since it may run several times; here the url is cloned per attempt. Reaching for this pattern is appropriate whenever a call crosses a network or process boundary and the failure mode is plausibly temporary; it is the wrong tool for deterministic errors like bad input, where a retry can never succeed.
Share this code
Here's the card — post it anywhere.
