Resumable uploads

Deep dive

How it works

In: a byte source of known length, an upload endpoint, a chunk size · Out: one stored object whose bytes and checksum equal the source · State: server: session id, declared length, committed bytes, expiry; client: id and the file (the offset is re-read from the server) · Cost: n bytes plus at most one chunk lost per failure

1. Create: send the total length and metadata; the server returns a session URL or id and an expiry. tus: POST with Upload-Length, answer 201 and Location. GCS: POST with uploadType=resumable, session URI in Location. S3: CreateMultipartUpload, answer an upload ID.
2. Send a chunk with its offset and a checksum of those bytes. tus: PATCH with Content-Type: application/offset+octet-stream and Upload-Offset. GCS: PUT with Content-Range: bytes FIRST-LAST/TOTAL. S3: UploadPart with a part number.
3. The server verifies the chunk checksum; on mismatch it discards the chunk and keeps the offset (tus: 460).
4. The server appends only bytes at or after its committed offset (skipping any stored prefix) and returns the new offset. tus: 204 and a new Upload-Offset; a wrong offset gets 409. GCS: 308 with a Range header; take its upper value, the server may not have received every byte.
5. On any failure (timeout, reset, lost reply), do not guess: query the offset. tus: HEAD, which always returns Upload-Offset. GCS: empty PUT with Content-Range: bytes */TOTAL; 308 without Range means nothing is stored.
6. Resume from the reported offset after a backoff with jitter. If the session is gone (404, 410, other non-retriable 4xx), create a new one.
7. Finalize: the last chunk completes the object (GCS: 200/201), or the client sends the part list (S3 CompleteMultipartUpload). The server checks length and whole-object checksum, then makes the object visible.
8. Abandoned sessions expire and their bytes are deleted.

S3 differs in step 4: parts are independent and may arrive in any order and in parallel, so there is no single offset; the client keeps the part list, can call ListParts, and re-sends only missing parts. The IETF draft generalises tus: any content-carrying method creates the upload, HEAD reads Upload-Offset, PATCH with application/partial-upload appends, DELETE cancels; an interim 104 response announces the upload resource.

Code

use std::collections::HashMap;
const fn table() -> [u32; 256] { // CRC-32 (IEEE, reflected), byte-at-a-time table
    let (mut t, mut i) = ([0u32; 256], 0);
    while i < 256 {
        let (mut c, mut k) = (i as u32, 0);
        while k < 8 { c = if c & 1 != 0 { 0xEDB8_8320 ^ (c >> 1) } else { c >> 1 }; k += 1; }
        t[i] = c;
        i += 1;
    }
    t
}
static T: [u32; 256] = table();
fn crc32(d: &[u8]) -> u32 { !d.iter().fold(!0, |c, &b| T[((c ^ b as u32) & 255) as usize] ^ (c >> 8)) }
#[derive(Debug, PartialEq)]
enum Rej { Crc, Gap(u64), Final, Unknown } // Gap carries the committed offset
struct Session { len: u64, data: Vec<u8> } // data.len() == committed offset
#[derive(Default)] struct Server { sessions: HashMap<u64, Session>, next: u64, done: HashMap<u64, Vec<u8>> }
impl Server {
    fn create(&mut self, len: u64) -> u64 {
        self.next += 1;
        self.sessions.insert(self.next, Session { len, data: Vec::new() });
        self.next
    }
    fn status(&self, id: u64) -> Result<u64, Rej> { // the HEAD / "bytes */total" query
        self.sessions.get(&id).map(|s| s.data.len() as u64).ok_or(Rej::Unknown)
    }
    /// Idempotent by offset: bytes below the committed offset are skipped, never applied twice.
    fn put(&mut self, id: u64, off: u64, body: &[u8], crc: u32) -> Result<u64, Rej> {
        let s = self.sessions.get_mut(&id).ok_or(Rej::Unknown)?;
        if crc32(body) != crc { return Err(Rej::Crc); } // nothing is applied
        let have = s.data.len() as u64;
        if off > have { return Err(Rej::Gap(have)); }
        let skip = ((have - off) as usize).min(body.len()); // already-committed prefix
        if have + (body.len() - skip) as u64 > s.len { return Err(Rej::Gap(have)); }
        s.data.extend_from_slice(&body[skip..]);
        Ok(s.data.len() as u64)
    }
    fn finish(&mut self, id: u64, whole: u32) -> Result<(), Rej> {
        let s = self.sessions.get(&id).ok_or(Rej::Unknown)?;
        if s.data.len() as u64 != s.len || crc32(&s.data) != whole { return Err(Rej::Final); }
        let s = self.sessions.remove(&id).unwrap();
        self.done.insert(id, s.data);
        Ok(())
    }
}
/// Link: breaks after a random number of bytes (mean `mean`); 10 % of ACKs are lost; 5 % of bodies get a bit flip.
struct Link { rng: u64, left: u64, mean: u64, sent: u64 }
impl Link {
    fn rand(&mut self) -> u64 { // LCG (Knuth MMIX constants), top 31 bits
        self.rng = self.rng.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
        self.rng >> 33
    }
    fn arm(&mut self) { // exponential distance to the next break
        self.left = 1 + (-((self.rand() as f64 + 1.0) / 2147483649.0).ln() * self.mean as f64) as u64;
    }
    /// Err = request never completed; Ok(acked, body as received)
    fn carry(&mut self, body: &[u8]) -> Result<(bool, Vec<u8>), ()> {
        let n = body.len() as u64;
        if n > self.left { self.sent += self.left; self.arm(); return Err(()); } // cut mid-body: server gets nothing usable
        self.sent += n; self.left -= n;
        let mut b = body.to_vec();
        if self.rand() % 100 < 5 { let i = self.rand() as usize % b.len(); b[i] ^= 1; }
        Ok((self.rand() % 100 >= 10, b))
    }
}
fn upload(srv: &mut Server, link: &mut Link, data: &[u8], chunk: usize) -> u32 {
    let (id, mut off, mut tries) = (srv.create(data.len() as u64), 0usize, 0u32);
    while off < data.len() {
        let end = (off + chunk).min(data.len());
        let crc = crc32(&data[off..end]); // computed on the client copy, before the link
        tries += 1;
        match link.carry(&data[off..end]) {
            Ok((acked, wire)) => match srv.put(id, off as u64, &wire, crc) {
                Ok(c) if acked => off = c as usize,
                _ => off = srv.status(id).unwrap() as usize, // ACK lost or CRC rejected: ask the server
            },
            Err(()) => off = srv.status(id).unwrap() as usize, // status is a tiny request; assume it gets through
        }
    }
    srv.finish(id, crc32(data)).expect("finalize");
    tries
}
fn main() {
    let data: Vec<u8> = (0..512 * 1024u32).map(|i| (i.wrapping_mul(2654435761) >> 13) as u8).collect();
    let (mut srv, mut link) = (Server::default(), Link { rng: 42, left: 0, mean: 128 * 1024, sent: 0 });
    link.arm();
    let tries = upload(&mut srv, &mut link, &data, 16 * 1024);
    assert_eq!(srv.done.values().next().unwrap(), &data); // stored object == input
    let x = |s: u64| s as f64 / data.len() as f64;
    println!("resumable: {tries} requests, {} bytes on the wire ({:.2}x)", link.sent, x(link.sent));
    // duplicate chunk at an old offset is ignored
    let id = srv.create(8);
    let c = b"abcd";
    assert_eq!(srv.put(id, 0, c, crc32(c)), Ok(4));
    assert_eq!(srv.put(id, 0, c, crc32(c)), Ok(4)); // replay at an old offset: no change
    assert_eq!(srv.put(id, 4, b"efgh", crc32(b"efgh")), Ok(8));
    assert_eq!(srv.put(id, 6, b"ghXY", crc32(b"ghXY")), Err(Rej::Gap(8))); // would overflow the declared length
    assert_eq!(srv.put(id, 0, c, 0), Err(Rej::Crc));
    assert_eq!(srv.finish(id, crc32(b"abcdefgh")), Ok(()));
    // baseline: restart from zero on every break (one request carries the whole file)
    let mut base = Link { rng: 42, left: 0, mean: 128 * 1024, sent: 0 };
    base.arm();
    let mut attempts = 0;
    while base.carry(&data).map_or(true, |(_, w)| w != data) { attempts += 1; }
    println!("restart-from-zero: {} attempts, {} bytes on the wire ({:.1}x)", attempts + 1, base.sent, x(base.sent));
    assert_eq!(crc32(b"123456789"), 0xCBF43926); // standard check value
    let (big, t) = (vec![7u8; 64 << 20], std::time::Instant::now());
    let c = std::hint::black_box(crc32(&big));
    println!("crc32 table: {:.0} MB/s (crc {c:08x})", 64.0 * 1.048576 / t.elapsed().as_secs_f64());
}

Complexity and constants

• Bytes sent: n plus at most one chunk per failure. Round trips: n/c plus one offset query per failure.
• Chunk size c trades retry cost (a break loses at most c bytes), overhead and memory (c per upload, c x parallelism for parts). Stop-and-wait efficiency is c / (c + bandwidth x RTT): 8 MiB at 5 MB/s and 100 ms RTT gives about 94 % (arithmetic).
• Limits: S3 at least 5 MiB (last part excepted), GCS multiples of 256 KiB, 8 MiB or more recommended. S3's 10,000-part cap forces parts of at least about 105 MiB for a 1 TiB object.

Efficiency

• Cost: a request costs one round trip plus c/bandwidth, plus checksum CPU on both ends. Measured on this Mac (Apple M4 Pro, rustc -O): the table CRC32 above ran 510-530 MB/s over 64 MiB (three runs), about 2 ns/byte. Simulation above (512 KiB, 16 KiB chunks, breaks every 128 KiB on average, 10 % lost replies, 5 % corrupted bodies): resumable sent 590,451 bytes in 39 requests (1.13x); restart-from-zero sent 9,632,179 bytes in 62 attempts (18.4x).
• Levers:
  • Adapt chunk size to failures — fewer round trips on clean links, less loss on bad ones (arithmetic above) — mobile networks.
  • Parallel parts — link kept full (S3 multipart) — big files, high latency.
  • Hardware CRC32C or CRC-64/NVME — checking at memory speed (CRC32C) — large uploads.
  • Trailing checksum in the request — one pass, no pre-scan of the file (S3 aws-chunked) — streaming sources.
• Naive → standard → state of the art:
  • One PUT, restart on failure · expected waste grows exponentially with size over mean time between breaks.
  • Sequential offset chunks with an offset query · at most one chunk lost per failure.
  • Parallel parts, per-part plus whole-object checksums, OS background transfer · loss bounded by a part, link kept full, survives app suspension.

Implementation notes

• Make put idempotent by offset: skip bytes below the committed offset, so a retry after a lost reply cannot duplicate data. The server's offset is the truth; persist the session id on the client.
• Checksum per chunk (retry one chunk) and per object (assembly bugs). S3 offers full-object checksums for CRC-64/NVME, CRC-32 and CRC-32C, and composite (checksum of part checksums) for the others.
• Expire sessions: tus Upload-Expires; GCS one week; S3 needs an AbortIncompleteMultipartUpload lifecycle rule (example DaysAfterInitiation: 7) to delete stranded parts.
• Mobile: an iOS background URLSession hands the transfer to the system so it outlives the app; uploads must come from a file, so write per-chunk files. After a network change re-query the offset; prefer Wi-Fi or charging when the user can wait.
• Security: an upload URL is a bearer token. S3 presigned URLs are documented bearer tokens, valid up to 7 days (SDK/CLI) or the credential lifetime if shorter; expiry is checked per request, so a resume after expiry fails: issue a fresh URL per part or resume (signed URLs). Cap size by declared length (tus 413), detect content type from the stored bytes, not the header, and process only after finalization.

Pitfalls

• Treating a lost reply as a lost chunk: the server may have applied it. Query the offset before resending.
• Parts below the S3 minimum, or non-consecutive part numbers with composite checksums, fail completion; unfinished uploads keep their parts stored.
• Retrying without backoff, or serving the upload before validation.

Variants and alternatives

In production

• Photo and video apps upload originals to object storage, then transcode (video ladder); backup clients, asset pipelines, telemetry agents and imaging archives use S3 multipart, GCS sessions or tus.

Implementations

Sources

• tus, Resumable upload protocol 1.0.0. spec
• IETF HTTP WG, Resumable Uploads for HTTP, draft-ietf-httpbis-resumable-upload-14 (Internet-Draft, active). datatracker
• AWS, Amazon S3 multipart upload limits. docs
• AWS, Checking object integrity for data uploads in Amazon S3. docs
• AWS, Configuring a bucket lifecycle configuration to delete incomplete multipart uploads. docs
• AWS, Download and upload objects with presigned URLs. docs
• Google Cloud, Performing resumable uploads. docs
• Apple, URLSessionConfiguration.background(withIdentifier:). docs