Resumable Chunked Uploads for Large Files

A single-request upload of a 2 GB video on a train fails at 93% when the connection drops, and starts again from zero — the user tries twice more and gives up; the same file sent in 8 MB chunks loses at most one chunk to each drop and resumes from where it stopped, even after the tab is closed and reopened.

The file upload fields lifecycle treats an upload as one uploading state. For large files, that state needs internal structure: an upload session on the server, an offset the client and server agree on, and a loop that sends one chunk at a time, retrying individual chunks and asking the server where to resume after any interruption.

Context and prerequisites

A resumable upload protocol has three operations, whatever the vendor:

1. Create a session. The client tells the server the file’s size (and name, type, checksum if used); the server returns an upload URL or id.
2. Send a chunk at an offset. The client sends bytes [offset, offset + chunkSize); the server appends them and returns the new offset.
3. Query the offset. After any failure, the client asks how many bytes the server has, and continues from there — never from its own belief.

The open tus protocol standardises exactly this over HTTP (POST to create, PATCH with Upload-Offset, HEAD to query). S3 multipart uploads use a different shape — independently uploaded numbered parts, then a complete call — with the same effect. The client code below uses the tus-style shape because it maps directly onto the three operations.

The core pattern: an offset-driven chunk loop

export interface ResumableOptions {
  createUrl: string;
  file: File;
  chunkBytes?: number;                 // 5–10 MB is a good default
  signal?: AbortSignal;
  onProgress?: (sent: number, total: number) => void;
  loadSession?: () => Promise<string | null>;   // e.g. from IndexedDB, keyed by file fingerprint
  saveSession?: (uploadUrl: string) => Promise<void>;
}

const sleep = (ms: number, signal?: AbortSignal) => new Promise<void>((res, rej) => {
  const t = setTimeout(res, ms);
  signal?.addEventListener("abort", () => { clearTimeout(t); rej(signal.reason); }, { once: true });
});

export async function resumableUpload(o: ResumableOptions): Promise<string> {
  const chunk = o.chunkBytes ?? 8 * 1024 * 1024;
  const total = o.file.size;

  // 1. Reuse a saved session (reload/resume) or create a new one.
  let uploadUrl = (await o.loadSession?.()) ?? null;
  if (!uploadUrl) {
    const res = await fetch(o.createUrl, {
      method: "POST", signal: o.signal,
      headers: { "Upload-Length": String(total), "Upload-Metadata": `filename ${btoa(encodeURIComponent(o.file.name))}` },
    });
    if (!res.ok) throw new Error(`create failed: ${res.status}`);
    uploadUrl = new URL(res.headers.get("Location")!, o.createUrl).toString();
    await o.saveSession?.(uploadUrl);
  }

  // 2. Always start from the SERVER's offset, never from local belief.
  const serverOffset = async () => {
    const r = await fetch(uploadUrl!, { method: "HEAD", signal: o.signal });
    if (r.status === 404 || r.status === 410) throw new Error("session expired");
    return Number(r.headers.get("Upload-Offset") ?? 0);
  };

  let offset = await serverOffset();
  let failures = 0;
  while (offset < total) {
    try {
      // File.slice creates a view; no bytes are read until the request sends them.
      const body = o.file.slice(offset, Math.min(offset + chunk, total));
      const r = await fetch(uploadUrl, {
        method: "PATCH", body, signal: o.signal,
        headers: { "Upload-Offset": String(offset), "Content-Type": "application/offset+octet-stream" },
      });
      if (r.status === 409) { offset = await serverOffset(); continue; }  // offset mismatch: resync
      if (!r.ok) throw new Error(`chunk failed: ${r.status}`);
      offset = Number(r.headers.get("Upload-Offset"));
      failures = 0;
      o.onProgress?.(offset, total);
    } catch (err) {
      if (o.signal?.aborted) throw err;
      if (++failures > 6) throw err;                       // give up; the item goes to "failed"
      await sleep(Math.min(30_000, 500 * 2 ** failures) * (0.5 + Math.random()), o.signal);
      offset = await serverOffset();                        // resync after any failure
    }
  }
  return uploadUrl;   // the server identifies the finished file by this URL/id
}

Progress here is chunk-granular — it advances every 8 MB. For a smoother bar within each chunk, send each chunk with the XHR helper from upload progress and cancellation and add its loaded to the committed offset.

Step-by-step walkthrough

1. Decide the threshold. Chunk only files that are slow to upload: above roughly 20–50 MB, or anything on mobile. Small files are faster as one request.
2. Create a session with the total size. The server allocates storage and returns an upload URL; persist it with a fingerprint of the file (name, size, lastModified) so a reload can find it.
3. Always resume from the server’s offset. After any error, HEAD the session and continue from what the server committed. Local counters can be wrong after a partial chunk.
4. Retry chunks with backoff and jitter. A failed chunk retries alone; several consecutive failures move the item to failed with a Retry button that resumes rather than restarts — the same retry policy as retrying failed submissions with backoff.
5. Resume after reload. Store the File in IndexedDB alongside the session URL, following storing large drafts in IndexedDB; on restore, call the same function and it continues from the server’s offset.
6. Handle expiry. Servers expire incomplete sessions. A 404 or 410 on HEAD means start over with a new session — tell the user the upload restarted.

Failure modes and edge cases

1. Trusting the local offset

If a chunk’s response is lost after the server committed it, the client believes the offset is lower than it is and re-sends — and the server rejects with 409 or, worse, appends twice. Always resync with HEAD after an error, and let the server reject mismatched offsets.

2. The file changed on disk

A user may edit and re-save a file between sessions. Store size and lastModified with the session, and if they differ on resume, discard the session and start again; optionally compute a hash of the first and last chunks to be sure.

3. Chunk size and proxies

Some proxies and serverless platforms cap request bodies (often a few MB to tens of MB). Choose a chunk size below the smallest limit on the path, and make it configurable per environment.

4. Parallel chunks

S3-style multipart allows uploading parts in parallel, which improves throughput on fast links. tus-style offset protocols are sequential by design (there is a concatenation extension for parallel partial uploads). Parallelism complicates resume bookkeeping; add it only when measurements show single-stream throughput is the bottleneck.

5. Completing the form before the upload finishes

A 2 GB upload may outlast the rest of the form. Let the user submit; the form sends a reference to the upload session and the server links the file when the upload completes — or, simpler, keep the submit gated and show clear remaining-time progress. Either way, never submit a reference to an upload the server has not finished.

Verification checklist

• Killing the network mid-upload and restoring it resumes from the last committed chunk.
• Reloading the page during an upload resumes after restore without re-sending committed bytes.
• After any error the client resyncs its offset from the server before continuing.
• A 409 offset mismatch is resolved by resync, not by failure.
• An expired session starts a new upload and tells the user.
• A modified file is not resumed onto its old session.
• Chunk size is below every body-size limit between client and storage.
• Cancel aborts the in-flight chunk and stops the loop.

Frequently Asked Questions

Related

• File Upload Fields and Binary State
• Queueing Form Submissions While Offline
• Upload Progress and Cancellation With XHR and Fetch

← File Upload Fields and Binary State