Design file uploads and storage

System design Mid-level Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Pick the fix

Users upload 2 GB videos through the API servers, which buffer each file, run short of memory and time out. Best fix?

Options
Hint 1 of 1

Do the file bytes need to pass through your servers at all?

Choose an answer, then open Show answer below to check it.

Boxes for client, upload API, Postgres metadata, object storage, scan queue and worker; follow one upload to ready.
Text version

Boxes on the board, in order:

  1. Client
  2. Upload API
  3. Postgres
  4. Object storage
  5. Scan queue
  6. Worker
Show answer

Answer: A Return pre-signed multipart URLs so clients upload straight to object storage

Correct. The bytes skip your servers, and a failed part is retried on its own.

Large uploads shouldn't flow through the app tier. A pre-signed URL lets the client send bytes straight to object storage under a narrow, expiring permission, and multipart means only failed parts are resent.

Why not the others?
  • B Raise the request timeout to 30 minutes. Servers still hold the bytes, and one dropped connection restarts the whole upload.
  • C Add more API servers. Spreads the memory problem without removing it, at a higher cost.
  • D Compress the files on the client first. Video is already compressed, and the bytes still pass through the API.

30-second answer

Keep file bytes out of your API servers. The API authorises the user and returns a pre-signed URL; the client uploads directly to object storage, in parts for large files so failed chunks can be retried. An event on upload completion triggers processing (virus scan, thumbnails) via a queue. Metadata (owner, name, size, checksum, status) lives in Postgres. Downloads go through short-lived signed URLs, often via a CDN.

Practise saying it · 35 s. No recording, just a timer

Design file storage.

Say your answer out loud, then check which of these points you covered:

  • Pre-signed direct upload
  • Multipart, resumable
  • Metadata in Postgres
  • Async scan + thumbnails
  • Signed downloads via CDN

Deep dive

Multipart uploads allow resume and parallel parts.

Verify the checksum after upload.

Mark files 'pending' until scanning finishes; don't serve them before.

Lifecycle rules move old files to cheaper storage classes.

Common mistakes

  • Proxying gigabytes through the API servers.
  • Public buckets for private files.
  • Serving files before the malware scan.

Senior take & follow-ups

Direct upload via pre-signed URLs, multipart, async processing, metadata, signed downloads.

Follow-up questions

Think of your answer first, then open the hint.

  1. Why pre-signed URLs?

    Show hint

    Clients talk to storage directly with narrowly scoped, expiring permission.

  2. How do you resume uploads?

    Show hint

    Multipart: retry only failed parts.

Related questions

Suggest a fix if something here is wrong or out of date.