Video upload and playback — a full worked answer

Assumptions: global audience, mobile-first for both upload and playback. Videos average a few minutes. Reads far outnumber writes: one upload is watched many times, and the design rests on that ratio. Storage is effectively unbounded and object storage is available.

First question, and it changes the caching design: is content public, or is it access-controlled per viewer? Public content can be cached at the edge and served to everyone identically. Per-viewer authorisation means either signed URLs with short expiries or authorisation at the edge, and getting that wrong means one person's private video cached and served to a stranger. I will assume public with some private videos, handled by signed URLs.

Second question: on-demand only, or live streaming too? These are different systems: live has no processing slack, needs low-latency segment production, and cannot pre-transcode. I will assume on-demand, and say I am consciously excluding live because it would change the pipeline fundamentally.

Scope: upload from an unreliable network, transcode to several qualities, and adaptive playback. Out of scope: recommendations, comments, monetisation, and copyright matching, though I would note the last one is a production requirement that sits in the processing pipeline.

Actors: the uploader, the viewer, and the system, which decides which resolutions to produce, in what order, when a video is watchable, what to do with a file it cannot decode, and when to give up retrying.

The chain: upload started → bytes stored → queued for processing → transcoded into renditions → published and playable → viewed → possibly taken down.

Failure branches. At upload: the connection drops at 80%. With a single request that is 400 MB thrown away and an angry user, so uploads must be chunked and resumable. Given the mobile assumption, that is a requirement, not an optimisation. At upload: the same chunk is sent twice after a retry. Chunks are addressed by index, so a repeat overwrites the same data and the upload is idempotent. At stored: the processing message is lost, so the video sits forever showing "processing" and nobody is alerted. This needs the event written durably alongside the upload record, and a sweeper that finds videos stuck in a state too long. The sweeper is the branch candidates tend to forget, and a missing one shows up as support tickets.

At transcoding: the worker crashes halfway through. The job retries, and because output is written per rendition to a deterministic path, a retry redoes work rather than corrupting anything. At transcoding: the file is broken or in an unsupported format. This is not retryable, so it must be told apart from a transient failure and moved to a failed state with a clear message to the uploader, rather than retried forever.

At published: some renditions succeeded and others did not. Publish as soon as one usable rendition exists and add the others as they finish, so the video becomes watchable sooner. At viewed: a segment is missing from the origin because of a partial publish, so the player stalls. Make publishing atomic by writing the manifest last, since the manifest is what makes segments discoverable.

At taken down: the video is cached at hundreds of edge locations, so removal is a purge with a real propagation delay, which matters for legal takedowns and is worth naming.

Ingest: assume 500 hours uploaded per minute. At roughly 1 GB per hour for source quality, that is 500 GB a minute, about 8 GB per second arriving. Meaningful, but it goes straight into object storage and never through my application servers, so it is not a compute problem.

Storage: 500 hours a minute is 720,000 hours a day, about 720 TB a day of source. Transcoding into five renditions roughly doubles to triples the total, so call it 2 PB a day. That is a lot, but it is only money: object storage is cheap per byte, and old, rarely-watched content moves to colder tiers.

Now egress, which is the number that changes my mind. Assume 100 million viewing hours a day, which is about 4 million concurrent viewers on average. At 3 Mbps each that is about 12.5 Tbit/s, or roughly 1,500 GB per second sustained: two orders of magnitude more than ingest, and continuous. If that traffic came from my origin servers, bandwidth would be by far the largest line item in the system, larger than storage or compute.

Transcoding compute: 500 hours a minute, and transcoding roughly real-time per rendition per core, so five renditions is 2,500 core-hours of work arriving every minute. That is about 150,000 cores busy around the clock. Significant, and elastic because of the processing slack.

So the hard part here is egress bandwidth, and the answer is that almost none of it may come from my origin. Storage, ingest and even the transcode fleet are smaller problems. This estimate shows that the CDN is not an optimisation in this design; it is the core of the read path.

Upload path: the client asks the API for a presigned upload URL, then uploads chunks directly to object storage. The bytes never pass through my application servers, justified by the 8 GB per second ingest figure: routing that through my services would mean building a bandwidth tier for no benefit. The API only records metadata and, when the upload completes, writes an upload record and a processing event in the same transaction so the event cannot be lost.

Processing: workers pull from a queue and transcode into renditions (say 240p through 1080p), chopping each into segments of a few seconds and writing an adaptive streaming manifest. Justified by the slack analysis: hours available, so this is asynchronous, retryable and can run on interruptible capacity at a fraction of the cost. Workers are idempotent by writing to deterministic paths per rendition.

Why segments, which is worth explaining properly: chopping video into short chunks turns playback into a series of ordinary static file requests. Each one is cacheable at the edge, resumable, and chosen independently, which is what makes both CDN delivery and adaptive quality possible. The whole read path rests on this decision.

Playback: the player fetches the manifest, then requests segments from the CDN. It starts at a low bitrate so playback begins fast, measures how quickly segments arrive, and steps up or down per segment. Justified by the two-second start budget and the fact that a mobile connection changes constantly.

Delivery: segments are static immutable files with long cache lifetimes and content-addressed paths, so the CDN hit rate is extremely high and purging is almost never needed. Justified by the egress number: the CDN is what makes this system affordable, and the hit rate largely decides what the bandwidth bill is.

Metadata: a normal database holding video records, owner, status, and rendition list. Small, ordinary, and separate from the media path. It is worth saying that the hard scale is in the bytes, not in the metadata.

Access control for private videos: signed URLs with short expiries, issued by the API after an authorisation check. The CDN validates the signature, so the origin is not consulted per segment. Cost: the player must refresh signatures during long videos.

Hard part one: keeping origin egress near zero, which means the cache hit rate is the number that matters. Popular videos are easy to cache: a viral video is watched by millions and its segments are hot everywhere, so the hit rate approaches 100%. The problem is the long tail: a huge number of videos watched a handful of times each. Those get evicted between views, so they miss, and while they are a small share of views they can be a large share of origin traffic. Mitigations: tiered caching, where edge misses go to a regional cache rather than straight to origin, which collapses many edge misses into one origin fetch; and keeping only the first few segments of unpopular videos warm, since many views are abandoned in the first seconds and never reach later segments. Cost: another caching layer to run, and more storage held at regional tier.

The first segments are worth special treatment in general: everyone who plays a video fetches segment one, and far fewer reach segment fifty. Prioritise transcoding and warming the opening segments so playback starts fast even for cold content. Cost: a more complex pipeline that treats parts of one video differently.

Regional popularity: a video going viral in one country makes that region's edges hot while others stay cold. The CDN pulling on demand handles this, but the origin sees a burst as many edges fetch the same segments at once. Request coalescing at the regional tier turns thousands of simultaneous edge misses into one origin read. Cost: a small added latency on the very first fetch.

Hard part two: uploads from unreliable networks. Chunked and resumable, with chunks of a few megabytes so a failure loses seconds of progress rather than everything. The client asks which chunks the server already has and sends only the missing ones, so resuming after an app restart, or a day later, works. Chunks are addressed by index, so a duplicate is an overwrite and retries are safe with no extra machinery. Cost: more complex client code, server state tracking partial uploads, and a cleanup job for abandoned ones. Without that job, partial uploads accumulate silently and cost real money.

Processing cost control: transcode lazily for rare resolutions. Produce the common renditions immediately, and generate 4K only when someone asks for it. Most videos are watched a few times at 720p or below, and 4K is by far the most expensive rendition to produce, so this avoids a large share of transcode cost. Cost: the first viewer wanting a rare rendition waits, or gets a lower quality. That is acceptable for the saving.

The stuck-video problem, revisited because it is a common production failure: a video whose processing event was lost or whose job failed silently sits in "processing" forever. A sweeper looking for videos in a non-terminal state past a threshold, which retries or fails them explicitly, is the fix. Cost: a background job to run and monitor. Without it, the failure is invisible to you and highly visible to the user.

Consistency per feature: metadata is strongly consistent, since a user renaming their video should see it immediately. The published state is eventually consistent across regions and a few seconds of lag is fine. Segments are immutable, so they cannot be stale: a benefit of content addressing that is easy to overlook. View counts are approximate and batched.

What I would monitor: playback start time at p50 and p95, rebuffer rate, CDN hit ratio broken down by popularity tier, transcode queue age, and the count of videos stuck in processing.