File sync across devices — a full worked answer

Assumptions: desktop and mobile clients, folders can be shared between users, files from a few kilobytes to several gigabytes, and the client works fully offline with sync resuming on reconnect.

The question that changes a box: do we need to merge concurrent edits to the same file, or is preserving both acceptable? Real merging means understanding file formats. You can merge text; you cannot merge a video. So a general file sync product cannot merge. I will assume we detect the conflict and keep both versions, naming one "Alice's conflicted copy". It sounds like a cop-out, but it is the correct engineering answer, because the alternative is silently discarding somebody's work.

Second question: is deduplication across users acceptable? Storing one copy of a file that ten thousand users all have saves a lot of storage, but it leaks information: an attacker can learn whether a file already exists somewhere by watching whether its upload completes instantly. I will assume deduplication within a user's own account only, which is where most of the benefit is without the disclosure.

Scope I propose: syncing a folder across a user's devices, sharing with other users, offline edits with conflict detection, resumable upload and download of large files, and version history. Out of scope: real-time collaborative editing inside a document, which is a different system built on operational transforms or CRDTs rather than on file sync.

Actors: the user, each of their devices as a separate actor with its own local state, other users sharing the folder, and the system itself, which decides what counts as a change, which version is current, when two versions are in conflict, and what history to retain.

A file's life: created or modified locally → detected by the client watcher → hashed and split into chunks → chunks not already on the server are uploaded → a new version is committed on the server → other devices are notified → they download the missing chunks → they apply the change locally. Plus deletion, which is a version like any other, and restore from history.

Failure branches. There are many, because this system runs on devices and networks it does not control. At detect: the file is still being written (someone is saving a 2 GB video), so uploading now captures a corrupt half-file. The client has to wait for the file to settle rather than react to the first event. At upload: the connection drops at 90% of 5 GB, which must resume from the last completed chunk, not restart. At commit: two devices commit versions derived from the same parent, which is the conflict case and the heart of the product. At notify: a device is offline, so it must find out on reconnect by asking what changed since the last version it saw, rather than depending on a push it missed.

The branches the system owns. A device offline for a month reconnects: sending it every individual change since then could be a hundred thousand events, so it needs a compacted answer: the current state of everything that changed, not the full history of how it got there. A user drops in a folder of 50,000 small files: that must not become 50,000 separate round trips, so the client batches. And the local disk fills mid-sync, which has to fail gracefully and resumably rather than leaving a half-written file that looks complete.

Assume 50 million users, 50 GB each on average, so 2.5 exabytes of raw data. It goes straight to object storage. The one decision it forces is a retention and tiering policy, because keeping every version of everything on hot storage is not affordable.

Chunking is where the arithmetic becomes interesting. Split files into fixed 4 MB chunks and store each by content hash. A user edits one paragraph in a 2 GB presentation: without chunking that is a 2 GB upload, with chunking it is one 4 MB chunk, because every other chunk hashes the same and is already on the server. That is a 500-fold reduction for a common case, and an interrupted upload can resume at chunk granularity. No other decision on the board does as much for the product.

Metadata is the part people underestimate. 50 million users times, say, 20,000 files each is a trillion file records. That does not fit on one machine and it is not object storage's job. It needs a partitioned database, and the partition key has to be the user or the account, so that "list my folder" is one partition and never a scatter across the cluster.

Change notifications: if 10 million devices are online and each sees a few changes an hour, that is roughly 10,000 notifications a second, which is small. The cost is holding 10 million open connections, not the messages. It is the same shape as chat.

So the hard part here is conflict handling and metadata at a trillion rows, not bandwidth. Chunking and object storage make the bytes a solved problem. The design is about knowing which version is current for every file on every device, and detecting when two devices disagree.

The client is a full part of this system, not a thin caller. It holds a local database of every file, its version, its chunk hashes, and its sync state. That is what makes offline work possible, and it is why the client can answer "what changed" without asking anybody. A watcher observes the filesystem, waits for files to settle, hashes them, and compares against its local record.

The metadata service owns the truth about versions, partitioned by account. Each file has a monotonically increasing version, and each account has a cursor: a single number that advances with every change in that account. A device syncs by saying "I am at cursor 8,412, what has changed since?" and getting back a compact list. The cursor is the most useful element in the design. Reconnecting after a month costs about the same as reconnecting after a minute, because the server answers with current state rather than replayed history.

Chunk storage is object storage, keyed by content hash. The client asks "which of these 50 hashes do you already have", uploads only the missing ones over presigned URLs directly to storage, and then commits a new file version referencing the full list. The bytes never touch my servers. At exabyte scale that is a requirement, not an optimisation.

Commit is where correctness lives. A new version is accepted only if its parent version matches what the server currently has: a conditional update, like the seat reservation. If the parent does not match, another device got there first, and this is a conflict rather than an overwrite. That one check is what prevents silent loss, and it takes about two lines of code.

Notification is a persistent connection per device, carrying only "the cursor moved" rather than the change itself. The device then pulls the delta over normal HTTP. That keeps the socket layer trivial and makes a missed notification harmless, since the next connection reveals the gap anyway.

Sharing adds a folder-level access record, and a change in a shared folder advances the cursor for every member. Permission checks happen at the metadata service on every access, not at the edge, because a chunk hash is effectively a capability and must never be servable without an ownership check.

Hard part one: the offline conflict. Two devices both had version 7 and both edited while disconnected. Device A reconnects and commits version 8 successfully, because its parent matched. Device B reconnects and tries to commit with parent 7, but the server is now at 8, so the conditional commit fails. That failure is the correct outcome. B has not lost anything; it has been told it is behind. B then downloads version 8, and because it cannot merge arbitrary files, it writes its own edit as a new file named for the conflict and syncs that as a separate object. Both versions now exist and both are visible. The costs: the user sees a file they did not create and has to deal with it, and a folder synced across many devices after a long offline period can produce several conflicted copies, which feels messy. It is still the right trade. The alternative is choosing a winner by timestamp, and consumer device clocks are often wrong, so that choice would delete real work with no error and no way to notice.

Hard part two: 50,000 files at once. A naive client does 50,000 hash checks, 50,000 uploads and 50,000 commits, and the round trips alone take hours while the metadata service sees a burst of writes for one account: a self-inflicted hot partition. So: the client batches. Hash existence is asked in batches of hundreds. Uploads run with bounded parallelism, a few at a time, so one user cannot saturate their own connection or my ingest. Commits are batched into a single call that advances the cursor once for the whole batch rather than 50,000 times, which is what stops every other device receiving 50,000 notifications. Cost: the batch commit is more complex, and a partial failure has to be resumable at file granularity within the batch.

Chunk garbage collection. Chunks are shared between versions, between files, and in shared folders between users. Deleting a file cannot delete its chunks, because another version or another file may reference them. So chunks are reference counted, or swept periodically by finding unreferenced hashes. A sweep must be conservative and ignore anything recently created, because a chunk uploaded seconds ago may not yet have a version referencing it. Deleting a chunk that is still needed is unrecoverable data loss, so the job errs toward keeping too much. Cost: real money spent on storage that nothing references, and a job nobody should make more aggressive without careful review.

History and restore. Versions are cheap because they share chunks, so keeping 30 days of history costs far less than 30 copies. Beyond that, thin them out. Restore is just committing an old version as a new one, which keeps history append-only and means restore is itself undoable.