Step 3 — Uploading a product image

The endpoint with the most ways to go wrong, and the four defences that fix them.

Topic 77 of 98 · 10 min to read · 50 min to build

After this topic you can

Build an upload endpoint that caps the body size with http.MaxBytesReader.
Check a file's real type with http.DetectContentType, not its extension.
Stream the file to a server-chosen name with io.Copy and clean up on failure.

What multipart means

A normal JSON request has one body. Uploading a file usually needs two things at once, the file itself and some fields describing it, so browsers use a format called multipart/form-data, which packs several named parts into one body.

Go parses it for you with r.ParseMultipartForm, and r.FormFile("file") hands back the part you want. The parsing is the easy bit. The safety is not.

None of the defaults are safe

Size: by default Go will read as much as the client sends. A 4GB upload can fill your memory or disk before any of your checks run.
Type: the file extension and the Content-Type header both come from the client, so both can lie.
Name: a filename of ../../etc/passwd writes wherever it likes if you use it to build a path. Recent Go versions strip the directory part from the name, but do not build a path from it anyway.
Cleanup: if the file is written and the database row fails, you have a file nothing points at, forever.

Part 1 — cap the body, then read the file

The size cap has to come first, before anything reads the body. MaxBytesReader enforces the limit. The number passed to ParseMultipartForm is how much it keeps in memory; anything larger spills to temporary files on disk.

image.go, part 1 of 4 — the top of Upload (the body of func (h *Handler) Upload(w http.ResponseWriter, r *http.Request))

1	const maxBytes = 5 << 20                       // 5 MB
2
3	// DEFENCE 1 — cap the body before anything reads it.
4	r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
5	if err := r.ParseMultipartForm(maxBytes); err != nil {
6		httpx.Error(w, 413, "file too large")
7		return
8	}
9
10	file, hdr, err := r.FormFile("file")
11	if err != nil {
12		httpx.Error(w, 400, "a file field is required")
13		return
14	}
15	defer file.Close()

Part 2 — check what the file really is

Read the first 512 bytes, ask Go what they look like, and reject anything not on your list. Then rewind, because you have just consumed part of the file.

image.go, part 2 of 4 — inside Upload

1	// DEFENCE 2 — sniff the real type from the first 512 bytes.
2	head := make([]byte, 512)
3	n, _ := file.Read(head)
4	mime := http.DetectContentType(head[:n])
5	if !allowed[mime] {
6		httpx.Error(w, 415, "unsupported file type: "+mime)
7		return
8	}
9	if _, err := file.Seek(0, io.SeekStart); err != nil {   // rewind before copying
10		httpx.Error(w, 500, "storage error")
11		return
12	}

Forgetting the Seek is a bug you only notice later: every stored file is missing its first 512 bytes.

Part 3 — you choose the filename, and you stream it

The client never gets to influence the path. Generate a random name, join it to your upload directory, and stream the bytes across.

image.go, part 3 of 4 — inside Upload

1	// DEFENCE 3 — we choose the filename. The client never touches the path.
2	var b [16]byte
3	if _, err := rand.Read(b[:]); err != nil {              // crypto/rand
4		httpx.Error(w, 500, "storage error")
5		return
6	}
7	stored := hex.EncodeToString(b[:]) + extFor(mime)
8	dst := filepath.Join(h.uploadDir, stored)
9
10	out, err := os.Create(dst)
11	if err != nil {
12		httpx.Error(w, 500, "storage error")
13		return
14	}
15	defer out.Close()
16
17	size, err := io.Copy(out, file)                         // streams; memory stays flat
18	if err != nil {
19		os.Remove(dst)
20		httpx.Error(w, 500, "write failed")
21		return
22	}

Part 4 — record it, and clean up if that fails

The file is on disk. If the database row fails now, that file is orphaned: nothing points at it and nothing will delete it. So remove it on the failure path.

image.go, part 4 of 4 — the end of Upload

1	img := &Image{StoredName: stored, MIME: mime, SizeBytes: size}
2
3	// DEFENCE 4 — no orphans. If the row fails, the file goes too.
4	if err := h.svc.Save(r.Context(), img); err != nil {
5		os.Remove(dst)
6		httpx.FromError(w, err)
7		return
8	}
9
10	_ = hdr.Filename          // keep only for display, never for a path
11	httpx.JSON(w, 201, img)

The whole file, assembled

Here is how the four parts fit together, with the imports and the two small helpers the parts use. Parts 2 to 4 go where the comment says.

internal/adapter/handler/http/image.go

1package http
2
3import (
4	"crypto/rand"
5	"encoding/hex"
6	"io"
7	"net/http"
8	"os"
9	"path/filepath"
10
11	"github.com/you/shop/internal/adapter/handler/http/httpx"
12)
13
14// httpx.Error, httpx.JSON and httpx.FromError are the JSON helpers you
15// copied across in step 1. They stay in their own package, httpx, so the
16// calls read the same as in project two.
17
18func (h *Handler) Upload(w http.ResponseWriter, r *http.Request) {
19	// part 1, part 2, part 3, part 4 — in that order
20}
21
22var allowed = map[string]bool{
23	"image/jpeg": true,
24	"image/png":  true,
25	"image/gif":  true,
26	"image/webp": true,
27}
28
29// extFor picks the extension from the sniffed type, never from the client.
30func extFor(mime string) string {
31	switch mime {
32	case "image/jpeg":
33		return ".jpg"
34	case "image/png":
35		return ".png"
36	case "image/gif":
37		return ".gif"
38	case "image/webp":
39		return ".webp"
40	}
41	return ""
42}

Why io.Copy and not io.ReadAll

io.ReadAll pulls the entire file into memory before writing it. Ten people uploading 5MB at once is 50MB of heap, and it grows with every concurrent upload and every megabyte you allow.

io.Copy streams through a small fixed buffer, so its memory use does not grow with the file. This is what io.Reader and io.Writer exist for: an uploaded file and a file on disk are both somewhere bytes come from and go to.

Watch out: http.DetectContentType reads the file’s first bytes (its magic number), so it reports what the content looks like, whatever the name says. It is a type check, not a malware check. Be careful serving uploads from your main domain: a stored HTML or SVG file that a browser renders is stored cross-site scripting on your own origin. Many sites serve user uploads from a separate domain for that reason.

Try to break it yourself

terminal — all five of these must be rejected or contained

1# a real image → 201
2curl -si -X POST localhost:8080/v1/products/1/image \
3  -H "Authorization: Bearer $ADMIN" -F "file=@photo.jpg" | head -1
4
5# too big → 413
6head -c 50M /dev/urandom > big.bin
7curl -si -X POST localhost:8080/v1/products/1/image \
8  -H "Authorization: Bearer $ADMIN" -F "file=@big.bin" | head -1
9
10# a script wearing a .jpg name → 415
11echo '#!/bin/sh' > evil.sh && cp evil.sh evil.jpg
12curl -si -X POST localhost:8080/v1/products/1/image \
13  -H "Authorization: Bearer $ADMIN" -F "file=@evil.jpg" | head -1
14
15# a path in the filename → stored under a name WE generated
16curl -si -X POST localhost:8080/v1/products/1/image \
17  -H "Authorization: Bearer $ADMIN" \
18  -F "file=@photo.jpg;filename=../../../tmp/evil.jpg" | head -1
19ls /tmp/evil.jpg          # must not exist
20
21# a customer trying to upload → 403
22curl -si -X POST localhost:8080/v1/products/1/image \
23  -H "Authorization: Bearer $CUST" -F "file=@photo.jpg" | head -1

The short version

MaxBytesReader before anything reads the body, then ParseMultipartForm with the same limit.
Sniff the type with http.DetectContentType. Never trust the extension or the client header.
Generate the stored filename with crypto/rand. The client filename is display text only.
io.Copy streams; if the database insert fails, os.Remove the file.

Remember: Every part of an upload is attacker-controlled: the size, the name, the type and the contents. Decide all four yourself.

Build this before moving on

Build the upload endpoint and run all five attacks above. Every one must be rejected or safely contained.

Then try these in your own editor

1Replace io.Copy with io.ReadAll and watch memory while uploading a large file.
2Remove the MIME check and upload an SVG containing a script tag. Then put the check back.
3Make the database insert fail on purpose and confirm the file is removed from disk.
4Add SHA-256 dedup with io.TeeReader, so the same photo uploaded twice is stored once.

Official documentation

Check yourselfTake the Project 3 — Small shop test