Files Over REST APIs: Designing and Consuming File Endpoints

A growing share of the files an administrator moves never touch a classic transfer protocol at all. The backup appliance exposes an API. The HR platform's exports come from an API. The ticketing system wants attachments pushed to an API. Somewhere in your scripts directory there is already a curl command with a bearer token pulling a nightly file from a vendor — and somewhere in your future is a request to help design such an endpoint for your own organization's application.

This article covers both sides of that coin. For the consumer: how file APIs are typically shaped, the patterns to recognize (and the traps — buffering, pagination, redirects) so your scripts survive contact with real data volumes. For the designer: the three upload patterns and two download patterns that cover nearly every need, how to stream instead of buffer, how to paginate listings so they stay correct, and how to version the whole thing so you can improve it without breaking every client. A REST API, for our purposes, just means: an HTTP interface where things (files, in our case) are resources at URLs, manipulated with the standard verbs.

This is part of our HTTP/HTTPS File Transfer series, and it leans especially on the curl and wget cookbook for the flags used throughout.

What Makes File Endpoints Different

Most API traffic is small, structured, and parseable: a request for JSON, a response of JSON, done in milliseconds. Files break every one of those assumptions. The payload is large — six to nine orders of magnitude larger than a typical JSON body — opaque, and binary. Requests run for minutes, not milliseconds, which drags timeouts, proxies, and interrupted connections into scope. And every file actually has two representations that want different handling: the bytes (the content itself) and the metadata (name, size, checksum, owner, state — small, structured, JSON-friendly).

Nearly every good file-API design decision traces back to respecting that split: metadata travels as JSON like any other resource, bytes travel as raw streams with the machinery large payloads need, and the API is explicit about which endpoint carries which. Designs that blur the two — say, base64-encoding file bytes inside a JSON field, inflating them by a third and forcing full-memory parsing — work in the demo and fail at the first real payroll archive.

Upload Endpoint Patterns

Three patterns cover essentially all file upload APIs you will meet or build:

Pattern 1: multipart POST — file and metadata together. One request carries JSON-ish fields and the file, packaged as multipart/form-data (dissected at the wire level earlier in this series). It is simple, browser-compatible, and fine for small-to-medium files. Its weaknesses appear with size: one all-or-nothing request, and the server must parse the envelope before it can do anything.

Pattern 2: raw body — bytes in the body, metadata around it. The file is the entire request body (a PUT to the file's URL, or POST to a collection), with metadata in the URL and headers (Content-Type, Content-Length). Clean for scripts — this is just curl -T — and streams naturally. Idempotent PUT makes retries safe.

Pattern 3: two-step — create, then upload. The client first POSTs the metadata; the server creates the file record and returns an upload destination — very often a presigned URL pointing at object storage; the client then PUTs the bytes there and finally confirms. On the wire it looks like this:

POST /v1/files
{"name": "q3-report.pdf", "bytes": 4821133, "sha256": "9f86d0..."}
→ 201 Created
{"id": "f_8412", "upload_url": "https://storage.example.com/...signed...", "state": "pending"}

PUT <upload_url>
(raw file bytes)

POST /v1/files/f_8412/complete
→ 200 OK   {"id": "f_8412", "state": "available"}

The two-step's virtue is that the heavy bytes bypass the application server entirely — it handles two tiny JSON requests while storage absorbs the gigabytes. It also creates a natural place for declared checksums, chunked strategies for large files, and a clean "pending until confirmed" state so consumers never see half-uploaded files. Its cost is client complexity: three requests and state to track. As a rule of thumb: pattern 1 for small files from browsers, pattern 2 for scripts and modest sizes, pattern 3 whenever files are big or volume is serious.

Three designer notes that pay off in every pattern. Return 201 Created with a Location header (and the metadata body) so clients learn the new file's URL from the response instead of guessing. Accept an idempotency key — a client-supplied unique token per logical upload — so a client that retries after a timeout gets the original result instead of creating a duplicate; it is the POST world's substitute for PUT's natural retry-safety. And make failures machine-readable: the correct status code (413 for too large, 415 for a disallowed type, 409 for a conflict) plus a small JSON body with a stable error code and a human sentence. Scripts branch on codes; humans read sentences; both parties get what they need.

Download Endpoint Patterns

Downloads want the metadata/bytes split made literal, with two URLs per file:

GET /v1/files/f_8412            → 200, JSON metadata (name, size, checksum, state)
GET /v1/files/f_8412/content    → 200, the raw bytes
                                  (or 302 → a short-lived storage URL)

The metadata endpoint lets clients check size, state, and checksum cheaply — poll it while an export is generating, verify after downloading. The content endpoint serves bytes with an accurate Content-Type, a Content-Disposition filename for browser-facing use, and — if files are large — support for Range requests so interrupted downloads can resume (the mechanics live in large files over HTTP).

The parenthetical in that sketch deserves emphasis, because it bites script authors constantly: many APIs answer the content request with a redirect (302) to a short-lived presigned storage URL rather than serving bytes themselves. A script without -L saves the empty redirect response and reports success; add -L and curl follows through to the real bytes. One subtlety inside the subtlety: some storage backends reject requests that carry the API's Authorization header into the redirected request — the signed URL is the authorization — which is why curl deliberately drops custom headers when following a redirect to a different host. If a download mysteriously fails only at the redirect hop, that dance is usually why.

The Not-Ready-Yet Pattern: Asynchronous Exports

One more download shape deserves its own name, because scheduled jobs meet it constantly. Many vendor "downloads" are not sitting on disk waiting — they are generated on demand: you request this month's report, the system spends four minutes assembling it, and only then is there anything to fetch. A synchronous GET cannot survive that politely (it would hold a connection open through the whole generation, at the mercy of every timeout in the path), so well-built APIs make it asynchronous: the request to create the export returns immediately — typically 202 Accepted plus an id — and the client polls a status resource until the state flips:

while :; do
  state=$(curl -fsS -H "Authorization: Bearer $API_TOKEN" \
    "https://api.example.com/v1/exports/$EXPORT_ID" | jq -r '.state')
  [ "$state" = "complete" ] && break
  [ "$state" = "failed" ] && { echo "export failed" >&2; exit 1; }
  sleep 30
done

Three habits make polling a good citizen. Poll at a civilized interval (and honor any Retry-After hint) rather than hammering every second. Put a ceiling on total wait, so a stuck export fails your job loudly instead of hanging it forever. And treat failed as a terminal state to report, not something to retry blindly — the retry belongs at the create-export step, if anywhere. Some APIs offer webhooks instead, calling your endpoint when the file is ready; elegant, but a cron job cannot answer the phone, so for classic scheduled work, disciplined polling remains the workhorse.

Streaming vs Buffering

Here is the difference between a file API that survives production and one that falls over on the first quarter-end archive. Buffering means holding the entire payload in memory before doing anything with it; streaming means passing bytes through in small pieces as they arrive, so memory use stays flat no matter the file size.

Server-side, the arithmetic is unforgiving: a handler that buffers uploads needs file-size × concurrent-uploads of RAM — ten simultaneous 2 GB uploads is 20 GB just to receive them. Frameworks help when configured (spooling big bodies to temp disk instead of RAM), but the design intent has to be streaming: read the request in chunks, write to storage in chunks, compute the checksum as bytes flow past. The same applies to downloads — stream from disk to socket, never "load file, then send."

Client-side, the trap is quieter. curl streams by nature: -o file and -T file move bytes without ever holding the whole payload. But most HTTP libraries in scripting languages buffer the entire response into memory by default, because that is convenient for JSON. The result is the classic bug: a Python or PowerShell script that has fetched daily exports flawlessly for a year dies with an out-of-memory error the day the export gets big. Every mainstream library has a streaming mode — a flag or an iterate-in-chunks API — and file-fetching scripts should use it from day one.

Listing Files: Pagination Done Sanely

Every file API needs a listing endpoint (GET /v1/files), and every listing endpoint eventually faces a directory with a hundred thousand entries. Returning them all in one response punishes everyone, so listings are paginated: each request returns one page plus a way to ask for the next.

The two schemes you will meet differ in an important way. Offset pagination (?offset=200&limit=100) is intuitive and fragile: if files are added or deleted while you walk the pages — which is exactly what happens on a busy system — entries shift across page boundaries and your walk silently skips or duplicates files. For a script reconciling "did I fetch everything?", silent skips are poison. Cursor pagination fixes this: each response includes an opaque bookmark, and presenting it returns the next page after that stable position, unaffected by churn. A robust consumption loop looks like:

cursor=""
while :; do
  page=$(curl -fsS -H "Authorization: Bearer $API_TOKEN" \
    "https://api.example.com/v1/files?status=complete&limit=100&cursor=$cursor")
  echo "$page" | jq -r '.items[].id' | while read -r id; do
    curl -fsSL -H "Authorization: Bearer $API_TOKEN" \
      -o "exports/$id.bin" "https://api.example.com/v1/files/$id/content"
  done
  cursor=$(echo "$page" | jq -r '.next_cursor // empty')
  [ -n "$cursor" ] || break
done

(jq is the standard command-line JSON processor; the loop fetches every completed file, page by page, until no cursor remains.) Designers: prefer cursors, keep them opaque so clients cannot construct or reorder them, and always include an explicit "no more pages" signal. Consumers: treat pagination as mandatory even when today's listing fits on one page — the API that returns 40 items now will return 4,000 someday, and only paginating clients will notice.

Versioning: The API and the Files

Version the API because clients calcify: that contractor-written script consuming your endpoint will run unmodified for years, and a renamed field breaks it at 2 a.m. The convention is a version marker in the path (/v1/files) or a header. The policy matters more than the mechanism: additive changes (new optional fields, new endpoints) are safe within a version; breaking changes (renames, removals, changed semantics) require a new version, with the old one kept alive through a published deprecation window. Consumers, mirror image: pin the version explicitly in your scripts and never parse by position or assume field order.

Version the files too, or at least decide explicitly not to. If an upload to an existing name overwrites, say so, and give concurrent writers a guard — the ETag plus If-Match pattern, where an update only succeeds if the file is still the version the client last saw, turns silent lost updates into visible conflicts. If instead each upload creates an immutable new version with its own id, downloads become perfectly cacheable and audit questions ("which version did we send the regulator?") stay answerable. Immutable versions cost storage and need retention rules; overwriting costs history. Either is defensible; ambiguity is not.

Consuming Third-Party File APIs from Scripts

Most admins meet file APIs from the consuming side, wiring a vendor's endpoint into scheduled jobs. A survival kit, distilled:

• Prototype with curl -v before scripting. Watch one full conversation — auth header in, status and headers out, redirect hops — so the script automates something you have seen work, not something you inferred from documentation.
• Respect rate limits properly. A 429 Too Many Requests response is an instruction, not an error: wait — for the duration in the Retry-After header if provided, with increasing backoff otherwise — then continue. A loop that hammers through 429s gets tokens revoked.
• Verify what you fetched. If the metadata includes a checksum, compare it against the downloaded bytes before declaring victory; if it includes a size, at least check that. Truncated downloads that "succeeded" are a classic silent failure.
• Log request ids. Most APIs return a request identifier header; write it into your job log. When you open a support ticket, that id is the difference between "please investigate" and an answer.
• Use the script-safety kit. Everything from the cookbook applies unchanged: -fsS, timeouts, bounded retries, secrets outside the command line.

And place the API leg inside the larger flow honestly. A common real-world job is a relay: pull the nightly export from a vendor's API, then deliver it onward to the systems and partners that expect files in folders — an SFTP inbox, an FTPS drop. Scripting the API half is natural; for the delivery half, a scheduling tool like Sysax FTP Automation handles the scheduled FTP/FTPS/SFTP legs with folder monitoring, retry, and error handling as configuration, so the fragile hand-rolled part of the pipeline stays as small as possible.

When an API Is the Wrong Interface

File APIs earn their complexity when transfer is woven into an application: uploads triggered by users, files tied to records, integrations reacting to events. They are the wrong tool for classic bulk exchange — a partner delivering ten thousand files nightly does not want to mint tokens and walk cursors, and their operations team's tooling speaks SFTP, with its standardized batch semantics, native resume, and decades of convention. The full decision framework is in when HTTPS beats SFTP; the short version is that APIs are for applications, transfer protocols are for batch, and mature environments run both without embarrassment.

The design core travels well, though: separate metadata from bytes, stream everything, paginate with cursors, version deliberately, and make every operation safe to retry. Master those five and you can consume any vendor's file API with confidence — and when someone asks you to help design one, you will know exactly which questions to ask first.

Frequently Asked Questions