How We Scaled Our API to Meet Customer Needs Without Compromising Performance
During onboarding for a new client, we identified the need for frequent, complete dataset downloads from our platform, such as on a daily or weekly basis. The frequency, combined with the growing dataset over time, posed challenges for our existing API structure that we needed to address. Rather than receiving data in small, targeted pieces, the client required a way to efficiently and reliably download complete datasets without impacting the performance of the API.
To keep our APIs predictable and easy to use for both our developers and clients we wanted to find a way to maintain that same predictability. To address both the performance and predictability concerns, streaming JSON seemed a good fit. Instead of many paged requests or generating a large JSON array, our solution could stream each JSON object independently. As each object is pulled from the database, it’s written directly to the response stream.
Our current APIs are built on top of a well-structured domain model, backed by OpenAPI specifications.
This approach would not only reduce memory usage server-side, but would also allow the client to start processing data immediately without waiting for the full response. It also matched our document-based databases very well. By creating a streaming endpoint for each resource collection there would be no need for any cross-collection joins or complex element queries.
To support streaming JSON, two popular data formats exist. JSON lines, or jsonl, in which each object is separated by a newline character and JSON-seq which uses the ‘record separator’-character to precede every object. JSON lines seemed like a more popular choice although JSON-seq is backed by RFC74641 where JSON lines currently does not2. Instead of choosing one we opted to implement both, allowing the client to specify the desired format using the Accept header.
A final concern was how to handle interruptions in the stream. We wanted to allow our client to be able to restart a stream right before the interruption occurred. To facilitate this, every object would include an index field which could be passed to the server in the form of a ‘continue-from’ query argument.
As of writing, support for streaming JSON has been developed and rolled out to production. It has proven to be an elegant solution for transferring large datasets smoothly and reliably, without introducing new technologies. This approach enables us to maintain high API performance and scalability, while upholding our core design principles.
The video below shows the new streaming capabilities of our APIs by using "curl".