71% less inference spend. More room to move.

Around 90,000 documents arrived each month: bills of lading, customs forms, and delivery notes. Traffic rose when European offices opened, peaked again with the US, then fell almost silent for fourteen hours.

Two GPU instances stayed on throughout. They were sized for the busiest part of the day, even when there was nothing to classify.

The bill brought us in. The deployment process gave us a bigger problem to solve.

Find what the workload actually needs

Four weeks of measurement showed 6% GPU utilisation.

The classifier was a fine-tuned encoder of around 400 million parameters. Its hardware configuration came from an earlier experiment with a much larger model. The experiment had ended; the infrastructure had not changed.

Only one person could safely deploy the service. Four months of model improvements were waiting in an unmerged branch.

Change the system around the model

We moved inference into serverless GPU containers that could scale to zero between requests. The migration took a week. Three supporting changes made it useful.

Keep capacity warm when people need it. The operations team used an interactive review screen, so cold starts were a poor fit during working peaks. A small warm pool followed the two predictable traffic windows.

Batch at the queue. Documents arrived in bursts from the same senders. Grouping them before dispatch reduced invocation count by about two thirds.

Make release and rollback repeatable. We pinned the model version in the repository, documented deployment and rollback commands, and added an evaluation gate. The operations team helped build the 1,400-document test set behind it.

The results

• 71% lower inference spend month over month.
• 610 ms p95 interactive latency, down from 890 ms.
• Five people able to deploy, up from one.
• 4.1 percentage points higher classification accuracy on the shared evaluation set, after the waiting improvements shipped.

The backlog reached production over the following three weeks. The infrastructure change did not itself make the classifier more accurate. It gave the team a dependable way to ship the work that did.

The lesson beyond the bill

Measure the workload you have today. Then look at the release process that has kept its infrastructure unchanged.

In this case, smaller infrastructure and a clearer deployment path helped together. The team gained both a lower bill and the confidence to keep improving the product.

Explore cloud and platform engineering or talk to us about your system.