The Hard Parts of Fine-Tuning Our First Production Model

We fine-tuned and deployed our first production model, achieving frontier-level accuracy at roughly one-third the cost. This is how we got there—and what we learned from the mistakes along the way.

Why build our own model? We had two main reasons. The first was economics. Frontier models still lead on complex, general-purpose tasks, but open models have become capable enough to match their performance on narrower tasks after fine-tuning, at a lower cost.

The second was data control. Many of our customers are increasingly concerned about where their data goes, who processes it, and how it may be retained. Because our models run in infrastructure we control, we can provide stronger guarantees around data isolation, storage, access, and retention—without sending every request to an external model provider.

1. Choosing the Base Model

Our first problem was finding a base model that could realistically reach our accuracy target and still pay for itself.

Self-hosting a fine-tuned model has a different cost shape than using pay-per-request frontier models. We pay a fixed cost per GPU, and each GPU handles a certain amount of traffic, so the curve looks like a step function rather than a line. Those steps rise far more slowly than frontier pricing does, so the economics get better in the long term—but we still need to clear the cost of the first GPU.

Pallet also cares deeply about data isolation. We could not pool data across customers and amortize one model across many similar workloads. A single customer’s use case had to break even on its own, which ruled out larger models that needed bigger or more numerous GPUs.

We addressed this in two steps. First, we limited the custom model’s operating envelope. Some inputs exceeded 250k tokens, and no sufficiently small model at the time could reliably handle that much context. Instead of sizing the entire deployment for those outliers, we routed them to a frontier model and optimized the fine-tune for the remaining traffic.

Second, we narrowed the candidates based on cost, GPU memory, and their likelihood of reaching our accuracy target. That last factor was the hardest to predict: fine-tuning a model with an insufficient capability ceiling would waste a nontrivial training run. For this first case, we relied on early experiments, public evidence, and a modicum of intuition. In the future, we want to formalize the decision with an internal benchmark.

Nemotron was quickly ruled out because of its size and several correctness issues. Qwen’s MoE model was cheaper to run, but our bottleneck was GPU memory rather than compute. Although it activated only a subset of its parameters per token, all 35B parameters still had to reside in memory, leaving less capacity for long inputs and the KV cache. The dense 27B model offered more memory headroom, making it the better fit for our workload.

2. Building the Training Data and Evaluation Loop

Once we had chosen a base model, the next problem was building the dataset and iterating on accuracy.

These sound like distinct problems, but they are coupled. Because we built the training set from production traces, it contained the production model’s mistakes as well as its successes. We had to adjust the data distribution to keep the fine-tuned model from overlearning those mistakes.

Most of these steps are self-explanatory, but anonymization deserves a closer look. Production traces can contain customer-sensitive information such as PII, shipment identifiers, locations, and even passwords. Before an example entered the training dataset, we replaced those values with typed placeholders while preserving the structure and relationships the model needed to learn.

Simply deleting sensitive values would have distorted the examples. The model may need to recognize that the same shipment identifier appears in both an email and a tool response. Consistent substitution preserves that relationship without retaining the original value. This let the model learn the workflow while minimizing the customer data included in the training set.

An Indecisive Catch-all

One decisive data issue was an indecisive catch-all category. When the frontier model could not confidently classify a task, it was allowed to select a label that effectively meant, “I cannot process this.” That category accounted for roughly 18% of the initial training set, making it the single largest label.

Rather than treating the catch-all as an exceptional escape hatch, the fine-tuned model learned that giving up was one of the safest and most common answers. Validation accuracy stalled around 84%. Once we redistributed the training data to remove the indecisive examples and force the model to select a real category, accuracy jumped to over 95%.

Replaying Tool Calls

Our examples were not simple prompt-response pairs. The model could call tools to decide what to do next.

During evaluation, a candidate might call the same tool with slightly different arguments. Calling production tools would be slow, nondeterministic, and potentially unsafe, while requiring an exact match with the recorded call would be too brittle.

We took a middle path: the eval runner fuzzy-matched each tool call against the recorded arguments and replayed the corresponding output. A separate judge evaluated whether the arguments were actually correct. This allowed harmless variations without ignoring meaningful differences in behavior.

Judging Correctness

We first narrowed the comparison to fields that actually affected the outcome. Raw outputs often differed for operationally irrelevant reasons: bookkeeping metadata changed, labels used different formatting, or distinct values triggered the same downstream action.

We kept the raw comparison visible and versioned every normalization rule. This let us distinguish serialized output equality, agreement on decision-driving fields, equivalent downstream actions, and whether the chosen action was actually correct.

The frontier model was a useful reference, but matching it did not prove correctness. That required an independently reviewed test set, especially for cases where the models disagreed.

3. Productionizing the Model

Offline evaluations told us the model was promising, but they were not enough to put it in front of customers. Before rollout, we needed to prove that it behaved correctly on live traffic and failed safely when it could not.

We built a shadow-testing system that ran the candidate alongside the production frontier model. The candidate received the same messages, tool schemas, and recorded tool results, but its output could not affect the live workflow. This let us test against thousands of real inputs without introducing customer-facing risk or calling production tools twice.

Once the model met our agreement, error-rate, and latency thresholds, we rolled it out gradually. The frontier model remained as an automatic fallback for timeouts, malformed outputs, unsupported context lengths, and other failures. This captured the economics of the fine-tuned model without making reliability depend on it handling every request successfully.

What’s Next

Our first fine-tune handled a simple, focused task. The harder and more valuable target is the long, multi-step interactions our agents actually run: conversations that span many tool calls, where each decision depends on the last and one wrong call derails everything after it. We plan to investigate Iterative SFT (iSFT) for these problems.

Further out, we plan to build Pallet LLM: a foundation model of our own, trained on synthetic logistics data we generate at scale rather than a generic open base. It would bake in the vocabulary, workflows, and tool patterns common to freight operations, so every customer model starts from a logistics-native foundation. Each customer’s model becomes a fine-tune on top of it, and every new customer is onboarded onto Pallet LLM from day one.

We’ve barely scratched the surface of what fine-tuned models can do in supply chain. If this kind of work interests you, we’re actively hiring across all functions.