Pallet’s memory layer
Ask an experienced freight operator which org code a familiar importer files under, and they rarely need to look it up. They have filed for that importer many times before, and the code appears nowhere on the booking documents anyway. Freight operations run on this kind of tacit knowledge, specific to the customer, built up over years in people’s heads.
Pallet’s memory layer moves that knowledge into the platform. For each organization it stores the record of past work: standard operating procedures, prior agent runs, and the cases where a human stepped in to correct an agent’s output. Every Pallet agent ships with it. When an agent takes on a new task, the relevant memories are pulled into the model’s context, so it decides with the precedent an operator would bring and handles more cases on its own with higher accuracy.
The benchmark
Memory-Bench measures what that memory is worth: the accuracy, reliability, and cost of Pallet’s agents with and without it, across seven large language models, both frontier and open-source. The test is deliberately narrow. A production agent runs a workflow of many steps; Memory-Bench scores a single one of them, the decision point where memory is served to the model as context. We built 78 tasks from decisions Pallet agents make in production for customers in brokerage, intermodal drayage, and truckload carriage. They come in four types:
- ISF importer of record. An Importer Security Filing needs the right org code in the brokerage’s TMS for the importer named on the booking documents.
- Arrival-notice reconciliation. A field — seal number, pickup number, container type — conflicts between the TMS and the scanned arrival notice, and the agent must pick which value goes on the record.
- EDI order triage. An inbound EDI (Electronic Data Interchange) order has failed validation, and the agent decides whether processing resumes or the order goes to manual entry.
- Email routing. An operational email must land in the right one of the organization’s 14 queue categories.
Every task runs twice over identical inputs, once with the retrieved memories and once without. Nothing else changes, so the gap between the runs is the memory layer’s contribution alone.
Results
Accuracy versus cost per decision, with and without the memory layer
Seven models on the benchmark’s 66 gold-graded decisions (Figure 3 in the full article).
| Group | No memory | With memory | Change |
|---|---|---|---|
| Gemini 3.1 Pro | 21.2% | 57.6% | +36.4 pp |
| Gemini 3.5 Flash | 22.7% | 57.6% | +34.9 pp |
| GPT-5.2 | 19.7% | 56.1% | +36.4 pp |
| GPT-5.5 | 24.2% | 56.1% | +31.9 pp |
| GLM 5.2 | 19.7% | 53% | +33.3 pp |
| Sonnet 5 | 21.2% | 53% | +31.8 pp |
| Opus 4.8 | 19.7% | 51.5% | +31.8 pp |
Accuracy. With the memory layer, pooled first-try accuracy across the seven models rises from 22% to 62% on the 44 tasks with a dependable human answer key; every model gains 36 to 45 points. The baseline is the finding to sit with: without organizational history, even the strongest models get about one decision in five right. The missing input is knowledge of the customer’s operation, and more reasoning power is a poor substitute for it.
Reliability. A production agent meets the same kind of case over and over, so we also test consistency: a case counts as solved only if the model gets it right on every one of its repeated attempts. Over 50 runs per case with GPT-5.5, the with-memory score barely decays as that bar rises — from 60% on a single attempt to 53% when all 32 attempts must be correct — while without memory the model sits at 15–20% throughout. The cases memory solves, it keeps solving.
Cost. Retrieved memories make each request longer, but with the precedent in view the model spends less effort getting to an answer, and the two effects nearly cancel: per-decision cost changes by under a quarter of a cent for every model on the panel ($0.0093 without memory versus $0.0094 with it, for GPT-5.5). Accuracy roughly triples at flat spend, so the cost per correct decision falls 2–3× — from 5.2 to 1.8 cents on GPT-5.5.
The full article breaks these numbers down by task type and by how much usable history each customer had, and reports the controls that rule out prompt format as the source of the lift. The knowledge was in the building all along. Memory-Bench measures what happens when the agents get to use it.
Authors
