Cutting an LLM agent prompt from 100K to 16K tokens

At an automotive services startup, the B2B team estimated service prices by calling workshops and waiting for replies. Replies were slow, and some clients left because of it. I was the only developer on a team with one PM and one data analyst, and the brief was: build an agent that estimates prices from historical data, on a budget that only allowed GPT-4o-mini.

The first version was 100K tokens

The naive prompt held everything: instructions, the full SKU list, brand data, and every example conversation. It was slow, expensive, and wrong more often than it was right. The model kept pulling bad references for rural areas and mixed SKUs, and tool calling fired every tool at once.

Four changes

1. A prompt loader instead of one string

Split the prompt into blocks: instructions, SKU list, brand data, use-case examples. Each block is loaded on demand. The system prompt stopped being a file and became a small function.

def build_prompt(intent: Intent, query: str) -> list[Message]:
    blocks = [INSTRUCTIONS]
    blocks += sku_blocks_for(intent)          # only relevant SKU groups
    blocks += few_shots_for(intent, k=3)      # see below
    return [system("\n\n".join(blocks)), user(query)]

2. Intent-based few-shot selection

Instead of sending every example, classify the intent first (price estimate, nearest workshop, follow-up question) and send only the three most relevant examples. This alone removed most of the tokens.

3. Grouped tool calling

Tools were grouped by category. The agent only sees the tools for the detected intent, so it stops calling all of them in parallel and the response time drops.

4. A fallback that asks

When confidence is low or the answer is unclear, the agent asks a clarifying question instead of guessing. For a B2B estimate, a short question beats a confident wrong number.

Result

The agent went live for the estimation team. Next step was WhatsApp as the frontend, which meant no UI to build for a one-person engineering team.