Small Models vs. Large Models: What 4B Parameters Can and Cannot Do

Running a local model to save money is a natural idea. Cloud APIs charge by the token, and multi-agent systems burn money quickly. Put an open model on a consumer GPU and inference becomes free. It sounds great.

I tried it. The conclusion is blunt: a 4B-parameter model is basically unusable for a multi-agent system.

The starting point: vLLM and a 4B model

The plan was to deploy an open 4B-parameter model with vLLM as the multi-agent system's local inference backend. VRAM set the size: one GPU couldn't run a much larger model.

Deployment itself was uneventful. The model loaded and the API started. The failure only appeared when it had to behave as an agent across several turns.

The first trap: the system prompt was too large

In a multi-agent system, every agent carries its own role definition, tool list, and behavioral rules. All of that lived in the system prompt. Together, it was about 14KB.

14KB is unremarkable for a large model. For the 4B model, it was fatal. The prompt consumed its short effective context window. The result was incoherent output, malformed tool calls, and failures to produce even basic JSON.

I wrote a strip plugin specifically to compress the system prompt. It removed redundant descriptions and reduced formatted tool definitions to their smallest useful form. The prompt went from 14KB to about 2KB.

After compression, tool calling finally worked—at least in a single-turn conversation.

The second trap: multi-turn conversations collapsed

A model that works for one turn may still fail completely over many turns.

An agent may need to call a search tool, inspect the result, call a code tool, and format the response. It must preserve its role while tracking history, tool state, and operation order. One valid tool call is only one link in the chain.

This is where the 4B model exposed its capability ceiling:

• Role amnesia: after a few turns, the agent forgot its role. The QA agent started writing code; the PM agent started running tests.
• Broken sequencing: workflows that required a query before an operation skipped the query, operated immediately, or called the same tool repeatedly.
• Format degradation: as the conversation grew, tool-calling JSON began to break. Parameters disappeared or arrived with the wrong types.

These failures survived the system-prompt compression. Once the context grew, the model couldn't track role instructions, conversation history, and tool state together. Compressing 14KB to about 2KB restored single-turn tool calling, but the end-to-end workflow still collapsed. The limit was model capability.

Switching to a large cloud model

I eventually switched the backend to the cloud-hosted Sonnet model. The stability improvement was obvious. On the same multi-agent tasks, the 4B model regularly failed halfway through. Sonnet could finish the entire workflow while keeping roles consistent, calling tools correctly, and maintaining stable output formats.

The cost went up, of course. But the local result was effectively unusable. Between "free and unusable" and "paid and useful," the decision was easy.

This was where I stopped trying to rescue the 4B setup. A working deployment and one successful turn meant little when the full process couldn't finish reliably.

Lessons

The attempt gave me a much clearer boundary for small models.

Good fits for a 4B model:

• Simple, structured, single-turn tasks such as classification, extraction, and format conversion
• A very short system prompt with simple inputs and outputs
• Workloads that tolerate errors and allow retries

Bad fits for a 4B model:

• Multi-turn conversations that need long context
• Coordination across multiple tools with ordering dependencies
• Role-playing agent systems
• Outputs with strict formatting requirements, including JSON schema

The workload shape matters. A bounded transformation is short and retryable. A multi-step loop accumulates roles, facts, tool state, and sequencing—and more chances to drift.

Advice for anyone trying to save money with a local model

If you're considering a local small model as a cloud API replacement, make four checks first.

1. Define the real task complexity first. A local small model can handle text classification or simple extraction, and the savings are real. In an agent system, token savings can disappear into debugging time.

2. Measure the system prompt. If it exceeds 2KB, a small model will probably struggle. Measure before choosing the model; prompt size is part of the workload.

3. Test the real scenario. A small model may look respectable on a benchmark, but benchmarks are usually single-turn and short-context. Run your actual prompt at your actual conversation length, with the real tools, schemas, and dependent steps. The result may look completely different.

4. Consider a hybrid architecture. Route simple tasks to a local small model and complex tasks to a large cloud model. The routing logic adds engineering work, but over time it can balance cost and capability better than forcing every task through one backend.

The relationship between parameter count and practical usability is much steeper than benchmark leaderboards imply. In a real engineering workflow, "slightly worse" often means "does not work at all."