lower token variance
Same task, real repo: with Cardumen it costs about 13k tokens each time; without it, one run jumps to 38.7k.
Evidence
Two ways to win. Both measured.
Cardumen does not ask you to trust a perfect demo. The benchmark holds the repo, model and task constant and makes the harness the variable.
tokens on a local model
qwen3.5-9b exploring a memory-rich repo: 16 down to 8 exploration steps with Cardumen's code graph.
SWE-bench reference
A public reference that shows scaffolding, not only the model, can change the outcome.
Pillar 1 · Frontier models
Predictable savings and precision.
Cross-service audit of the advanced signature flow a real NestJS monorepo · 11 microservices · NATS · frontier model · n=3 per arm. The same complex question, six times: with Cardumen the cost stays in a narrow band; without it, one run jumps.
with Cardumen — 3 grouped runswithout Cardumen — dispersed runsworst case · runaway
30×lower token variance
2.8×better worst case
−36%average tokens
−26%average time
Precision: It found the real flow bug (idempotency latch) with 6 of 6 file:line citations verified against the code. Precision without hallucination. Cardumen narrows cost and removes the worst case.
Claude · frontier · n=6 · variance11× cost variance3.8× token variance4.9× time variance
Pillar 2 · Local models
More quality, fewer hallucinations.
The local model runs privately and performs better: it explores less with the code graph, and the harness cuts hallucinations that appear without verified context.
Memory-rich repo. The cost saving shines on local models: they explore inefficiently without reliable context.
- Verify actually runs: the model cannot declare done without running the suite and closing against evidence.
- Anti-phantom-citation guard: flags when the model claims to have read a file it never opened.
- Project memory as ground truth: the model cannot rewrite the repo rule to fit its assumption.
In a task with a seeded business rule, a model without Cardumen context rewrote the project convention and declared all tests passed (false). With Cardumen it applied the correct rule and closed against evidence.
Methodology
How to test a promise without inflating it.
Same repo, same task
Cardumen on/off over the real repo: same model, same prompt, only the harness changes.
External judge
The verdict comes from tests, build or an independent adversarial reviewer against the code, not the agent that wrote the answer.
Full cost
Measure tokens, time, variance and worst case, not only the average or monthly price.
Declared limit
Record where the harness does not help, because that edge defines the right routing.
Honest limits
What Cardumen does not promise.
- Cardumen does not make a model intelligent when it does not understand the base problem.
- With frontier models, the value is savings and predictability; the quality jump shows most on local or less capable models.
- Results depend on the repo, existing tests, memory quality and team discipline.
- Small n (3 to 6): directional signal, not statistical closure. Reproduce the metrics on your repo before turning them into a promise.
Next step
Request a benchmark for your kind of repo.
Bring one real task. We compare the same work with and without Cardumen and return the result, including the limits.