VigilSAR, a defense-ISR software product, has published an LLM benchmark focused on whether language models can be trusted with intelligence-surveillance-reconnaissance work. It measures the reasoning, reporting, and restraint an analyst actually needs—not performance on general trivia.
The evaluation covers 14 models across 300 tasks, scored on 2026-07-17. Aggregate results appear on the public leaderboard, giving tech readers a comparative view without exposing the underlying evaluation material.
The task set is deliberately private so models cannot train on it. A separate private held-out set adds another check, while the published gap between public and held-out scores for each model helps flag memorization.
In the current standings, claude-fable-5 leads with 67.77 in Band A and serves as the pinned reference row. The benchmark emphasizes bands rather than rank because confidence intervals within a band overlap.
The notable new entry is Moonshot’s Kimi K3, debuting at #3 with 64.65 in Band B. That places it ahead of every GPT and Gemini row on the board.
The GPT-5.x family fills Bands C-D, while Gemini rows sit in Bands E-F. One locally runnable open model is scored as “sovereign-deployable”, reflecting that deployment reality is part of the score.
The benchmark starts from a blunt premise: “Vendor claims are not evidence.” Its operators built the evaluation to decide which models get anywhere near their own product and to rank the models they use themselves. They state that they are not paid by any vendor and that “we would rather be measured than believed.”
Its honesty features reinforce that position: bands instead of pseudo-precise ranks, published confidence intervals, published held-out gaps, and a pinned reference row. The board also reports per-model cost-per-correct-answer economics, pairing capability results with practical model economics.
Engineering with Small Language Models: Efficient AI Design, Training, and Deployment for Developers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Cyber Reconnaissance, Surveillance and Defense
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Emerging Science of Machine Learning Benchmarks
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
