By Thorsten Meyer | November 4, 2026 | ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Disclosure: Thorsten Meyer AI runs Lexassay, the test described here. The answers were graded blind by several Claude agents (Anthropic), and a Claude model ranks first. That is a conflict of interest. Lexassay’s method page states it, and every public answer, gold answer and citation check is published so the grades can be checked. Not legal advice: this article explains a test of AI models. For a deadline, a contract or a letter to an authority, ask a lawyer.
Two clients lose a civil case in Germany. Both judgments are served on the same day, Tuesday 27 April 2027. One case was heard in Munich, the other in Berlin. Both want to appeal. When is the deadline?
It sounds like one answer. It is two. The appeal period is one month, so it would end on Thursday 27 May 2027. But 27 May 2027 is Corpus Christi, a public holiday in Bavaria and not in Berlin. So the Munich deadline moves to Friday 28 May, and the Berlin deadline stays on Thursday 27 May. The deadline for the written grounds of appeal is Monday 28 June in both cities.
This is one of the fifteen tasks on Lexassay, the legal AI test I run. Most strong models got it right. One model gave Berlin the Friday, a day too late. Another put the deadline for the grounds a full month too late, on 27 July. In German civil procedure, an appeal filed late is rejected as inadmissible. A day can end a case.
So I asked the obvious next question. Every big AI company now sells AI for law. What about the model that was built for it?
Key takeaways
- The model built for law scored 8 out of 100. GLM-5.3 FlashX Legal, “optimized for professional legal workflows” according to its maker Z.ai, fully solved none of the 15 tasks and gave no answer at all on two.
- Europe’s flagship scored 10. Mistral says Mistral Large 4 is “state-of-the-art among open models” on enterprise work “including cybersecurity, finance and law”. In my test it got about half the substance right but lost most of its points to wrong citations.
- Another test ranks the same models almost the other way round. On Vals AI’s legal agent benchmark, where models work with supplied documents, Mistral Large 4 ranks 8th, ahead of Claude Opus 5.5 (21st) and GPT-6 Astra (24th).
- The top is strong, not flawless. Claude Opus 5.5 leads with 79, then GPT-6 Astra (72) and GPT-6.1 Sol (71). Even the leader fully met every criterion on only 8 of 10 public tasks, and thinking harder did not help.
- Citations are where it breaks. Across all published runs, Lexassay confirmed 2,644 citations as real and found 804 that were invented or did not say what the model claimed.
- What to do: use AI as a starting point, check every cited law and ruling in the source, never take a deadline from a chatbot, and ask a lawyer when money, a deadline or an authority is involved.
Products Worth Considering
Amazon Kindle Paperwhite (2024), 16 GB, 7-inch Display, Black
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Bose QuietComfort Wireless Noise Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
How the test works, in plain English
Lexassay gives every model the same fifteen legal questions: ten published and five hidden, so that future models cannot learn the answers from the web. For the hidden tasks only the score is published. The questions cover German, US and EU law: deadline calculations, recent changes to the law, leading court decisions, and a few trick questions built on a false premise, which is a question that assumes something untrue. A good answer spots the false assumption instead of playing along.
The models answer without internet access and without tools. That measures what a model knows by itself, roughly what you get from a plain chatbot that does not search the web. “Effort” is how long a model may think before it answers; different settings are tested separately, so one model can appear several times.
Each answer is graded against a gold answer with three to six weighted criteria, and every citation is checked against official databases. A citation is a reference to a specific law or court decision. It can be real, “misattributed” (the law or ruling exists but does not say what the model claims), “invented” (it does not exist) or not findable. Each misattributed or invented citation costs a quarter of the task’s points. An answer without a “not legal advice” note or a pointer to a lawyer loses 5 percent. The full rules are on the method page.
It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
The model built for law: GLM-5.3 FlashX Legal
Z.ai, the Chinese company formerly known as Zhipu, describes its legal model in its developer documentation as “optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.” It went through extra legal training on top of GLM-5.3-Flash.
On Lexassay it scored 8 out of 100, with 0 on the hidden set. The citation check found 21 real citations, 25 misattributed and 1 invented. On two German tasks, recent changes to the law and repair-cost claims, it spent its whole budget of 64,000 tokens thinking and produced no answer. Its base model, GLM-5.3 FlashX, scored 3, so the legal training helped a little.
Two answers show what that means for a reader. Asked about compensation after a data scraping incident, it called a four-figure sum conceivable, where the German Federal Court of Justice works with around 100 euros for the mere loss of control over your data; a consumer would overestimate the claim. It also treated the court challenge to the EU–US Data Privacy Framework as pending, although the EU General Court dismissed it on 3 September 2025.
To be fair to Z.ai: the model ranked 7th of 81 on the first version of Harvey’s legal agent benchmark, scored independently by Vals AI, as I reported in Z.ai’s Legal AI Fully Passes About 1 in 5 Benchmark Tasks. That piece also had my first spot check with four tasks, where GLM-Legal scored 63. That set was too easy (11 of 21 configurations scored 100), so I built the harder fifteen-task set. The two are not comparable, and 63 is not the current figure. But the early warning sign, citations that could not be found, held up.
The general models: frontier, middle, small
The 42 configurations fall into four rough groups.
- The top (67 to 79): Claude Opus 5.5 at all three effort levels (79, 78, 77), GPT-6 Astra (72 at high, 68 at extra-high), GPT-6.1 Sol (71, 68, 67), Claude Sonnet 5.5 (69, 68, 67).
- The middle (41 to 59): Claude Fable 5.1, GPT-6 Luna at high effort, GPT-6.1 Terra, Gemini 4 Argon, Claude Haiku 5.5, Grok 4.7, Gemini 3.8 Flash.
- The weak group (0 to 35): DeepSeek, Kimi, Qwen, Mistral Large 4, the GLM family, MiniMax and Cohere’s Command A Plus.
- Small local models (1 to 12): run on my own hardware. Two of them ran with a small context window and are marked “not comparable”.
Three findings stand out.
More thinking did not help at the top. Claude Opus 5.5 scored 79 at medium effort, 78 at high and 77 at extra-high, and the extra-high run cost about 3.2 times as much as medium, by Lexassay’s estimate at list price. GPT-6 Astra and GPT-6.1 Sol also scored lower at extra-high than at high. The one big jump came from a small model: GPT-6 Luna went from 25 at low effort to 34 at medium and 56 at high.
Price and quality are loosely linked. GPT-6.1 Sol at high effort scored 71 for an estimated $0.37 per full run; Qwen3.8 Max Prime cost $10.19 and scored 13.
The leader is less far ahead than the headline number suggests. On the five hidden tasks, GPT-6.1 Sol at high effort scored 62 and the overall winner, Opus 5.5 at medium effort, scored 59. The GPT models also never added a “not legal advice” note, which costs 5 percent of a task’s points each time. That explains part of the gap.
And the top models made real mistakes. On the US task about the TAKE IT DOWN Act, the new US law on removing deepfakes, GPT-6.1 Sol at high effort got the content right, but every legal reference it gave for it was invented. Claude Sonnet 5.5 at high effort cited a real German Federal Court of Justice ruling for the opposite of what it decided. All three Opus settings scored only 23 on the task about recent German legal changes, partly because they cited the wrong transition provision.
Infographic 02 · Thinking harder
More thinking did not buy better legal answers at the top
Score and estimated cost per effort setting, same 15 tasks.
| Model | Effort | Score | Cost, full run |
|---|---|---|---|
| Claude Opus 5.5 | medium | 79 | ≈ $1.33 |
| Claude Opus 5.5 | high | 78 | ≈ $1.80 |
| Claude Opus 5.5 | extra-high | 77 | ≈ $4.27 |
| GPT-6 Astra | high | 72 | ≈ $1.47 |
| GPT-6 Astra | extra-high | 68 | ≈ $2.87 |
| GPT-6.1 Sol | medium | 67 | ≈ $0.24 |
| GPT-6.1 Sol | high | 71 | ≈ $0.37 |
| GPT-6.1 Sol | extra-high | 68 | ≈ $0.69 |
| GPT-6 Luna | low | 25 | ≈ $0.005 |
| GPT-6 Luna | medium | 34 | ≈ $0.005 |
| GPT-6 Luna | high | 56 | ≈ $0.019 |
Effort = how long a model may think. Costs are Lexassay estimates from tokens at API list price (≈), full run of 15 tasks. One answer per task, so a few points are noise.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Infographic 03 · Citations
Where the weak models lose their points
Real, misattributed, invented: what the citation check found for seven models.
Bars: share of checked citations on the 10 public tasks (citations without an automatic source are left out). Each misattributed or invented citation costs a quarter of a task’s points.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Sources: Lexassay, release of 4 Nov 2026, Lexassay method.
Powered by Thorsten Meyer AI
Europe’s sovereign bet: Mistral Large 4
Mistral is Europe’s best-known AI company, and it sells sovereignty. “Forged in Europe. Built for AI sovereignty,” says its announcement of Mistral Large 4 on 6 October 2026. The model is offered through “a European deployment that Mistral operates end-to-end, independently of other digital service providers and under European law.” Mistral’s sovereignty page puts it simply: “You control where your data goes and who can use it.”
Mistral also claims legal strength: “On critical enterprise workloads, including cybersecurity, finance and law, we find it to be state-of-the-art among open models,” and “On HarveyAI’s Legal Agent benchmark, ML4 outperforms all open-source models.” The weights followed on 29 October (Hugging Face).
On Lexassay, Mistral Large 4 scored 10 out of 100, with 0 on the hidden set. What I tested: mistral-large-4-0 at its default reasoning setting, once, on 3 November 2026; the model card lists it as “Public Preview”, version 26.10. I make no claim about Mistral’s other models.
Where did the points go? On the ten public tasks it met about half of the content criteria (49.7 percent, my calculation from Lexassay’s published cells), but of 87 checkable citations, 32 were misattributed. On the EU task about the Digital Services Act, it got 89 percent of the content right and still scored 0, because it cited seven provisions that do not say what it claimed.
Some content errors would hurt a small business directly. On recent German legal changes, Mistral gave the wrong threshold for the small-business VAT exemption (50,000 euros for the previous year instead of 25,000) and said incoming invoices must be kept for ten years instead of eight. On the US deepfake law, it invented a private right to sue with $150,000 in damages.
Now the counterweight. On Vals AI’s version of Harvey’s legal agent benchmark (version 1.1, checked 4 November 2026), models work through legal tasks with documents and tools, without the internet. There Mistral Large 4 ranks 8th (15.83 percent) and its preview 7th (17.50 percent), ahead of Claude Opus 5.5 (21st, 9.17 percent) and GPT-6 Astra (24th, 7.08 percent). Gemini 4 Argon is first with 27.08 percent; on Lexassay it scored 55.
Both can be true. Vals measures work with the material in front of the model; Lexassay measures what it knows from memory about German, US and EU law, and whether its references are exact. The lesson for any model: give it the sources and check its references.
Infographic 04 · Two tests
Same models, almost opposite order
Knowledge from memory with exact references, or work with the documents in front of the model: the results differ.
Lexassay
From memory · German, US, EU law · every citation checked · score 0–100 · rank of 42
- #1Claude Opus 5.5medium effort79
- #4GPT-6 Astrahigh effort72
- #5GPT-6.1 Solhigh effort71
- #16Gemini 4 Argondefault55
- #32Mistral Large 4.0default10
Vals AI · Harvey legal agent benchmark 1.1
With supplied documents and tools · no internet · tasks fully passed · rank of 33
- #1Gemini 4 Argon27.08%
- #7Mistral Large 4 Preview17.50%
- #8Mistral Large 415.83%
- #21Claude Opus 5.59.17%
- #24GPT-6 Astra7.08%
- #25GPT-6.1 Sol6.25%
Lexassay rank among 42 configurations including local models. Vals ranks and effort settings as listed by Vals AI.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Sources: Lexassay, release of 4 Nov 2026, Vals AI, HLAB-Vals 1.1 (updated 30 Oct 2026, checked 4 Nov 2026).
Powered by Thorsten Meyer AI
Sovereignty answers where your data goes and who controls the system. It does not answer whether the answer is right, and neither does the EU AI Act. Its rules for general-purpose model providers, in force since 2 August 2025, cover documentation, information for downstream companies, a copyright policy and a training-data summary (Article 53), not correct legal answers. Mistral, Anthropic, OpenAI, Google and Cohere are listed as signatories of the EU’s code of practice; Z.ai is not. That is a compliance signal, not a quality score.
Cohere makes a similar pitch for Command A+, “purpose-built for sovereign critical infrastructure” (Cohere, 20 May 2026), without claiming legal expertise. On Lexassay it scored 0.
Infographic 05 · Claim vs result
What the makers say, and what the test found
Three models sold for law or for regulated, sovereign use.
Z.ai · GLM-5.3 FlashX Legal
“optimized for professional legal workflows”
- Hidden set: 0
- 21 real · 25 misattributed · 1 invented
- No answer on 2 tasks: thinking budget used up
- Base model: 3
Mistral · Mistral Large 4
“including cybersecurity, finance and law, we find it to be state-of-the-art among open models”
- Hidden set: 0
- About half the content criteria met (49.7%)
- 32 of 87 checkable citations misattributed
- Tested: mistral-large-4-0, default, 3 Nov 2026
Cohere · Command A+
“purpose-built for sovereign critical infrastructure”
- No legal claim
- 16 real · 41 misattributed · 8 invented
- Confirmed a false premise on EU platform rules
49.7% is Thorsten Meyer AI’s calculation from Lexassay’s published cells. Quotes from the makers’ own pages.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Sources: Z.ai docs, Mistral, 6 Oct 2026, Cohere, 20 May 2026, Lexassay run 6.
Powered by Thorsten Meyer AI
What I did not test
Anthropic sells Claude for Legal, plugins and connections to legal databases on top of its Claude models (“A lawyer stays in charge of the work and judgment”). OpenAI has announced Astra for Law, built on GPT-6 Astra. I tested neither product. They connect the model to sources; my test measures the model without them.
What the worst case looks like
None of this makes AI useless for legal questions. It means the worst case deserves a plan. Four of them:
1. A missed deadline. At least four of the 42 configurations gave a date that was too late. GPT-6 Luna at low effort mistook Corpus Christi for Ascension Day and gave Berlin the Friday; Claude Fable 5.1 at high effort put the grounds on 27 July instead of 28 June. A German court must reject a late appeal as inadmissible (§ 522(1) ZPO), and you only get a second chance if you missed it “without fault” (§ 233 ZPO). Likelihood: real, even for good models. Protection: never take a deadline from AI alone; confirm it with the court or a lawyer.
Infographic 06 · The deadline trap
Same day, two cities, two deadlines
Why one public holiday changes the appeal deadline, and what some AI models got wrong.
- Tue 27 Apr 2027Both judgments servedMunich and Berlin, same day
- Thu 27 May 2027One month later: Corpus ChristiA public holiday in Bavaria, not in Berlin
- MunichAppeal by Friday 28 MayThe deadline moves to the next working day
- BerlinAppeal by Thursday 27 MayNo holiday, no shift
- Mon 28 Jun 2027Grounds of appeal, both citiesTwo months from service; 27 June is a Sunday
What some models answered
- Most strong modelsAll dates right✓ right
- GPT-6 Luna · low effortTook 27 May for Ascension Day: Berlin on Friday 28 May! 1 day late
- Claude Fable 5.1 · high effortGrounds on Tuesday 27 July instead of Monday 28 June✗ 1 month late
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Sources: Lexassay run 7, § 522 ZPO.
Powered by Thorsten Meyer AI
2. An invented ruling in a court filing. In Mata v. Avianca in 2023, a New York federal judge found that two lawyers and their firm “abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT”, and imposed a $5,000 penalty (opinion and order on sanctions, 22 June 2023). In London in 2025, the High Court wrote that freely available AI tools “are not capable of conducting reliable legal research” and referred lawyers to their regulators (Ayinde v Haringey, 6 June 2025). A researcher’s database lists 2,265 court decisions worldwide dealing with AI hallucinations (as of 4 November 2026), and in South Korea about eight in ten judges handling civil, family and administrative cases had seen false or nonexistent precedents, as I described in Fake Precedents in Korean Courts. Likelihood: high for weak models, lower but real for the best. Protection: open every cited law and ruling in an official source before it goes anywhere.
3. A wrong legal position in a contract or a letter. A small business may get an outdated VAT threshold. A company answering a data protection claim may build on a position the EU Court of Justice has rejected: Mistral wrote that a successful hack regularly proves inadequate security, contrary to the Court’s case law. And a question built on a false premise can get a confident answer that plays along: Cohere’s Command A Plus confirmed the question’s false claim that a platform automatically becomes a “very large online platform” under the EU’s Digital Services Act at 10 million users. The threshold is 45 million, and the European Commission has to designate the platform. Likelihood: high for the weak group. Protection: treat AI answers on your rights and duties as a draft, and have anything that leaves the building checked.
4. Out-of-date law. Law moves faster than training data. On 3 November 2026 an adviser to the EU Court of Justice gave the first opinion on chatbots and press publishers’ rights (my explainer); no model knows that from memory, and GLM-Legal even missed a ruling from September 2025. Protection: for anything recent, use today’s sources.
And one more risk: what you paste in. Check where a provider processes contracts and client files before you use real names.
Infographic 07 · The worst case
What a wrong legal answer can cost, and how to protect yourself
Four realistic worst cases, each with an example from the test.
| Worst case | Seen in the test | What can happen | How likely | Protection |
|---|---|---|---|---|
| Missed deadline | Berlin appeal a day late; grounds a month late (task DE1) | Appeal rejected as inadmissible (§ 522 ZPO) | ! real, even for good models | Count it yourself; confirm with the court or a lawyer |
| Invented ruling in a filing | Right content, invented references; a real ruling cited for its opposite | $5,000 penalty in Mata v. Avianca; regulator referrals in Ayinde (2025) | ✗ high for weak models | Open every cited law and ruling in an official source |
| Wrong position in a contract or letter | Outdated VAT threshold; false premise confirmed; a defence the EU court rejected | Wrong decision, lost claim, extra cost | ✗ high for the weak group | Treat answers as a draft; have outgoing texts checked |
| Out-of-date law | A 2025 ruling missed; nothing after the training data | Advice that was right last year | ! grows with time | Use today's sources for anything recent |
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Sources: Lexassay run 6, Lexassay run 7, Mata v. Avianca, S.D.N.Y., 22 Jun 2023, Ayinde v Haringey [2025] EWHC 1383 (Admin).
Powered by Thorsten Meyer AI
What this means for companies
- Give the model the sources. Tools that supply the relevant documents play to what models do well.
- Check every citation. Make it a rule: no AI-drafted text with a legal reference leaves the company until someone has opened the source.
- Deadlines are human work. Use AI to explain a rule, never to compute the date that counts.
- Don’t buy the label. “Built for law” and “sovereign” are product claims. Test the model on your own questions, in your jurisdiction, before you rely on it.
What this means for you
- A chatbot is a starting point, not a lawyer. It can explain terms and help you prepare questions for one.
- If an answer names a law or a ruling, look it up. If you can’t find it, don’t use it.
- Deadline, money, authority: if any of these three is involved, ask a lawyer or a consumer advice service.
- Watch for confident answers to odd questions. If the question itself might be wrong, a good answer says so.
The limits of this test
- One answer per model and task. Results can change between runs and model versions.
- No tools and no web access. Products that search legal databases may do much better.
- Claude agents grade, and Claude models are graded. Grading is blind and answers are public, but style can give a model away; the Vals results, where Claude ranks far lower, are a useful check.
- Four of ten public tasks are German law, which may disadvantage some models. Mistral and Z.ai were tested in one configuration each.
- Corrections from model makers and readers: contact@lexassay.com.
Disclosure: Thorsten Meyer AI runs Lexassay. The answers were graded blind by several Claude agents (Anthropic), and a Claude model ranks first; that is a conflict of interest, stated on Lexassay’s method page. This article is general information, not legal advice. Researched and drafted with AI assistance, including Anthropic’s Claude.
Sources
- Lexassay, results and leaderboard, release of 4 November 2026: lexassay.com; method and limits: lexassay.com/method; tested models: lexassay.com/models; answers and grades for GLM-5.3 FlashX Legal, Mistral Large 4.0 and others: run 6 of 3 November 2026; deadline errors: run 7 of 3 November 2026
- Z.ai, GLM-5.3-Flash and GLM-5.3-FlashX-Legal documentation: docs.z.ai
- Mistral AI, “Introducing Mistral Large 4”, 6 October 2026: mistral.ai; Mistral Large 4 model card: docs.mistral.ai; “Sovereign AI”: mistral.ai; “Mistral Large 4 (Le Chonk) is now open weight”: mistral.ai
- Hugging Face, “Welcome Le Chonk: Mistral Large 4 on Hugging Face”, 29 October 2026: huggingface.co
- Vals AI, HLAB-Vals 1.1 leaderboard, updated 30 October 2026, checked 4 November 2026: vals.ai
- Cohere, “Cohere Releases Command A+”, 20 May 2026: cohere.com
- Anthropic, Claude for legal teams: claude.com; OpenAI, Astra for Law (announcement page, not tested): openai.com
- Regulation (EU) 2024/1689 (AI Act), Articles 53 and 113: EUR-Lex; European Commission, General-Purpose AI Code of Practice, signatories: digital-strategy.ec.europa.eu
- German Code of Civil Procedure, § 522 and § 233: gesetze-im-internet.de, gesetze-im-internet.de
- Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), S.D.N.Y., Opinion and Order on Sanctions, 22 June 2023: CourtListener (PDF)
- R (Ayinde) v London Borough of Haringey; Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin), 6 June 2025: judiciary.uk (PDF)
- Damien Charlotin, AI Hallucination Cases database, checked 4 November 2026: damiencharlotin.com
- The Korea Herald (Yonhap), “8 in 10 judges experienced AI-generated fake legal precedents: survey”, 2 November 2026: koreaherald.com
