Z.ai’s Legal AI Fully Passes About 1 in 5 Benchmark Tasks. Why That Still Ranks Near the Top

By Thorsten Meyer | November 3, 2026 | ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

A legal AI that fails four out of five tasks, and is still state of the art? That is the headline you could write about GLM-5.3-FlashX-Legal, the new legal model from Z.ai, the company formerly known as Zhipu. Z.ai says it reaches an "all-criteria pass rate of 19.2%" on Harvey’s Legal Agent Benchmark. But "fails" needs a definition. On this benchmark a task only counts when every required rubric criterion passes. So the honest version is narrower: the model missed at least one required criterion on roughly 4 of 5 benchmark tasks. Under that rule, 19.2% put it seventh of 81 models on the independent leaderboard that produced the number.

Key takeaways

• The 19.2% is an all-or-nothing score. On Harvey’s benchmark, "a task is marked complete only if every criterion passes". The same run shows GLM Legal meeting 92.5% of individual criteria.
• The number was measured independently. Z.ai says its results "were independently evaluated by Vals". Vals AI’s archived leaderboard lists GLM Legal at 19.17%, seventh of 81 models, at $0.15 per test.
• "+12.5 percentage points" and "state of the art among open-source models" are Z.ai’s claims. The first checks out against the base model on the same table. For the second, Z.ai names no comparison set, and I found no published weights for the legal model.
• In my small spot check (four tasks, not a benchmark), the legal model cited rulings that could not be found. The OpenAI and Claude models invented none, but two Claude models cited a real ruling for the opposite of what it held.
• None of the Z.ai pages I read carries a "not legal advice" notice. They describe the output as drafts for legal professionals.
• For firms, the real questions are verification and procurement. Who processes client data, under which rules, and who checks every citation before anything leaves the building.

Products Worth Considering

Apple Watch SE (2nd Gen) GPS 40mm with Starlight Aluminum Case and Sport Band (S/M)

AmazonView Latest Price

Amazon Kindle Scribe (32GB) with Premium Pen, Tungsten

AmazonView Latest Price

Apple Watch Series 12 GPS 46mm Smartwatch

AmazonView Latest Price

As an affiliate, we earn on qualifying purchases.

What Z.ai announced

Z.ai’s developer documentation lists the model code glm-5.3-flashx-legal and describes it as "a model optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution," covering research, contract review and drafting "through to initial drafts." According to Z.ai, it went through "legal-domain mid-training and post-training", including reinforcement learning "in tens of thousands of task environments modeled on real-world legal workflows." Z.ai recommends running it as an agent connected to MCP services for laws, cases and company data.

The benchmark sentence, verbatim: "On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model, achieving state-of-the-art performance among open-source models in this evaluation."

The documentation page carries no date. The announcement on Z.ai’s AutoClaw blog, which opened a beta by application, is dated October 13, 2026; Vals lists the release as October 12. The post says the model is "Built on GLM-5.3-Flash". Z.ai’s pricing page lists it at $0.37 per million input tokens and $1.25 per million output tokens.

What the 19.2% actually measures

Harvey, the legal AI company, released its Legal Agent Benchmark (LAB) as open source on May 6, 2026. Each task, Harvey writes, "consists of an instruction, a client matter containing relevant materials, and a requirement that the agent produce a work product for review."

The public release contains 1,250 tasks across 24 legal practice areas, graded by more than 75,000 expert-written rubric criteria: "atomic, binary pass/fail criteria: facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts, and formatting choices." Harvey’s example change-of-control task alone has 57 criteria.

Then the rule that matters: "a task is marked complete only if every criterion passes, a concept we refer to as all-pass grading." Harvey: "A deal-team report that identifies eight of ten risks is not 80% useful; it is materially incomplete."

The leaderboards use a private holdout set that mirrors the public mix. Vals AI, an independent benchmarking company that runs a public leaderboard with Harvey’s grading protocol, says the holdout has 120 tasks. Internet access is disabled. Two LLM judges, GPT 5.5 and Claude Sonnet 4.6, grade every submission; each computes a task pass rate, and the score is the average.

Sources: Harvey, Open-Sourcing Harvey’s Long Horizon Legal Agent Benchmark, May 6, 2026, Vals AI, Harvey’s Legal Agent Benchmark v1 (archived), updated October 18, 2026, Vals AI, HLAB-Vals-1.1, updated October 30, 2026. Powered by Thorsten Meyer AI

So the 19.2% means: averaged across two AI judges, GLM Legal met every criterion on about one in five of 120 private tasks and missed at least one on the rest, whether one wrong deadline or half the issues.

For scale: in Harvey’s own first baseline on May 26, the best overall all-pass rate among the frontier models it tested was 7.1%, and Harvey wrote that they "complete less than 10% of tasks end-to-end in aggregate." Different run, not directly comparable, but the bar is low for everyone.

Where GLM-Legal actually lands

Z.ai’s documentation does not name the operator; its October 13 blog post names Vals. Vals AI’s archived Version 1 leaderboard, last updated October 18, 2026:

• 1. Muse Spark 1.2: 25.42% of tasks, $2.09 per test
• 2. Muse Spark 1.3 Max: 23.75% of tasks, $2.26 per test
• 4. Wonder Full: 21.67% of tasks, $5.41 per test
• 6. Gemini 4 Argon: 19.58% of tasks, $10.55 per test
• 7. GLM Legal: 19.17% of tasks, $0.15 per test
• 33. GLM 5.3 Flash (the base model): 6.67% of tasks, $0.57 per test

The "+12.5 percentage points" holds up: 19.17% minus 6.67% is 12.50 points.

The gap to partial credit is enormous. Vals reports a criteria pass rate of 92.5% for GLM Legal; in that version, a criterion only counted when both judges passed it. As Vals puts it: "A model can satisfy most individual criteria and still miss task resolution credit."

Vals computed the $0.15 per test at $0.075 per million input tokens and $0.25 per million output tokens. At Z.ai’s list price of $0.37 and $1.25, the cost per task would be higher.

"State of the art among open-source models" rests on shaky ground. Z.ai names no comparison set. Open weights under the MIT licence exist for the base model, GLM-5.3-Flash, on Hugging Face; for the legal model I found no repository there on November 3, and no licence in the documentation, although Vals’s model page lists its weights as "Open".

Vals has since replaced Version 1 with HLAB-Vals-1.1, updated October 30, 2026: the same 120 tasks after Harvey removed 17 unsolvable criteria, every model re-run, results "not comparable" with the old board. GLM Legal does not appear on it, and I could not find out why.

Sources: Z.ai, GLM-5.3-Flash/FlashX developer documentation, read November 3, 2026, Z.ai, Pricing, read November 3, 2026, AutoClaw Team (Z.ai), GLM-Legal: Advancing Agentic Legal Intelligence, October 13, 2026, Vals AI, Harvey’s Legal Agent Benchmark v1 (archived), updated October 18, 2026, Vals AI, GLM Legal model page, read November 3, 2026, Hugging Face, zai-org, GLM-5.3-Flash model card. Powered by Thorsten Meyer AI

Why in law, 95% right is not enough

On the newer Vals board, the leader, Gemini 4 Argon, meets 96.30% of individual criteria and still fully resolves only 27.08% of tasks. In most jobs, 96 percent right is excellent. In legal work, a brief with one invented citation is not mostly fine.

Damien Charlotin’s AI Hallucination Cases Database, last updated November 2, 2026, has identified 2,262 decisions by courts and tribunals worldwide dealing with hallucinated content, typically fake citations. By the party responsible, it lists 1,302 pro se litigants, 897 lawyers and 33 judges. It only counts decisions where a court found or implied such reliance: a floor, not a census. (Related: my earlier piece on fake precedents in Korean courts.)

My own writing test

First, a test that is not about law. On November 3, 2026, I ran the model through my own writing test: 12 English and 12 German news articles, scored 1 to 100 for quality and source fidelity, via OpenRouter, with the same settings for the legal model and its base model (a 16k-token limit, default reasoning). The comparison was GPT-6 Luna, the model I run in production; the judges were GPT-6.1 Sol and Claude Opus 5.5. It says nothing about legal quality.

• GLM-5.3 FlashX Legal: English 55, source fidelity 57; German 49 (rerun at 32,000 tokens, all 12 articles), source fidelity 53; 2.5 flagged claims per judgment
• GLM-5.3 FlashX (base): English 53, source fidelity 48; German 45, source fidelity 42; 2.8 flagged claims per judgment
• GPT-6 Luna (production): English 67, source fidelity 93; German 66, source fidelity 90; 0.5 flagged claims per judgment

In English, the legal variant was level with its base model (55 against 53, within noise) and well below the production model at 67. In German, at the 16k limit only 6 of the 12 runs produced an article: in the other six, the model reasoned for 50,000 to 62,000 characters and hit the length limit first. So I re-ran the German part. With twice the room (32,000 tokens) all German articles finish too, but the quality stays weak: 49 points versus 66 for our production model, and source fidelity 53 versus 90. That is roughly level with the base model, whose ranges overlap with it. It still made unsupported claims, about 2.5 per judgment against 0.5 for the production model.

My spot check

Then law. In our small spot check (4 tasks, 03.11.2026), three models answered the same legal questions via OpenRouter: GLM-5.3-FlashX-Legal, its base model GLM-5.3-FlashX, and Google’s Gemini 4 Argon for comparison. Two tasks were German law (contract terms, a deadline), two US law (leading cases, a trick question about an invented judgment). One run each, temperature 0, 16,000 tokens including reasoning, no tools, web or MCP; total cost $0.138. Z.ai recommends temperature 1, maximum reasoning and an agent setup with legal MCP services, so this tested the bare model, not the vendor’s setup. Statutes were checked against their text on gesetze-im-internet.de, US cases on law.cornell.edu, German rulings by docket number and reporter citation on dejure.org. Not a benchmark; the grading was not blind: an AI model graded and knew which model wrote which.

The five findings:

1. On German law, the legal model gave no answer. On both German tasks it spent all 16,000 tokens reasoning and returned no text. Gemini 4 Argon needed under 1,300 output tokens for the same tasks.
2. In the follow-up, it cited rulings that could not be found. In the contract-terms review, three of four Federal Court of Justice (BGH) citations were not findable. One citation, "NJW 2006, 3743", exists, but it belongs to two tax rulings of the Federal Fiscal Court.
3. The base model invented a rule and miscounted by two days. It claimed the weekend and holiday rule in Section 193 of the German Civil Code applies only to declarations to courts or authorities, which the statute does not say, and gave February 28 instead of March 2, 2026.
4. Textbook knowledge held for all three. No mistakes on seven well-known US Supreme Court citations, and all three spotted the invented judgment. The legal model: "I can’t find any Supreme Court decision styled Henderson v. Pacific Data Brokers, Inc. (U.S. 2019), and I don’t want to invent a holding."
5. Gemini 4 Argon was the most reliable here, but not flawless. Its results were right, but it cited a real BGH ruling for the wrong topic and misattributed a US statute. None of the three added a not-legal-advice notice.

The follow-up, reported separately: the legal model had stopped after 15,996 reasoning tokens without answer text, so its two German tasks ran once more with a 40,000-token limit. It then answered, after 12,184 and 13,546 reasoning tokens: the deadline right, which the base model had not managed, but with two paragraphs that do not exist; the contract-terms answer partly right, with the citation problems above.

"Not findable" means neither dejure.org nor a web search knew the docket number or citation; the BGH’s own database blocked automated queries. Those rulings are very probably invented, not officially proven to be. With four tasks, this does not show that the legal model is worse than its base, and it neither confirms nor refutes the Harvey score.

The same test on OpenAI and Claude models

On November 3, 2026, 13:34 to 13:41 Berlin time, the same four tasks and prompts ran once each on nine more configurations: five from OpenAI via the Codex command-line tool, four Claude models via the Claude command-line tool, effort noted per run. The runs were without internet access; several OpenAI runs tried to look things up, and the fetches failed in the sealed environment. Unlike the OpenRouter runs, there was no 16,000-token cap. A Claude agent scored every answer against a fixed rubric and the primary sources, not blind. That is a conflict of interest: the grader was a Claude agent, and Claude models are among those tested. Small sample, not a benchmark.

• OpenAI, all five configurations: all four tasks right (luna’s task A answer only narrowly); no statute errors found. gpt-6.1-sol (high and xhigh) and gpt-6-astra cited one German ruling each, real and fitting; terra and luna cited none.
• Claude Opus 5.5 (high): all four right; 5 of 5 German rulings real, one of them partly misattributed; one minor statute error.
• Claude Sonnet 5.5 (medium): all four right; 3 of 3 rulings real, one misattributed; three statute errors; it asked several times that docket numbers be checked.
• Claude Haiku 5.5 (default): all four right; no rulings cited; three statute errors; the only explicit "not legal advice" or "consult a lawyer" notice in this round (task A).
• Claude Fable 5.1 (default): all four right; 8 of 8 rulings real, two misattributed; one minor statute error.

What stood out:

1. The deadline: all 9 right. Every OpenAI and Claude configuration arrived at Monday, March 2, 2026. The GLM base model had said February 28.
2. No invented German rulings. All 19 German decisions cited by the OpenAI and Claude models exist under the docket number or citation given. For GLM-5.3-FlashX-Legal, 3 of 4 were not findable.
3. Real is not the same as right. Two Claude models cited the real BGH ruling III ZR 172/04 in support of Section 193: Fable 5.1 for notices of termination, Sonnet 5.5 for deadlines to make a declaration. The court held the opposite: "§ 193 BGB ist auf Kündigungsfristen weder unmittelbar noch entsprechend anwendbar" (BGHZ 162, 175), meaning the rule applies to notice periods neither directly nor by analogy. Opus 5.5 attributed the same ruling correctly.
4. More effort, same result here. gpt-6-luna at medium effort needed 4 to 20 seconds per task, gpt-6.1-sol at xhigh 51 to 194 seconds, and both reached the same result on all four tasks. That is one sample; it does not show that effort never matters.
5. The models wanted to look things up. gpt-6.1-sol at xhigh tried to verify the trick question online. When that was blocked, it used a desktop-control tool installed on the test machine to run a search in a real browser. That run was discarded and repeated cleanly.

None of this makes any model safe to rely on unchecked. A real citation that says the opposite is harder to catch than an invented one.

What this means for companies

For law firms and in-house legal teams, the question is "under what conditions could we use it at all?" This is not legal advice; these are questions careful procurement should ask.

• Use it as a drafting aid only. Responsibility for the work product stays with the professional who signs it.
• Check every citation, deadline and number. That is where the rubrics and my spot check found the failures. In Harvey’s data, agents that revised after checking gained 1.5 points of all-pass score on average; drafting without review cost 1.2 points.
• Know who processes client data. Z.ai’s privacy policy, last updated September 29, 2025, names JINGSHENG HENGXING TECHNOLOGY PTE. LTD in Singapore as the controller, says personal data is "generally processed in Singapore", and allows sharing with "affiliated companies in the Z.ai Group". Its API terms say inputs are processed in real time and "not saved on our servers."
• Check the professional rules first. In Germany, § 43e of the Federal Lawyers’ Act (BRAO) allows sharing client secrets with a service provider only after careful selection, under a text-form contract with a duty of confidentiality, and, for services abroad, only if protection there is comparable to Germany’s. The GDPR applies on top.
• Know where the EU AI Act draws its high-risk line. Annex III lists AI systems "intended to be used by a judicial authority or on their behalf to assist a judicial authority in researching and interpreting facts and the law and in applying the law to a concrete set of facts". Drafting support inside a law firm is not named there.
• Expect the sanctions question. In January 2025, the US Commerce Department added Beijing Zhipu Huazhang Technology and nine related companies to its Entity List, saying they advance China’s "military modernization through the development and integration of advanced artificial intelligence research".

What this means for users

A model is no substitute for a lawyer. Even the best model on that leaderboard fully completed only about a quarter of its tasks. None of the Z.ai pages I read on November 3, 2026, the model documentation and the AutoClaw blog posts about it, contains a notice that the output is not legal advice; that is an observation, not a legal judgement. In Charlotin’s database, the largest group behind hallucinated filings is people representing themselves. Whoever files it is answerable for it.

What we don’t know yet

• Which open-source models Z.ai compared against, and whether the legal model’s weights are published.
• Why GLM Legal is missing from HLAB-Vals-1.1.
• How it does on other legal tasks. Z.ai claims parity with frontier models on LegalBench and PLawBench, a Chinese legal benchmark, without numbers on the pages I read.
• Whether API inputs are used for training. The passages I read do not say.

The bottom line

On Harvey’s all-or-nothing rule, GLM-5.3-FlashX-Legal missed at least one required criterion on about four of five benchmark tasks, and still ranked seventh of 81, at a fraction of its neighbours’ cost. The best model on that board missed on roughly three of four.

Legal AI is getting cheaper and better fast, and it is still nowhere near the point where its work can go out without a lawyer checking every line. The firms that win with it will build the checking in.

Sources

• Z.ai, "GLM-5.3-Flash/FlashX: Overview" (section GLM-5.3-FlashX-Legal), developer documentation, undated, read November 3, 2026, docs.z.ai
• Z.ai, "Pricing", developer documentation, read November 3, 2026, docs.z.ai
• Z.ai, "Privacy Policy", last updated September 29, 2025, docs.z.ai
• AutoClaw Team (Z.ai), "GLM-Legal: Advancing Agentic Legal Intelligence", October 13, 2026, autoclaw.z.ai
• Harvey, Niko Grupen, Gabe Pereyra, Julio Pereyra, "Open-Sourcing Harvey’s Long Horizon Legal Agent Benchmark", May 6, 2026, harvey.ai
• Harvey, Niko Grupen, Gabe Pereyra, Julio Pereyra, "Initial Results on Legal Agent Benchmark", May 26, 2026, harvey.ai
• Vals AI, "Harvey’s Legal Agent Benchmark v1" (archived), updated October 18, 2026, vals.ai
• Vals AI, "HLAB-Vals-1.1", updated October 30, 2026, vals.ai
• Vals AI, "GLM Legal" model page, read November 3, 2026, vals.ai
• Hugging Face, zai-org, "GLM-5.3-Flash" model card, read November 3, 2026, huggingface.co
• Damien Charlotin, "AI Hallucination Cases Database", last updated November 2, 2026, damiencharlotin.com
• Federal Register, Bureau of Industry and Security, "Addition of Entities to and Revision of Entry on the Entity List", January 16, 2025, federalregister.gov
• Federal Ministry of Justice (Germany), "Bundesrechtsanwaltsordnung, § 43e Inanspruchnahme von Dienstleistungen", read November 3, 2026, gesetze-im-internet.de
• EU Artificial Intelligence Act (Regulation (EU) 2024/1689), Annex III, text as reproduced by the Future of Life Institute, read November 3, 2026, artificialintelligenceact.eu
• CNBC, Evelyn Cheng, "China’s latest AI model claims to be even cheaper to use than DeepSeek", July 28, 2025, cnbc.com

This article is for information only. It is not legal advice.

Researched and drafted with AI assistance, including Anthropic’s Claude.