By Thorsten Meyer | November 5, 2026 | ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
This is the third experiment in a week in which AI models went into the numbers of another AI model. The first edit worked technically and changed nothing where it counted: Claude made our local model finish more legal answers, and the legal score stayed at zero. The second AI, OpenAI’s GPT-6 Astra, then reviewed that work and found real faults. Now the reviewer ran its own experiment. And this time something measurable got better.
Not much. One matrix out of 398, one very narrow task, and a test that was frozen before anyone saw the result. The edited model got 70 of 120 test cases exactly right instead of 49. That is +17.5 percentage points, with 21 cases fixed and none broken. It is also a result with clear edges, and the edges are half the story.
This article describes an internal AI experiment. It is not legal advice.
Key takeaways
- Who did it: OpenAI’s GPT-6 Astra, working as a Codex agent, ran the whole experiment on its own: it read the weights, chose the edit, froze the tests and measured the result. Claude (Anthropic) was not involved in this experiment; it drafted this article.
- The model and the task: Qwen3-4B-Instruct-2507, a small open model (Apache-2.0). The task: copy short texts that contain tabs and other special characters into JSON without losing a single character. A real tab must come out as
\tand decode back to exactly the source text. - Prompting did not fix it: the original model solved 16 of 32 screening cases. A stronger, more explicit escaping prompt also scored 16 of 32.
- The edit: a calculated rank-one update to one matrix, the output projection of the model’s last MLP layer. No gradient training. The other 397 stored tensors are bit-identical to the original.
- The number to remember: on the frozen final test, 49 of 120 → 70 of 120 exact answers, +17.5 percentage points, 21 wins, 0 losses.
- What did not improve: extracting a table row stayed at 0 of 30. Fifty of 120 final answers are still wrong. The cases are synthetic, and the work has not yet been checked by a second AI.
Products Worth Considering
Apple MacBook Air 15-inch with M5 chip, 24GB RAM, 512GB SSD, Midnight
Apple MacBook Air 13.6-inch with M5 Chip, 16GB Memory, 512GB SSD, Sky Blue
Sony WH-1000XM4C (M4 2nd Gen) Wireless Noise-Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
Episode three of a week of weight editing
The series started with an outside idea. MIT CSAIL researchers described how an AI can read, repair and write other neural networks directly from their numbers. We explained that in our overview of weight-space programming, and then tried it at home.
Experiment 1, Claude edits Ornith. A Claude Opus 5.5 session found a “loop direction” inside our local model Ornith and weakened it in the weights. The model finished 12 of 15 legal answers instead of 4. The legal score stayed at 0. The full story of that edit.
The review, GPT-6 Astra checks Claude. GPT-6 Astra did not build anything in that round. It checked the work, found a resume bug, a broken command line and side effects of the weight conversion, and made us take back several sentences. What the second AI found.
Experiment 2, GPT-6 Astra’s first own attempt. Before this one, GPT-6 Astra tried its own edit on the same Qwen model: pulling five fields, such as an email address or a due date, out of synthetic business messages. It scaled down one attention head and saw early gains, but on the frozen final test both versions scored 97 of 200. No improvement. A clearer prompt did better on the validation cases than the edit.
That negative result left one useful clue. In experiment 2, both versions produced the same 47 invalid answers, mostly because they copied real tab characters into JSON strings where an escaped \t belongs. In the report’s own words, the starting hint for experiment 3 “came from the previous attempt”. This was not a blind search for every weakness in the model. It was a targeted look at one known failure.
Who did what
The owner of this site gave the order on 5 November: inspect the Qwen model directly, find a weakness, and try to improve it. The experiment report labels the work “Codex’s own experiment”. We checked which model that was: the Codex session that wrote the experiment code and the report ran GPT-6 Astra throughout, with its reasoning effort set to extra high.
- GPT-6 Astra (OpenAI), as a Codex agent: planned the experiment, inspected the weights, built the test cases, computed and saved the edit, froze the tests and wrote the result report, including the result graphic shown above.
- Claude (Anthropic): ran experiment 1 a day earlier. It had no part in experiment 3. A Claude agent researched and drafted this article from the experiment records and checked every number against the stored results.
- What is still missing: an independent review. In experiment 1, the second AI found real faults. Experiment 3 has not had that check yet.
The task: carry text into JSON without losing a character
JSON is the format most software uses to pass data around: web services, logs, forms, the structured output of AI tools. Text inside JSON has rules. A tab or a line break cannot sit in a JSON string as it is. It has to be written as an escape sequence, a backslash followed by a letter: a tab becomes \t, a line break becomes \n.
The task in this experiment is deliberately boring. The model gets a short text that contains tabs, line breaks, quotes or backslashes, and must return it inside a JSON object. The answer only counts if a normal JSON parser decodes the field back to exactly the original text. Valid JSON alone is not enough. Dropping a tab, turning it into spaces or copying the raw tab into the string all count as wrong.
It sounds trivial. It is exactly the kind of error that breaks data pipelines quietly: an invoice line with a tab, a log entry, a pasted table cell, and suddenly the next program cannot read the output. The test cases are our own synthetic texts. They are not legal tasks and not customer data, and we do not publish them.
Step 1: reading the weights
GPT-6 Astra started with the stored numbers, not with prompts. It read the headers of all 398 tensors in the model file. It loaded the complete embedding table, 151,936 token vectors with 2,560 numbers each, and 44 projection matrices: the output projections of all 36 attention layers and of the last eight MLP layers.
The interesting pair was two tokens. One is the raw tab character. The other is the two characters backslash and t, the escaped form. In the model’s vector space the two point in quite different directions: their cosine similarity is about 0.1. The difference between them suggested a direction that could push the model from “write the raw tab” toward “write the escape”. The strongest alignment with that direction showed up in the last two MLP layers.
The report is careful here, and so are we. Weight geometry gives a hypothesis, not a diagnosis. The model uses the same token vectors for reading and for writing, so simply suppressing the raw tab token everywhere could also damage how the model reads input that contains tabs. That rules out the crude fix.
Step 2: confirming the failure, and that a better prompt is not enough
Before touching a weight, the failure had to be real under a fair instruction. In 32 new screening cases with an explicit instruction to escape special characters, the original model produced 16 exactly correct answers. A stronger prompt, with the character codes spelled out and an example, also produced 16 of 32.
That matters for the whole story. In experiment 2, a better prompt beat the weight edit. Here, prompting alone did not move the number at all. That is exactly what makes a weight edit worth trying.
Step 3: the edit, one matrix and no training
The edit has two parts.
- A direction, u. Computed from the difference between the raw-tab and escaped-tab token vectors and scaled by the model’s final normalisation weights. In four diagnostic examples, temporarily adding this direction just before the first tab decision shifted the next token to the escape. That alone proves nothing about free answers, but it showed the direction does something.
- A context selector, v. The edit should only push toward the escape when the model is about to write a tab inside a JSON string, not everywhere. GPT-6 Astra collected 728 internal states from 40 development cases, 48 of them right before a desired escape token, and fitted v with a regularised least-squares calculation.
The two combine into one change to one matrix, the output projection of the last MLP layer:
W_new = BF16(W_old + 64 · u · vᵀ)
In words: take the old matrix, add a correction of strength 64 that is built from one output direction and one input pattern, and store the result in the model’s usual 16-bit number format (BF16). This is a “rank-one” update, the simplest kind of change a matrix can get. In use, the correction only becomes large when the model’s internal state matches the pattern in v.
The size of the change, measured on the stored file:
- The matrix has 2,560 × 9,728, about 24.9 million values. After rounding to BF16, 11.3 million of them differ from the original. Most moved by tiny amounts.
- The relative change of that matrix, measured with the Frobenius norm, is 1.11%. That is a measure of how much this one matrix moved, not a percentage of the whole model.
- The other 397 tensors are bit-identical to the original. Tokenizer, configuration, license and model card match byte for byte.
This is not gradient training and not fine-tuning. It is also not “just reading the weights”: v was estimated from examples, so the report calls it data-dependent analytical weight optimisation. The edited copy is internal and is not published.
Step 4: a test the edit could not see coming
A development score that jumps from 24 to 40 of 40 sounds great, and means little: the same 40 cases were used to fit the edit. What makes this result worth an article is the discipline around it.
- Frozen first. On 5 November at 13:55 UTC, the validation set (40 cases), the final test (120 cases) and the controls (40 cases) were frozen with checksums. At 14:02 UTC, one candidate was chosen, before any independent output of the edited model existed. Other prepared strengths were never run. Nothing was tuned after the final test.
- Different material. The final test uses templates, character combinations and values that differ from the development cases.
- The saved file, nothing else. The final answers came from the saved, freshly reloaded model copy, with no diagnostic hooks, no grammar constraints that force valid JSON, no repair parser and no post-processing.
- Same conditions for both. Original and edit used the same prompt, tokenizer, number format, temperature 0, seed and output limit. All final answers ended normally. Each 120-case run took about 51 seconds on one of our machines.
- Pre-declared criteria. At least +10 percentage points, a paired 95% confidence interval above zero, at most 5 points lost on the controls, and no loss on plain-text tabs.
The results
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
| Check | Original | Edit | What it means |
|---|---|---|---|
| Development | 24/40 | 40/40 | Same cases shaped the fit; not independent proof |
| Validation (new) | 10/40 | 19/40 | 9 wins, 0 losses |
| Frozen final test | 49/120 | 70/120 | +17.5 points, 21 wins, 0 losses |
| Controls | 35/40 | 35/40 | No measured loss |
| Plain-text tabs (part of the controls) | 10/10 | 10/10 | Real tabs still survive where they belong |
| Random orthogonal direction (validation) | 10/40 | 10/40 | Same strength, wrong direction: no gain |
In percent, the final test moved from 40.8% to 58.3% exact answers. The paired 95% bootstrap interval for the gain runs from +10.83 to +25.00 percentage points, from 10,000 resamples of the 120 cases. All pre-declared criteria were met.
The random-direction control is the part we like most. It uses the same context selector and the same strength, but a random direction at right angles to the real one. It gained nothing. So the gain does not come from “any change of this size to that matrix”. It comes from this direction.
The five control failures are the same for both models: correctly sorted lists with extra spaces where a compact format was asked for. The unchanged export control, a copy saved through the same pipeline with no edit, matches the original in all 398 tensors, and five probe answers came out identical down to the token count.
Where it worked, and where it did not
| Final test group | Original | Edit |
|---|---|---|
| Texts with tabs | 19/30 | 30/30 |
| Line breaks | 30/30 | 30/30 |
| Mixed special characters | 0/30 | 10/30 |
| Extract the second table row | 0/30 | 0/30 |
The tab group went to a perfect score. Line breaks were already perfect and stayed that way. Mixed special characters went from nothing to a third. And the hardest group, pulling a complete table row into JSON, stayed at zero.
There is a second warning in the validation data. The validation set also had a tab group, phrased differently. It stayed at 0 of 10 with the edit. The final tab group reached 30 of 30. Same skill, different wording, opposite outcome. The effect depends on context in ways this experiment does not explain.
The limits
- Synthetic cases. All tasks come from our own generator, with repeated templates. The bootstrap interval measures uncertainty across these cases, not across new kinds of text.
- Context dependence. The validation tab group stayed at 0 of 10 while the final tab group reached 30 of 30.
- Narrow scope. No evidence for real documents, for data extraction in general, or for legal knowledge. Fifty of 120 final answers are still wrong, and table rows still fail completely.
- Small controls. 40 control cases cannot show that every other ability is intact.
- No independent review yet. The same AI planned, built, tested and reported this experiment. A second AI has not checked it. After what the review found in experiment 1, we say that plainly.
Infographic 02 · The series so far
From zero gain to one narrow, measured win
Two AI models from two companies took turns editing, reviewing and editing again. What each step showed.
- 4–5 Nov
Experiment 1Claude edits Ornith for lawA Claude Opus 5.5 session weakened a “loop direction” in a local model. Finished answers 4 → 12 of 15; legal score 0 → 0.Anthropic ○ No gain on the target - 4–5 Nov
ReviewGPT-6 Astra checks Claude’s editReview only, no second model. A resume bug, a broken command line, conversion side effects, and claims we took back.OpenAI → Faults found - 5 Nov
Experiment 2GPT-6 Astra’s first own editQwen, five fields from synthetic business messages, one attention head scaled down. Final test 97 → 97 of 200. A clearer prompt did better on validation.OpenAI ○ No gain on the target - 5 Nov
Experiment 3One matrix, JSON tab escapingThe tab failure seen in experiment 2, targeted with one rank-one edit. Frozen final test 49 → 70 of 120, 21 wins, 0 losses.OpenAI ✓ Measured gain ! Not yet reviewed
Experiments 2 and 3 used synthetic, non-legal test cases. Experiment 1 used our legal benchmark, whose hidden tasks are not published. The edited models are internal and not distributed. Experiment 3 has not yet been reviewed by a second AI.
Sources: our experiment records, 4–5 Nov 2026 (Lexassay research, internal): experiment 1 results report (Claude Opus 5.5), review and explanation by GPT-6 Astra in Codex, experiment 2 and experiment 3 result reports by GPT-6 Astra in Codex.
Powered by Thorsten Meyer AI
What this means for companies
A weight edit can remove one concrete, measurable error. Not “make the model smarter”, but fix a specific failure that you can count. Here: raw tabs where escapes belong.
Try the simple fix first, and measure it. In experiment 2, a clearer prompt beat the edit. In experiment 3, the prompt did nothing, and that is the only reason the edit was worth it. If a better instruction, a JSON library, or constrained output solves your problem, use that.
Freeze the test before you see the result. The development score was 40 of 40. The honest score was 70 of 120. Without a frozen final test, you would have shipped a story instead of a number.
Insist on controls. A random direction of the same size, an unchanged export copy, and checks on things that should not change. Without them, you cannot tell surgery from side effects.
“Better on the test” is not “better in practice”. The cases were synthetic. Before relying on a change like this, test it on your own real data, and have someone other than the author check it.
What this means for users
AI models still make small, boring mistakes with special characters, and those mistakes can break whatever program reads the output. If you use AI to produce structured data, check it with a real parser, not by eye. And when you read that someone “edited a model to make it better”, ask three questions: better at what exactly, measured on which test, and checked by whom.
The bottom line
The first edit fixed loops and not the law. The review fixed what we were allowed to say. The third experiment is the first one with a clean, measured gain: one matrix, one direction, 49 to 70 of 120, no losses, controls intact. It is precise surgery on a narrow task, not a smarter model. And until a second AI has checked it, it is a promising result, not a proven one.
This article is not legal advice. Thorsten Meyer AI runs the Lexassay research behind these experiments; the synthetic JSON cases used here are not legal benchmark tasks and are not published. Experiment 3 was planned, carried out and reported by GPT-6 Astra (OpenAI) working as a Codex agent. Experiment 1 was run by a Claude Opus 5.5 session (Anthropic). This article was drafted by Claude. The edited model weights are internal and not distributed.
Researched and drafted with AI assistance, including Anthropic’s Claude.
Sources
- Our own experiment records, 5 November 2026 (internal, from our Lexassay research): the experiment 3 plan and result report by GPT-6 Astra in Codex, the machine-readable final result, the weight-inspection and export-comparison records, and the frozen candidate selection. Every number in this article was checked against the stored final result.
- Experiment 2 result report, 5 November 2026 (internal): the earlier Qwen extraction edit with a negative final result, 97/200 for both versions.
- Model: Qwen3-4B-Instruct-2507 by the Qwen team (Alibaba), released under the Apache-2.0 license.
