TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Hugging Face has released NeoMME, a family of 260-million- and 800-million-parameter encoders that process text tokens and raw image patches through one bidirectional Transformer. The team reports competitive visual-document retrieval, higher throughput than a named comparison model and large index-storage savings, but the supplied results have not been independently reproduced.
Hugging Face has released NeoMME, a family of 260-million- and 800-million-parameter multimodal encoders built to process text tokens and raw image patches within one bidirectional Transformer. The models are available through Hugging Face Transformers under the Apache 2.0 license, giving developers a new architecture for multilingual visual-document retrieval without a separate pretrained vision tower or causal language model.
NeoMME divides images into non-overlapping 32-by-32-pixel patches and projects them through a small multilayer perceptron before sending them along the same computational path as text. Both model sizes support a 16,384-token context window, dynamic image resolution and a mixture of sliding-window and global attention. Hugging Face says the shared path is intended to simplify pretraining, fine-tuning, parallel processing and serving across the two modalities.
The team trained both models from scratch with a masked discrete-diffusion objective. Text tokens are hidden at varying rates while image patches remain visible, requiring the model to reconstruct missing language. According to Hugging Face, high masking rates reduce the model’s ability to rely on surrounding words and push it toward visible image evidence. Training mixed multilingual text, code, mathematics, natural images and document images, with each model processing about 524 billion packed input tokens, including 290 billion from text-only examples.
For visual-document retrieval, Hugging Face fine-tuned the backbone as NeoMME-Retriever using the page-image method associated with ColPali. It ranks screenshots of document pages directly, preserving tables, charts, layout and typography without requiring a separate optical-character-recognition pipeline. Its dual-head design produces dense embeddings and late-interaction embeddings during one forward pass.
NeoMME: One Encoder for Text, Images and Multilingual Documents
Hugging Face’s NeoMME family sends text tokens and raw image patches through one bidirectional Transformer. The architecture targets faster visual-document retrieval and dramatically smaller indexes—without a separate vision tower or causal language decoder.
Two encoder sizes released with Hugging Face Transformers support.
Dynamic image resolution with sliding-window and global attention.
Permissively licensed checkpoints available for outside evaluation.
Packed input tokens per model
Tokens from text examples
Non-overlapping pixels
Pages per second
Per page after compression
A multimodal-native path, not two systems stitched together
NeoMME is trained from scratch to represent language and vision inside the same encoder. Images become projected patch tokens; text remains tokenized language. Both then travel through a shared bidirectional Transformer.
Raw page patches
Images are divided into 32-by-32-pixel patches and projected through a small multilayer perceptron before entering the backbone.
Multilingual tokens
Language, code and mathematics use the same computational pathway, supporting representation learning across diverse content types.
Bidirectional attention
A mixture of global and sliding-window attention supports long contexts while avoiding an autoregressive generation stack.
Dynamic visual input
Pages can be processed at varying resolutions, preserving layout, typography, tables and charts as visual evidence.
One encoder to operate
The unified design is intended to simplify pretraining, fine-tuning, parallel processing and production serving.
Representations, not prose
NeoMME is designed for embeddings and token-level features rather than generating answers one token at a time.
Visible images help reconstruct hidden language
The models use a masked discrete-diffusion objective. Text is hidden at varying rates while image patches remain visible, encouraging the encoder to recover language from both linguistic context and visual evidence.
Pack mixed data
Multilingual text, code, mathematics, natural images and document pages.
Patch images
Raw images become non-overlapping visual tokens through a small projector.
Mask text
Language tokens are hidden at varying rates; visual patches stay visible.
Encode jointly
Both modalities move through the same bidirectional Transformer.
Recover language
The model reconstructs missing tokens using text and image evidence.
Less opportunity to guess from nearby words
Hugging Face says stronger masking reduces reliance on surrounding text and pushes the model toward information visible in the image.
Mechanism is plausible; impact needs testing
The claimed effect on cross-modal grounding remains a vendor-reported interpretation until independent evaluations isolate it across languages and document types.
How NeoMME changes the visual-search stack
NeoMME-Retriever ranks page screenshots directly using the page-image approach associated with ColPali. A dual-head design emits dense and late-interaction embeddings during one forward pass.
| Capability | NeoMME-Retriever | Conventional vision-language retriever | OCR-first retrieval |
|---|---|---|---|
| Vision processing | ✓ Native patches in shared encoder | ~ Separate pretrained vision tower | ✗ Images usually reduced to extracted text |
| Language pathway | ✓ Bidirectional encoder | ~ Often a causal language model | ✓ Text encoder or search index |
| Page layout retained | ✓ Direct page-image representation | ✓ Typically retained visually | ~ Depends on OCR and parsing fidelity |
| Tables, charts and typography | ✓ Available as visual evidence | ✓ Available as visual evidence | ~ May lose spatial and non-text cues |
| Generative decoding | ✗ Not required | ~ Frequently retained | ✗ Usually not required |
| Operational tradeoff | Many visual tokens must still be indexed efficiently | Potentially heavier components and serving stack | OCR preprocessing and layout reconstruction |
Faster encoding and a much smaller late-interaction index
These figures come from Hugging Face’s supplied comparison. They are promising deployment signals, but they are not independently reproduced measurements.
Page-encoding throughput
Matched 2,048-by-2,048-pixel input on an NVIDIA L40S GPU.
Reported as roughly 2× throughput under the named release configuration. Hardware, batching and serving software can materially change results.
Late-interaction storage per page
Hierarchical token pooling plus asymmetric quantization.
Hugging Face states a 255-fold reduction while retaining more than 95% of its baseline nDCG@10 score. The quality threshold is tied to the team’s stated baseline.
Compelling architecture, provisional performance claims
The release makes NeoMME practical to examine, but the supplied material does not establish peer review or independent replication. Production value will depend on behavior outside the release configuration.
The strongest case is efficiency for non-generative retrieval
A shared encoder, Apache 2.0 checkpoints and Transformers support directly address deployment complexity. The reported speed and storage savings target costs that can determine whether visual retrieval works at scale.
The counterargument: specialization may conceal lost capabilities, and compression that preserves most of one benchmark score may still cause unacceptable failures for particular languages, layouts or search tasks.
Test throughput and memory on multiple GPUs with disclosed batch sizes.
Include preprocessing, index construction, memory use and query latency.
Evaluate scans, handwriting, dense tables, rare languages and degraded pages.
Measure which queries, languages and layouts degrade under smaller indexes.
The checkpoints now move from release claims to real-world tests
Developers can compare the 260M and 800M variants with OCR-based systems, dual-tower encoders and generative vision-language models using their own pages, languages and hardware.
Independent ViDoRe v3 reproductions
Comparable evaluation settings will show whether benchmark placement holds outside the original team.
Transparent latency and memory data
Useful reports should disclose resolution, batching, quantization, storage calculations and baselines.
Multilingual production collections
Results on scans, complex layouts and visually dense pages will reveal the architecture’s practical limits.
NeoMME is a credible argument for retrieval-specific multimodal encoders: one bidirectional Transformer processes language and page imagery, while reported optimizations attack throughput and index cost. The architectural contribution is available to test; the headline performance numbers still require independent confirmation.
Smaller Indexes for Visual Retrieval
The release targets a practical cost problem in visual retrieval: models adapted from generative vision-language systems may retain components that classification, retrieval and token-labeling workloads do not use. By removing the separate vision tower and causal decoder, NeoMME concentrates computation in one encoder. That design could reduce deployment complexity for teams building multilingual document search and visual RAG, although real-world savings will vary with hardware, batching, page resolution and serving software.
Hugging Face reports that, at a matched 2,048-by-2,048-pixel input size on an NVIDIA L40S GPU, NeoMME’s 260M version encoded about 51 pages per second, roughly twice the throughput reported for ColModernVBERT under that comparison. The team also says hierarchical token pooling and asymmetric quantization reduced late-interaction storage from about 1.5 megabytes to 6 kilobytes per page, a stated 255-fold reduction, while retaining more than 95% of its baseline nDCG@10 score.
Products Worth Considering
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
Anker SOLIX S2000 Portable Power Station with Power Strip
Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, Eucalyptus Green
As an affiliate, we earn on qualifying purchases.
One Encoder Replaces Two Towers
Many visual-document retrievers begin with a pretrained vision encoder, use a projector to translate its features and pass the result into a language model. NeoMME instead trains image and text processing together from scratch. Hugging Face positions the project as a step beyond ModernVBERT, which used an efficient bidirectional text architecture but retained a separate pretrained SigLIP2 vision tower.
The retrieval approach also differs from text-first systems that extract and divide PDF text before indexing it. Treating each page as an image preserves spatial relationships and non-text elements that OCR output may lose. The tradeoff is that image-native retrieval must process many visual tokens, making throughput and index size central deployment constraints rather than secondary benchmark details.
Benchmarks Still Need Outside Replication
The supplied material does not establish whether the technical report has undergone peer review, and the performance figures are vendor-reported results. Independent testing is still needed to determine whether the reported throughput, retrieval quality and storage savings carry across other GPUs, document languages, page types, batch sizes and production serving stacks.
The source also does not give enough detail to judge end-to-end operational cost, including preprocessing, index construction, memory use and query latency. It remains unclear how the models perform on handwritten material, low-quality scans, rare languages or documents requiring fine-grained visual reasoning. The claim that more than 95% of baseline nDCG@10 is retained refers to the team’s stated baseline; broader quality effects may differ by dataset.
Developers Begin Testing Released Checkpoints
The immediate milestone is outside evaluation of the released checkpoints through Hugging Face Transformers. Developers can compare the 260M and 800M variants against OCR-based retrieval, dual-tower encoders and generative vision-language systems using their own documents, languages and hardware.
Watch for independent ViDoRe v3 reproductions, detailed latency and memory measurements, and results from production-scale indexes. Tests that disclose image resolution, batch size, quantization settings, storage calculations and comparison baselines will show whether NeoMME’s reported efficiency extends beyond the release configuration.
Where I land
My view is that NeoMME presents a persuasive architectural case for retrieval systems that do not need generative decoding. A shared encoder, permissively licensed checkpoints and support in Transformers make the work practical to examine, while the reported throughput and index-storage figures address costs that can decide whether visual retrieval is deployable at scale. This is my interpretation, not a finding established independently of Hugging Face’s release.
The strongest counterargument is that specialized efficiency can hide broader tradeoffs. A unified encoder trained from scratch may lose capabilities inherited from large pretrained vision or language components, and benchmark placement does not by itself show reliability across messy production documents. Storage compression that retains more than 95% of one baseline score may still create unacceptable losses for particular languages, layouts or search tasks.
I would become more confident if independent teams reproduced the results across several accelerators, disclosed full serving costs and found stable quality on multilingual scans, tables and visually dense pages. I would revise my positive assessment if those tests showed that the reported gains depend narrowly on one benchmark configuration or that compressed indexes cause material retrieval failures in realistic collections.
Key Questions
What is NeoMME?
NeoMME is a multilingual, multimodal encoder family with 260M and 800M parameters. It creates vector representations from text, images or combined inputs using one bidirectional Transformer.
How does NeoMME differ from a generative vision-language model?
NeoMME does not use a separate pretrained vision tower or an autoregressive language decoder. It processes image patches and text tokens through the same encoder pathway and is intended to produce representations rather than generate answers token by token.
What does NeoMME-Retriever search?
It ranks document-page images against a query. This lets the retriever use text alongside page layout, charts, tables and typography without depending on an OCR extraction stage.
Are the reported speed and storage results independently verified?
No independent verification is included in the supplied source. The figures are results reported by the NeoMME team, including about 51 pages per second for the 260M model under a specified L40S test and a 255-fold stated index reduction.
Can developers use the models commercially?
Hugging Face says all released checkpoints use the Apache 2.0 license. Developers should still review the model cards, dataset disclosures and license text for their intended deployment.
Source: Hugging Face
