Jev vs Laya: I Tested Both on 27 Decisions

A guide on Hugging Face asked this week whether you should build on Jev or on Laya. It answered with a benchmark named after one of the two contestants. So I ran both myself, same inputs, same question JSON, on my laptop and against the hosted API. Jev got 27 of 27. Laya got 17.

I expected a gap. I did not expect Laya to beat the hosted API on speed for short English text, which it did, from my laptop, on CPU. So the answer depends on what you feed it.

Jev vs Laya: Same Request, Very Different Models

Both are decision models, not chatbots. You send a `state` (a ticket, an email, a JSON blob) and typed questions. They return probabilities in one forward pass instead of generating text. Three question types exist on both: `noul` (a calibrated yes/no), `choice` (pick one option) and `score` (a level on an ordered rubric).

Jev is TypeSafe's hosted model, launched mid-September 2026. Laya comes from Convai Innovations, published under Apache 2.0, built on a ModernBERT-large encoder with a decision head on top. The Node port (`@receptron/laya`) matches Jev's request and response shape, so I could send the exact same payload to both with one line changed.

The spec sheet, from TypeSafe's docs and Laya's model card:

There are three Laya checkpoints. The English root I tested. A multilingual one on mmBERT-base (322M, 100+ languages, up to 8,192 tokens). And a typed-decisions variant fine-tuned for this exact task. Only the English one has a ready ONNX export right now, so that is what most people will actually install.

I Ran Both on the Same 27 Questions

I wrote 27 items by hand, labels first, before running anything. Five groups: English support tickets to route, the same kind of tickets in German, shell commands that may or may not be destructive, product reviews to score from 0 to 4, and one long text with a single customer complaint hidden in it. Every item went to both models with identical JSON.

English routing was a tie. Laya put 0.95 on "billing" for a double charge, which is exactly what you want from a local model at zero cost. If your workload is short English text into a handful of buckets, Laya is a real option today.

Everything else went sideways. On the commands, Laya was confident where it mattered least: 0.95 on `rm -rf` (correct), then 0.50 on `git log` and 0.61 on `ls -la`, both of which are read-only. Jev put 0.01 on those. A 0.5 on a yes/no question is the model telling you it has no idea, and a destructive-command gate built on that would block half your harmless commands.

Jev's whole run was 28 calls and 19,006 input tokens. Cost: under a tenth of a cent. Laya's cost was a 1.6 GB download and 131 seconds for the first load.

Small set, yes. 27 items do not make a benchmark, and I would not quote the percentages. But the failures were not random. Each one lines up with a limit Laya's own model card admits.

Where Laya Breaks, and Where It Wins

The 512-token wall hurt most. I took a 14,000-character article and put one sentence at the end, a customer saying their order arrived broken and asking for their money to be returned. The question was whether the text contains that request. Jev found it at 0.96. Laya answered 0.00, and I don't blame the model for that one: it never saw the sentence, because the state gets truncated after 512 tokens. Move the same sentence to the top and Laya scores 0.88. That is not a model you can hand a support thread, a log file, or a contract to without chunking it yourself.

Then the reviews. "Absolutely love it. Best shirt I own, already ordered two more." Laya put 66% of its probability on "very negative". The model card says the `score` primitive "shows weak performance", and it does. Jev returned 4.00 on the same review.

German is partly my fault. I ran the English checkpoint, which routed a white checkout screen to billing and a lost parcel to "bug". The multilingual checkpoint exists and should do better. I could not test it without building the ONNX export myself.

The Hugging Face guide undersold Laya on a few things.

On short inputs Laya answered in 135 to 206 ms on my Apple-silicon laptop, CPU only. Jev's median from Berlin was 219 ms. Past 400 tokens Laya climbs to 1.2 seconds on CPU, where a GPU helps: the model card measures 33 to 40 ms per question on a Tesla T4.

Nothing leaves the machine, either. Jev does not train on requests, but zero data retention is an enterprise contract, not the default.

Fine-tuning is Laya's real pitch, and the model card is blunt about why. The base English checkpoint scores 0.362 on the card's 2,000-decision benchmark, below the 0.461 you would get by always picking the most common answer. The fine-tuned variant reaches 0.766. The base model is a starting point for training, not something to ship as-is.

I covered the same trade in general terms back when Ollama changed how I think about AI infrastructure. This is that argument with a decision model instead of a chatbot.

The Numbers the Guide Left Out

The Hugging Face guide quotes JevBench v1.3.0: 534 decisions, Jev 74.4 composite against Laya 54.4, hard cases 74.1% against 34.1%. I could not find who runs JevBench, and a benchmark named after one contestant deserves a second source. My 27 items point the same way, for what that is worth.

The operational stuff matters more than the headline score once you build something:

If you tuned a 0.8 threshold on one Jev release, pin that release. The alias will move under you.

Cost is where the comparison flips depending on volume. At 4 cents per million input tokens, I scanned all 428 of my blog articles with Jev for less than a cent earlier this month. You need a very large workload before a GPU box beats that on price. Privacy and offline use are the reasons to self-host, not the bill.

If you are wiring either into a routing layer, my notes on how I route work between Opus, Sonnet and Haiku cover the gating logic. It is the same idea one layer down. And the self-host calculus for the rest of my tools is in the solo studio stack.

Bottom Line

Start with Jev if you have no labelled data, your inputs run longer than a page, you need scores or non-English text, or nobody on your team wants to run a GPU. It was right on every item I gave it, and the whole test cost less than a tenth of a cent.

Pick Laya when the data cannot leave your network, or when you have a few thousand labelled examples and will fine-tune. Keep inputs short and English, stick to `choice` questions, and calibrate before you trust a probability. Out of the box it is a training base. The fine-tuned version is the one to compare against Jev, and nobody has published that head-to-head yet.

I ran the whole test from Claude Code. The setup I use there, hooks and commands included, is packaged as the Claude Blueprint. The test script itself is 122 lines, one file.

Has anyone fine-tuned Laya on their own tickets yet and compared it with Jev?