Why the same AI gives two companies very different results (and what to ask before you buy)

The half nobody shows you in the demo

People who work with AI call that setup the "harness". A Hugging Face glossary published in May 2026 describes it as the layer that calls the model, runs the actions it asks for and decides when to stop. In plain words, it is four things: the instructions the AI gets, the tools it can use, the checks on its work, and the rules for when the job counts as done.

Picture a skilled new hire on day one. Nobody explains how your company does things. There are no keys, no logins and no truck. Nobody looks at the work before it reaches the customer. The talent is real, and the results are still poor. An AI model with no proper setup around it is that new hire.

The talent is real, and the results are still poor.

What the study measured

In July 2026, Joel Niklaus at Hugging Face published a test built on one open AI model and a set of realistic legal tasks, such as reviewing documents and drafting memos. An automated grader scored each piece of finished work against a long checklist.

He ran the same model through five different setups. The model, the tasks and the grader did not change. The share of checklist items passed came out at 3.5%, 23.2%, 45.4%, 63.4% and 80.1%. In his words: "Same model, same judge, same tasks, five very different scores."

Same model, five setups

3.5%Lowest general purpose setup

63.4%Standard setup supplied with the test

80.1%Setup tuned step by step for the job

The 63.4% came from the standard setup supplied with the test. Three general purpose setups scored below it. The 80.1% came after the researchers improved the setup step by step and kept only the changes that raised the score. Nobody retrained or swapped the model at any point.

The biggest fix was a boring one

The model often did good work and then lost the marks on the last step. It saved the finished file under the wrong name, or left it in a working folder. The grader only looked in one place, so good work scored zero.

The handover

The biggest single improvement in the whole study was a check that put the finished file where the grader looked, under the right name. Think of the new hire who does a clean job on site and leaves the paperwork in the truck. The skill was there. The handover was missing.

Checks held up better than clever wording

Most of the best results came from plain, fixed checks written in code. Five of the six top scoring setups relied on them, and only one relied on better wording of the instructions.

The researchers also took a setup tuned for one model and tried it on a model from a different family. Overall it barely moved the score (+0.4 points). The fixed checks still helped. The instructions written for the first model hurt some tasks the second model could already finish.

For a business owner, the lesson is simple. Wording that suits one AI model may not suit the next one. A check that confirms the quote landed in the right folder keeps working either way.

A good setup still has a ceiling

The gains flattened at about 83% on the practice set the researchers used for tuning. The researchers expect the misses that remained to need a stronger model, better tools or better instructions. More checks alone would not close them.

Perfect work stayed rare. The share of tasks where every checklist item passed went from 0% to 5.0%. So your people still review the output and make the judgement calls. That fits how we think about AI in a company: you, amplified. The same team stops doing work they were never hired for, like retyping numbers and chasing files, and spends that time on the work that needs their experience.

What to ask before you buy AI for your business

You do not need a technical background for this. Ask these in plain language and listen for plain answers.

1. What checks the work before I see it? Ask for the actual list of checks. "The AI is very accurate" is not an answer.
2. How does it know a job is finished, and where does the finished work end up? The quote should land in your estimating folder and the note in the customer record, every time.
3. Which of my tools can it use, and what can it not touch? An AI that cannot open your calendar, inbox or job software can only give advice.
4. If the AI model underneath changes, what keeps working? You want an answer about checks and process, not only about wording.
5. Can you show me results on my kind of work, misses included? A vendor who measures will have the misses. A vendor with no misses to show has not measured.

The right setup is different for every company

This part is practical judgement, not something the study measured. A five person renovation crew and a forty person mechanical contractor do not need the same setup. The type of work decides which tools the AI needs. The size of the team decides who reviews what.

How fast an answer is needed decides how many checks can run first. A missed call needs a reply in seconds. A bid package can wait for a careful second pass. Be wary of any vendor who offers the same package to everyone.

What this does not prove

This was one study, on legal tasks, tuned on one model family. It does not show that a construction or service company will see the same gap, or the same numbers. It does not say the model is unimportant, because the researchers expect the last misses to need a stronger one.

A program did the grading against a checklist, and the grader was not a client. Treat the study as strong evidence for one idea: setup can decide a large part of the result. Then test that idea on your own work.