Lab · Field test

Can a cheap yes/no model check AI product images?

By Angelo Pullen, for the team at LARGE FORMAT · November 3, 2026

When the team at LARGE FORMAT generates product images for a brand, every part has to match the real product: the chain sits on the bike's right side, the kickstand is folded up, the headlight cage has the right pattern, and there is no stray text anywhere. A full vision audit checks 43 items per image, and it is the slow, expensive step.

So we asked a narrow question: can a cheap yes/no model take the first pass, so that only the images and checks it flags go on to the full audit? We tested Jev, TypeSafe AI's decision model, on 29 AI-generated images of an electric bike. Jev gives advice only. A person approves every image.

1. The problem

Jev does not write text. You send it a state (the thing to judge) and typed questions, and it returns a probability from 0 to 1 for each yes/no question, or a choice with a confidence. That makes it cheap and fast to ask many narrow questions about one image.

Images are experimental. Jev 1.13 and 1.14 both accept them, but each request needs an opt-in header, and TypeSafe says images "aren't yet up to our standards to call ready for production". Its release note warns that "all models (Jev included) perform worse on multimodality over pure text", meaning on images compared with text.

There were three ways to use it. A: no first pass, the full audit on everything, as now. B: Jev as a gate on every check, so only flagged checks reach the audit. C: Jev on the few checks it gets right, beside the full audit. We tested B and ended up at C. Jev's experimental Box question, which returns where something is in an image, was not tested.

2. Setup

You need Python 3 and a TypeSafe API key. The run script uses the standard library only; the preparation script uses Pillow to resize images, because the API accepts 280 to 1,568 pixels per side and rejects anything outside that range instead of resizing it. The repo holds the questions, scripts, labels and every saved answer, so the results can be recomputed without a key.

git clone https://github.com/lrgfrmt/jev-image-checks.git
cd jev-image-checks
python3 score.py                       # recompute every result here, no key needed

pip install pillow                     # only to prepare your own images
python3 prepare.py path/to/images
read -s JEV_API_KEY && export JEV_API_KEY
python3 run_jev.py --dry-run           # no network: builds every request, estimates cost
python3 run_jev.py --dev               # 3 images, 9 calls, checks the wiring
python3 run_jev.py

The key is read from your environment and never printed or saved. Only the prepared images can be sent: a filter refuses any file whose fingerprint is not in the manifest. The script stops before any call that could take spend past a cap. We pinned jev-1.14.0, which TypeSafe made available on 14 October 2026, so a later alias change cannot move the results.

3. Asking Jev about one image

One call per image sends the picture once, as a placeholder in the state, and every question refers to it by name. The endpoint is POST https://api.typesafe.ai/v1/systemone, with the header TypeSafe-Danger-Zone-Images: true. Every question is worded so that yes means "a problem is visible". Here is one request, abridged: the real descriptions are longer and the real call carries all the questions.

{
  "model": "jev-1.14.0",
  "state": {
    "product": "A black electric bike with a moped-style frame and 20 inch fat tyres, shown in a generated studio product photograph.",
    "sides": "The bike's right side is on the rider's right when sitting on the seat facing forward.",
    "rule": "Judge only what is visible. A part that the camera angle hides is not a problem.",
    "intended_view": "Right side profile: the camera looks straight at the bike's right side at 90 degrees.",
    "photo": "img-6f6e8e516cc94c4bfebd"
  },
  "questions": {
    "a8_kickstand_down": {
      "type": "noul",
      "instructions": "Is the bike's kickstand down, reaching the ground, in `photo`?",
      "criteria": {
        "true": "A stand leg reaches down from the frame to the floor.",
        "false": "The kickstand is folded up against the frame, or is not visible."
      }
    }
  },
  "assets": {
    "img-6f6e8e516cc94c4bfebd": {"type": "image", "format": "jpg", "data_base64": "..."}
  }
}

The full request carried 129 such questions, grouped under the 43 checklist items. 52 were the questions we wrote first (the core set), and 77 were finer questions added afterwards (the detail set). Two more passes asked the same questions with four full-resolution close-ups, and asked one camera-angle question on its own.

4. Turning answers into a gate

The labels came from a blind Claude vision audit: 1,247 image-by-check verdicts, of which 98 were problems, 1,046 were fine and 103 were unsure and not scored. Three images checked the wiring; the other 26 are the held-out images, used for every gate result.

# score.py, in short
item_score = max(answers[q] for q in questions_for_item)   # yes means "a problem is visible"
flagged = item_score > 0.3                                 # 0.3 or lower passes the gate
false_pass = label == "problem" and not flagged            # the costly error
false_alarm = label == "fine" and flagged                  # costs review time only

The gate setting is how sure Jev must be that a check is clean before that check skips the audit. We started at 0.3, the starting point TypeSafe's docs suggest for "no", and also report 0.5 and 0.7.

5. What we measured

All 87 calls succeeded, every one answered by jev-1.14.0. They used 1,291,016 input tokens, about 5.4 cents at the published price. The whole-image calls with all 129 questions cost about 1.5 cents.

Questions and gate setting	Problems caught	Let through	Checks sent on
52 core questions, 0.3 (suggested)	80 of 91	11	48%
52 core questions, 0.5	59 of 91	32	25%
52 core questions, 0.7	34 of 91	57	9%
Core plus 77 detail questions, 0.3	87 of 91	4	66%
Core plus detail, 0.3, with close-ups	89 of 91	2	64%

As a gate on every check, Jev lets too much through. At the suggested setting it let 52% of checks skip the audit, but 11 of 91 known problems (12%) went with them, one of them serious: a handlebar display drawn wrong, which Jev scored at 0.02. Every held-out image had at least one known problem, so flagging all 26 was right at the image level, and this test cannot show whether a truly clean image passes. More questions did not mean better answers: the detail set sent 66% of checks on instead of 48%, and false alarms rose from 371 to 593.

One check it handles: the kickstand. The kickstand item has two core questions: is the stand down, and is the bike tipped over. Together, on 28 images with a clear verdict, they caught 12 of 12 kickstands left down, with 1 false alarm in 16. The stand-down question alone caught 9 of 12: it missed all 3 top-down images, and the tipped-over question flagged those instead, at 0.52 to 0.59, which looks like luck. On 66 new images from two different image models, a single kickstand question agreed with the audit 49 times out of 49.

Checks to never trust it with. It flagged all 26 held-out images as looking like a render, though the audit failed none. It caught 17 of 20 stray-text problems but flagged all 3 images whose text was fine, so its text flags are hints, not verdicts. With the core questions it caught 2 of 5 problems with the seat's reflective marks.

Sorting by camera angle. On the 29 generated images it named the right view on 20: all 20 images of 7 kinds of view, none of the 3 top-down images (each called a right-side view at 0.81 to 0.91 confidence) and none of the 6 rear three-quarter images. On real studio photos of the bike, with a simpler four-way question, it got 60 of 76 right, and every answer it gave at 0.8 confidence or higher was correct: 33 of 33.

Speed and cost. One whole image with 129 questions took a median 290 ms and about 12,100 input tokens: $0.51 per 1,000 images at $42 per billion input tokens. Four close-ups raised that to $1.13 per 1,000 and a median 525 ms, and moved the results little. In our earlier test on Instagram brand-lead messages, the same model caught 28 of 33 held-out leads with no false alarms, at a median 90 ms; one image with a single question used about 6.3 times the input tokens of a text message. It did far better on that text task, which fits TypeSafe's own warning; the two tests asked different kinds of questions.

How we would run it. Generate the views. Ask both core kickstand questions on each image, and mark any image above 0.3 on either for a retouch before the audit. Run the full audit on every image, as now. A person reviews and approves every image and checks the kickstand on every top-down view. For a new product's shoot, file the photos Jev sorts at 0.8 confidence or higher, and send the rest, and every top-down shot, to a person: on this shoot that would have filed 33 of the 76 photos with no mistakes.

6. Sample output

Two real answers to the stand-down question, from the saved run log. On an image where the audit found the kickstand down, Jev said 0.95. On an image where the stand was folded up, it said 0.09. The same folded-up image also scored 0.66 on "looks like a render rather than a photo", an item the audit never failed: that question is one Jev over-flags.

{"image": "A-02", "a8_kickstand_down": 0.95, "a8_not_upright": 0.01}
{"image": "A-06", "a8_kickstand_down": 0.09, "a11_not_photo": 0.66}

From data/runs/outputs.jsonl, trimmed

7. Limits and costs

The labels are a Claude audit, not a person. 35 of its reported problems were re-checked by a second Claude pass, and 34 held. Jev's score is agreement with that audit.
Small sample, one run. 29 images, one call per image for each of the three ways of asking. Nothing was sent twice, so how much answers move between identical image requests is untested.
The detail questions were written after reading the audit's findings, so on those failure types they may look better than a fresh question would.
The round of 66 new images used different questions and a 0.5 gate setting, and its truth was also a Claude audit.
No product images are published. The repo holds the questions, labels and saved answers, not the images.
Costs. $42 per billion input tokens, output tokens currently free, on 3 November 2026. Images count as input tokens. The audit's own cost was not recorded, so we claim no saving against it.

8. Links

The code, labels and saved results, MIT licensed: github.com/lrgfrmt/jev-image-checks
TypeSafe AI (price): typesafe.ai
Jev images guide: docs.typesafe.ai/images
Models and versions: docs.typesafe.ai/models
Confidence guide: docs.typesafe.ai/confidence
API reference: docs.typesafe.ai/api/system-one
Jev 1.14 release note: typesafe.ai/blog/jev-1-14

Questions, or an image workflow you want checked? Start a project with the team at LARGE FORMAT.

Lab · Jev-image-checks.txt