Lab · Field test

The audit passed 15 images. We rejected all 15.

By Angelo Pullen, for the team at LARGE FORMAT · November 4, 2026

An electric rideables brand needed a reference sheet for one e-bike: ten fixed views, from both side profiles to a top-down view and a rider's-eye cockpit view, generated as photoreal images from the brand's own studio photos. A reference sheet only works if every image shows the real bike: the right number of buttons on the right side, the headlight pointing forward, the real proportions, no metal that is not on the bike. A checklist audit run by AI is fast, but it is not the judge. A person at LARGE FORMAT marks every image approve, hold or reject on our review board, and those marks are the answer key for every number below.

• Three rounds and 221 generated images, on fal, Higgsfield and Runway.
• In the round 2 pilot the checklist audit passed 15 images, and the reviewer rejected all 15.
• After the checklist was rebuilt around the reviewer's own criteria, the round 3 audit matched the reviewer on 50 of the 74 images the reviewer approved or rejected. Marking everything approved would have matched 44.
• Approved images used as references came back nearly unchanged: 23 near-copies in round 3.
• Jev, a cheap yes/no model, flagged at least one problem on every round 3 image, including all 44 the reviewer approved.

1. The problem

Image models and platforms. Nano Banana 2.1 ran in rounds 1 and 3, Nano Banana Pro and GPT Image 2.5 in the round 2 pilot, and GPT Image 2.5 again on fal in round 3. Each round sent the same prompt and the same reference photos to fal, Higgsfield and Runway, so the platforms could be compared.

The blind audit. Claude vision auditors saw each image under a code, without knowing the platform or model (a letter in round 1, a random three-letter code after that), and scored each image against the studio photos: 43 checks per image in rounds 1 and 2, and 29 or 30 in the rebuilt round 3 checklist. Each check is marked as a match, a mismatch, missing, extra, hidden from this angle, or unsure.

Jev 1.14. Jev is a typed decision model from TypeSafe AI: you send it a state and yes/no questions, and it returns a probability for each one instead of writing text. TypeSafe calls it "TypeSafe's first public System One Model, optimized for automation". It reads images experimentally: TypeSafe says images "aren't yet up to our standards to call ready for production", and each request needs an opt-in header. It costs $42 per billion input tokens, each image counts as input tokens, and output tokens are currently free.

Our review board. A person at LARGE FORMAT looks at every image and marks it approve, hold or reject, sometimes with a note on what is wrong: 22 of the 221 images carry one, 1 of them in round 3.

1. Checklist audit alone. Blind auditors score every part, and an image passes on its score. It is the most thorough on parts, but round 2 showed it can pass images a person rejects.
2. Jev alone, as a cheap gate. Ask yes/no questions about every image and keep only what it clears. It is the cheapest and fastest, but it flagged something on every image it saw, so a gate that needs zero flags would pass nothing.
3. Audit plus a person, with Jev on the side. The audit finds and names flaws, Jev's answers are logged next to it, and a person makes every call. This is what we run. It gives up speed: a person looks at every image. Review time is not in the round records we kept.

2. Setup

• Reference photos. The brand's studio photos of the bike, each later labelled by side and confirmed by a person at LARGE FORMAT.
• Round 1. Nano Banana 2.1 at 2K, one image per view per platform, ten views.
• Round 2 pilot. Nano Banana Pro and GPT Image 2.5 at 4K on all three platforms, for three views: left side, rear-left three-quarter and cockpit.
• Round 3. Nano Banana 2.1 at 4K on all three platforms, plus GPT Image 2.5 at its highest quality setting on fal, for all ten views.
• Spend. Read from each platform's balance before and after a run where the platform shows one.

3. How the checks decide

In round 3 the audit stopped deciding by percentage. An image fails if any of the reviewer's criteria fails, or one of a short list of whole-image checks fails. Smaller flaws, such as stray text, make it "pass with retouch". The rule, simplified from the scoring script, which also applies the mid-audit correction on button visibility described below:

DECIDE = ['A1', 'A2', 'A6', 'A9', 'B13', 'B29'] + [f'C{i}' for i in range(1, 13)]
for item, result in checks.items():
    if result not in ('MISMATCH', 'MISSING', 'EXTRA'):
        continue
    if item == 'A1' and reassign_to:      # right view, wrong slot: judge it as the view it shows
        continue
    (deciding if item in DECIDE else flaws).append(item)
verdict = 'FAIL' if deciding else ('PASS-RETOUCH' if flaws else 'PASS')

Near-copies of the reference images were found by shrinking both images to 512 pixels wide in greyscale and taking the mean pixel difference. Under 6 out of 255 counted as a copy:

a = Image.open(output).convert('L').resize((512, h))
b = Image.open(anchor).convert('L').resize((512, h))
diff = ImageStat.Stat(ImageChops.difference(a, b)).mean[0]   # 0 = identical
near_copy = diff < 6

4. What we measured

Round 1: one image per view, and the wrong-side photos

Round 1 made 29 images: ten views on each platform, minus one Runway job that failed and was refunded. The reviewer approved 11, held 11 and rejected 7.

The audit found the same problems on every platform. 22 of 29 images had stray or garbled text, and 12 of 29 had the kickstand down. 4 images showed the wrong view or the wrong side of the bike. The cause was in the inputs. Right-side photos were mixed into all three left-view reference sets, six in total, not counting one whole-bike profile sent on purpose. The round 1 review traced the wrong-side images to two of them, which had been filed as left-side shots. Every reference photo was labelled by side before round 2.

Audit scores did not separate the reviewer's marks. Approved images averaged 91.8% of checks matching, from 80.8 to 97.4. Rejected images averaged 86.4%, from 74.4 to 92.3. The best rejected image scored higher than the worst approved one. Platform averages were close: fal 89.4%, Higgsfield 89.1%, Runway 90.3%. Higgsfield cost 2 credits an image and Runway 6, both read from the balance. No charge was recorded for fal.

Round 2 pilot: 15 passes, 15 rejections

The pilot made 66 images. The audit passed an image with no serious failure and at least 95% of checks matching. It passed 15, all from GPT Image 2.5: 8 on fal and 7 on Higgsfield. The reviewer rejected all 15. Across the pilot the reviewer approved 1, held 4 and rejected 61. The audit failed the one approved image, for stray text only.

Our read of the reviewer's notes: the checklist asked whether each part was there and looked right. It did not measure what the reviewer judged:

• overall proportions, such as seat and battery length against the wheels;
• invented metal, such as extra rails or tubes;
• the direction of the physical headlight: its lens and wire cage must face the direction of travel;
• the count and side of the handlebar controls.

The pilot cost 228 Higgsfield credits for 24 images and 628 Runway credits for 18 images, read from the balances. The fal cost was not read from a balance; the review board notes only a list-price estimate.

Recalibration: teaching the checklist what the reviewer judges

Proportions measured on approved images. The allowed ranges came from two round 1 images the reviewer approved: wheelbase 1.8 to 2.0 tyre diameters, seat length 0.9 to 1.1, battery length 0.55 to 0.67.

Checklist version 3. It adds 12 criteria written from the reviewer's notes, including proportions, no invented metal, headlight facing forward, controls count and side, and no invented emblems. Any one failing fails the image.

The reviewer's rulings:

• A correct image of a different view than the one requested is judged as the view it shows and reassigned to that slot.
• Small stray text is a flaw a retoucher can fix: the image passes with retouch, and the flaw is named.
• The kickstand stays. The reviewer wanted it kept in the images, so a lowered kickstand is noted but does not fail an image.

Corrections found during the audits themselves:

• Round 1: the truth for the headlight bracket was wrong. At full size the studio photos show one round hole and three chevron-shaped slots, and three scores were corrected.
• Round 2: the bracket check was dropped, because the pilot prompt and the checklist both described it wrongly.
• Round 3: the rule on when the red and green buttons may be visible was corrected mid-audit, after a real studio photo of a front three-quarter view showed them edge-on.

Round 3: all ten views at 4K

Round 3 made 126 images: 24 from Nano Banana 2.1 on fal, 40 on Higgsfield, 22 on Runway, and 40 from GPT Image 2.5 on fal. The reviewer approved 44, held 52 and rejected 30.

Against the reviewer, the new checklist stopped failing most good images, but still let bad ones through. Of the 74 images the reviewer approved or rejected, the audit matched on 50: it did not fail 29 approved images and it failed 21 rejected ones. For scale, marking every image approved would have matched 44 of 74. It still failed 15 images the reviewer approved, and let 9 of 30 rejected images through. In round 2 the old checklist matched on 46 of 62 and let 15 of 61 rejected images through.

The audit was far more lenient on GPT Image 2.5 than the reviewer. Leaving out near-copies, the audit did not fail 26 Nano Banana 2.1 images, and the reviewer approved 26, though only 14 of them were the same images. For GPT Image 2.5 the audit did not fail 18 of 40, and the reviewer approved 6, 4 of them among those 18. All 3 images the audit found with no flaw at all were GPT Image 2.5, and the reviewer rejected one of them. On the left-side view, all 4 GPT Image 2.5 images measured a seat of 1.15 to 1.25 tyre diameters, over the 1.1 limit, and 3 of the 4 a battery over the 0.67 limit, up to 0.72. The same pattern showed in round 2, where every audit pass was a GPT Image 2.5 image the reviewer rejected. Where the checklist measured proportions, it caught the stretch: it failed all 4 of those left-side images on proportions. But proportions were measured on only 55 of the 126 images, and judged by eye or not applicable on the rest.

Approved references came back nearly unchanged. For five views, earlier approved renders were sent as references next to the studio photos. 23 Nano Banana 2.1 images came back as near-copies of one of them, with a mean pixel difference of 4.2 out of 255 at most: 6 on fal, 10 on Higgsfield, 7 on Runway, all in those five views. GPT Image 2.5 made none. The reviewer approved 12 of the near-copies, held 11 and rejected none. Those approvals are mostly inherited from the earlier image, not new work. With near-copies left out, approvals were 8 of 18 on fal, 10 of 30 on Higgsfield, 8 of 15 on Runway, and 6 of 40 for GPT Image 2.5.

Cost and failures, as observed.

• fal: Nano Banana 2.1 failed on 4 of 10 views with "no_media_generated" and no reason given. The jobs were not resubmitted. fal shows no balance, so its cost was not observable.
• Higgsfield: 120 credits for 40 images, quoted at 12 credits for each four-image call and matched by the balance. Other work was spending on the same account at the time.
• Runway: 12 credits for each 4K image. After the first view, 310 credits were left, so two images per view was the most that fit for the other nine views: 22 images for 264 credits.
• One view, the straight rear view, got no approved image at all.

Platform credits are not dollars, and the dollar cost of the image runs was not recorded.

Jev: where it helped and where it should not be trusted

Jev ran beside the audit in rounds 2 and 3 and decided nothing. All 66 round 2 calls and all 126 round 3 calls succeeded on version jev-1.14.0. Round 3 asked 21 questions per image, with a median reply in 304 milliseconds. Round 3 used 1,238,374 input tokens, which is $0.052 at the published price, about $0.41 per 1,000 images.

Where it agreed. On the round 3 checks with a clear answer, Jev agreed with the auditors on the kickstand 98 of 110 times, and caught 25 of the 29 kickstands they found down. It also agreed on the display 100 of 117 times and on the handlebar controls 102 of 121, but that agreement is mostly on images where nothing was wrong: it caught 1 of 18 display failures and 3 of 18 controls failures. Only the kickstand shows real catches, and here a lowered kickstand does not fail an image.

Where it should never be trusted.

• As a gate. It flagged at least 3 items on every round 2 image. In round 3 it flagged at least one item on every image, including all 44 the reviewer approved.
• Invented icons and emblems. It agreed with the auditors only 23 of 110 times, and it flagged an invented emblem on 42 of the 44 approved images.
• Headlight direction. It agreed 56 of 116 times, close to a coin toss.
• Catches that look real but are not. Its "no chain" flag hit 15 of 30 rejected images against 3 of 44 approved ones. But 13 of those 15 were left-side, front and left three-quarter views, where the chain is hidden or barely visible anyway. Within those four views it flagged 1 of 11 approved images and 13 of 19 rejected ones, but rejected images cluster in exactly those views, so these numbers cannot separate camera angle from quality.

5. Sample output

One round 3 audit record for a GPT Image 2.5 left-side image, shortened:

verdict: FAIL  (A1 wrong view, C1 proportions)
C1: wheelbase 2.10 D (limit 2.0), tyre gap 1.10 D (limit 1.0),
    seat 1.25 D (limit 1.1), battery 0.72 D (limit 0.67): stretched bike, long seat and battery
A7: background is light grey, not pure white; retouchable

D is the outer diameter of one tyre, so the measurements do not depend on image size. One reviewer note from round 2, on an image the audit scored 100%: "battery is too large and there is extra metal that does not exist and proportions are wrong."

6. How we would run it

1. Label every studio photo by side and view, and have a person confirm the labels before anything is generated.
2. Start with one image per view on each platform, and let a person mark them before spending on more.
3. Write the person's reasons into the checklist as pass or fail criteria, with measured ranges taken from images the person approved.
4. Run the blind audit to find and name flaws. Treat its pass as "worth a look", never as approval.
5. Log Jev's kickstand answer as a cheap second opinion, and do not rely on its other answers.
6. When approved images go back in as references, measure how close each new image is to them, and do not count a near-copy as a new result.
7. A person at LARGE FORMAT approves every image. Jev and the audit are advice only.

When to use what. In this test the image model mattered more than the platform. With near-copies left out, Nano Banana 2.1 had 26 of 63 images approved and GPT Image 2.5 had 6 of 40. On the 6 views all three platforms completed, Nano Banana 2.1 approvals were level across platforms: 8 of 18 on fal, 8 of 20 on Higgsfield and 4 of 9 on Runway. So choose the platform for reliability and budget rather than quality. Higgsfield returned every image it was asked for in all three rounds: 10, 24 and 40. Runway charged 12 credits per 4K image, which limited it to two images per view. fal's Nano Banana 2.1 failed 4 of 10 views with no reason given, and its cost was not observable. GPT Image 2.5 drew clean parts that the audit passed more often than the reviewer did, and its left-side images measured stretched, so keep it off a reference sheet unless its proportions are measured on every image. Jev is worth its very small cost as a logged second opinion on the kickstand. It is never a gate, and a person still approves every image.

7. Limits and costs

• The auditors are AI. Only the reviewer's marks are ground truth, and holds count as neither approved nor rejected here.
• One reviewer, one product, three rounds. The numbers describe this sheet, not image models in general.
• Platform cost was read in each platform's own credits. Credits are not dollars, fal's cost was not observable, and other work was spending on the Higgsfield account during round 3.
• Jev's price is the published price on the retrieval date. Round 3's 1,238,374 input tokens come to $0.052.
• Review time per image is not in the round records we kept.

Questions people ask

Can an AI checklist approve product images on its own? Not in this test. In the round 2 pilot it passed 15 images, and a person rejected all 15.

Which platform was best? With the same model, no clear winner. On the 6 views all three platforms completed, approvals were 8 of 18 on fal, 8 of 20 on Higgsfield and 4 of 9 on Runway. fal's Nano Banana 2.1 failed 4 of 10 views outright, while GPT Image 2.5 on fal completed all 10.

Is Jev useful for image checks? As a cheap second opinion on the kickstand, yes. As a gate, no: it flagged something on every image.

Why did approved references come back unchanged? Our read: when an approved render of the same view is sent as a reference, the model treats it as the answer. 23 Nano Banana 2.1 images were near-copies.

8. Links

• TypeSafe AI, Jev: https://typesafe.ai/
• TypeSafe AI, images: https://docs.typesafe.ai/images
• TypeSafe AI, System One API: https://docs.typesafe.ai/api/system-one.md
• Last checked: 3 November 2026.

Questions, or an image workflow you want checked? Start a project with the team at LARGE FORMAT.