PixelProof findings

Verifiable visual tasks for gauges, charts, wiring, counting, layouts and routes · dataset demo · generator v1.0.0 · seed 7

240
tasks (120 counterfactual pairs)
6 × 5
task families × difficulty levels
197
candidates screened by the validity gauntlet
100%
shipped tasks whose answer was re-derived from pixels by an independent verifier
46%
pair accuracy · vision model
groq:qwen/qwen3.8-27b
0%
pair accuracy · text-only control
baseline:blind-prior
0%
pair accuracy · text-only control
blind:groq:openai/gpt-oss-120b

Model scorecard

Pair accuracy counts a pair only when both twins are right. Twins share the exact question text but have different answers, so text-only guessing scores 0% by construction. Visual sensitivity = share of pairs where the model changed its answer when the decisive detail changed.

ModelTask acc.Pair acc.Chance (task)Visual sensitivityUnparsedTasksTokens in/out
groq:qwen/qwen3.8-27b50%46%12%90%23%115176,058 / 84,659
baseline:blind-prior33%0%11%0%0%2400 / 0
blind:groq:openai/gpt-oss-120b3%0%12%40%8%12017,640 / 35,505

Where groq:qwen/qwen3.8-27b breaks

Pair accuracy by family and difficulty level. Darker = more pairs fully correct. Hover for counts.

FamilyL0L1L2L3L4Breaks at
Report chart reading100%100%100%100%100%not reached
Conjunction counting0%0%0%0%0%L0
Analog gauge reading100%50%50%0%0%L3
Route planning0%0%0%0%0%L0
Spatial relations100%50%50%50%50%not reached
Wire tracing100%100%100%0%–L3

Difficulty curves

Pair accuracy as each family's difficulty knobs are turned up.

groq:qwen/qwen3.8-27bbaseline:blind-priorblind:groq:openai/gpt-oss-120b

Report chart reading

0%50%100%L0L1L2L3L4groq:qwen/qwen3.8-27b · chart L0: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · chart L1: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · chart L2: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · chart L3: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · chart L4: pair acc 100%, task acc 100% (n=2 pairs)baseline:blind-prior · chart L0: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · chart L1: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · chart L2: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · chart L3: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · chart L4: pair acc 0%, task acc 50% (n=4 pairs)blind:groq:openai/gpt-oss-120b · chart L0: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · chart L1: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · chart L2: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · chart L3: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · chart L4: pair acc 0%, task acc 0% (n=2 pairs)

Conjunction counting

0%50%100%L0L1L2L3L4groq:qwen/qwen3.8-27b · count L0: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · count L1: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · count L2: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · count L3: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · count L4: pair acc 0%, task acc 0% (n=2 pairs)baseline:blind-prior · count L0: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · count L1: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · count L2: pair acc 0%, task acc 38% (n=4 pairs)baseline:blind-prior · count L3: pair acc 0%, task acc 38% (n=4 pairs)baseline:blind-prior · count L4: pair acc 0%, task acc 38% (n=4 pairs)blind:groq:openai/gpt-oss-120b · count L0: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · count L1: pair acc 0%, task acc 25% (n=2 pairs)blind:groq:openai/gpt-oss-120b · count L2: pair acc 0%, task acc 25% (n=2 pairs)blind:groq:openai/gpt-oss-120b · count L3: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · count L4: pair acc 0%, task acc 0% (n=2 pairs)

Analog gauge reading

0%50%100%L0L1L2L3L4groq:qwen/qwen3.8-27b · gauge L0: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · gauge L1: pair acc 50%, task acc 50% (n=2 pairs)groq:qwen/qwen3.8-27b · gauge L2: pair acc 50%, task acc 75% (n=2 pairs)groq:qwen/qwen3.8-27b · gauge L3: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · gauge L4: pair acc 0%, task acc 0% (n=2 pairs)baseline:blind-prior · gauge L0: pair acc 0%, task acc 38% (n=4 pairs)baseline:blind-prior · gauge L1: pair acc 0%, task acc 0% (n=4 pairs)baseline:blind-prior · gauge L2: pair acc 0%, task acc 12% (n=4 pairs)baseline:blind-prior · gauge L3: pair acc 0%, task acc 25% (n=4 pairs)baseline:blind-prior · gauge L4: pair acc 0%, task acc 25% (n=4 pairs)blind:groq:openai/gpt-oss-120b · gauge L0: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · gauge L1: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · gauge L2: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · gauge L3: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · gauge L4: pair acc 0%, task acc 0% (n=2 pairs)

Route planning

0%50%100%L0L1L2L3L4groq:qwen/qwen3.8-27b · maze L0: pair acc 0%, task acc 25% (n=2 pairs)groq:qwen/qwen3.8-27b · maze L1: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · maze L2: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · maze L3: pair acc 0%, task acc 0% (n=2 pairs)groq:qwen/qwen3.8-27b · maze L4: pair acc 0%, task acc 0% (n=2 pairs)baseline:blind-prior · maze L0: pair acc 0%, task acc 38% (n=4 pairs)baseline:blind-prior · maze L1: pair acc 0%, task acc 12% (n=4 pairs)baseline:blind-prior · maze L2: pair acc 0%, task acc 0% (n=4 pairs)baseline:blind-prior · maze L3: pair acc 0%, task acc 0% (n=4 pairs)baseline:blind-prior · maze L4: pair acc 0%, task acc 0% (n=4 pairs)blind:groq:openai/gpt-oss-120b · maze L0: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · maze L1: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · maze L2: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · maze L3: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · maze L4: pair acc 0%, task acc 0% (n=2 pairs)

Spatial relations

0%50%100%L0L1L2L3L4groq:qwen/qwen3.8-27b · spatial L0: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · spatial L1: pair acc 50%, task acc 50% (n=2 pairs)groq:qwen/qwen3.8-27b · spatial L2: pair acc 50%, task acc 50% (n=2 pairs)groq:qwen/qwen3.8-27b · spatial L3: pair acc 50%, task acc 75% (n=2 pairs)groq:qwen/qwen3.8-27b · spatial L4: pair acc 50%, task acc 75% (n=2 pairs)baseline:blind-prior · spatial L0: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · spatial L1: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · spatial L2: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · spatial L3: pair acc 0%, task acc 38% (n=4 pairs)baseline:blind-prior · spatial L4: pair acc 0%, task acc 38% (n=4 pairs)blind:groq:openai/gpt-oss-120b · spatial L0: pair acc 0%, task acc 25% (n=2 pairs)blind:groq:openai/gpt-oss-120b · spatial L1: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · spatial L2: pair acc 0%, task acc 25% (n=2 pairs)blind:groq:openai/gpt-oss-120b · spatial L3: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · spatial L4: pair acc 0%, task acc 0% (n=2 pairs)

Wire tracing

0%50%100%L0L1L2L3L4groq:qwen/qwen3.8-27b · wires L0: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · wires L1: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · wires L2: pair acc 100%, task acc 100% (n=2 pairs)groq:qwen/qwen3.8-27b · wires L3: pair acc 0%, task acc 67% (n=1 pairs)baseline:blind-prior · wires L0: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · wires L1: pair acc 0%, task acc 50% (n=4 pairs)baseline:blind-prior · wires L2: pair acc 0%, task acc 25% (n=4 pairs)baseline:blind-prior · wires L3: pair acc 0%, task acc 25% (n=4 pairs)baseline:blind-prior · wires L4: pair acc 0%, task acc 12% (n=4 pairs)blind:groq:openai/gpt-oss-120b · wires L0: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · wires L1: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · wires L2: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · wires L3: pair acc 0%, task acc 0% (n=2 pairs)blind:groq:openai/gpt-oss-120b · wires L4: pair acc 0%, task acc 0% (n=2 pairs)

Validity gauntlet

Every candidate must pass all gates before it can be shown to a model. Rejections per gate (all families):

Candidates generated197
Rejected · Image carries sufficient evidence (sizes, margins, no ambiguous overlaps)55
Rejected · Independent pixel-level verifier recomputes the same answer from the rendered image0
Rejected · A minimal counterfactual twin exists: same question, different answer, also verified9
Rejected · Not a near-duplicate of an accepted image (perceptual hash)13
Accepted pairs (×2 tasks)120

Per-family breakdown: chart: 20/30 · count: 20/20 · gauge: 20/33 · maze: 20/23 · spatial: 20/43 · wires: 20/48

Task families

FamilyReal-world applicationVisual skill probedDifficulty knobs
Report chart reading
read, rank, diff
Extracting numbers from charts in PDFs, dashboards and scanned reports.Reading values against axis gridlines, ranking near-equal bars, arithmetic over read valuesn_cats: number of categories; series: 1 = simple, 2 = grouped bars with legend; sub: value resolution relative to gridline step (1, 1/2, 1/4); label_every: label every k-th gridline; min_gap_px: smallest gap between ranked bars
Conjunction counting
count
Counting parts on trays, items on shelves, defects on boards (inventory & QC).Exact enumeration under clutter; binding colour+shape (conjunction search)targets: number of target objects; distractors: number of distractors; conjunction: distractors share colour OR shape with target; size: object radius range; overlap: background distractors may overlap; grid: background clutter grid
Analog gauge reading
read
Reading plant instruments from site photos / CCTV (pressure, temperature, flow).Fine angular estimation + interpolation between tick marks + scale arithmeticn_major: number of labelled major intervals; minor: minor ticks per major interval (1 = none); half_steps: value may sit half-way between minor ticks; setpoint: red set-point needle as distractor; sparse_labels: label only every other major tick; offset_zero: scale starts below zero; noise: sensor noise sigma; needle_w: needle width (px)
Route planning
shortest
Floor-plan / warehouse navigation: shortest route between two marked locations.Multi-step spatial planning with global constraints (walls), path length countinggrid: maze size n x n; loops: fraction of extra openings (multiple routes); min_dist: minimum shortest-path length
Spatial relations
nearest, count_rel, order
Layout checks: planogram compliance, site-plan / floor-plan questions, robot pick targets.Metric distance comparison, relational counting and ordering by positionn_objects: objects in scene; ratio: 2nd-nearest / nearest distance ratio; margin_px: min clearance from decision boundaries; radius: object size
Wire tracing
trace
Verifying wiring / piping diagrams, patch panels and control-cabinet photos.Continuous contour following through crossings (a known weak spot of patch-based vision)n_wires: number of wires; waypoints: bends per wire; coloring: distinct / paired / mono; width: wire thickness (px)

Failure gallery · groq:qwen/qwen3.8-27b

Verified tasks the model got wrong (57 of 115). Truth comes from the generator and was re-derived from pixels.

How many pink triangles are in the image?
count · L4 · unparsed
How many pink triangles are in the image?
Truth 15 · model None
Twin: truth 13, model None ✗
How many pink triangles are in the image?
count · L4 · unparsed
How many pink triangles are in the image?
Truth 13 · model None
Twin: truth 15, model None ✗
How many teal squares are in the image?
count · L4 · unparsed
How many teal squares are in the image?
Truth 10 · model None
Twin: truth 8, model None ✗
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
gauge · L4 · wrong
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
Truth 135.0 ± 1.125 · model 140.0
Twin: truth 45.0, model 60.0 ✗
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
gauge · L4 · wrong
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
Truth 45.0 ± 1.125 · model 60.0
Twin: truth 135.0, model 140.0 ✗
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
gauge · L4 · unparsed
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
Truth 3.0 ± 0.09 · model None
Twin: truth 4.8, model 4.0 ✗
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
maze · L4 · unparsed
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
Truth 20 · model None
Twin: truth 22, model None ✗
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
maze · L4 · unparsed
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
Truth 22 · model None
Twin: truth 20, model None ✗
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
maze · L4 · unparsed
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
Truth 32 · model None
Twin: truth 20, model None ✗
How many objects have their centre above the centre of the brown star?
spatial · L4 · unparsed
How many objects have their centre above the centre of the brown star?
Truth 6 · model None
Twin: truth 5, model 5.0 ✓
How many objects have their centre above the centre of the pink triangle?
spatial · L3 · unparsed
How many objects have their centre above the centre of the pink triangle?
Truth 8 · model None
Twin: truth 9, model 9.0 ✓
Follow the wire that starts at terminal C on the left. Which numbered terminal on the right does it end at?
wires · L3 · wrong
Follow the wire that starts at terminal C on the left. Which numbered terminal on the right does it end at?
Truth 7 · model 3.0
Twin: truth 3, model 3.0 ✓
How many objects have their centre left of the centre of the purple star?
spatial · L2 · wrong
How many objects have their centre left of the centre of the purple star?
Truth 2 · model 5.0
Twin: truth 3, model 5.0 ✗

Honest limitations

Reproduce: pixelproof run --seed 7 --pairs 4. Audit: pixelproof audit demo re-derives every answer from pixels and checks image hashes.