PixelProof findings
Verifiable visual tasks for gauges, charts, wiring, counting, layouts and routes · dataset demo · generator v1.0.0 · seed 7
groq:qwen/qwen3.8-27b
baseline:blind-prior
blind:groq:openai/gpt-oss-120b
Model scorecard
Pair accuracy counts a pair only when both twins are right. Twins share the exact question text but have different answers, so text-only guessing scores 0% by construction. Visual sensitivity = share of pairs where the model changed its answer when the decisive detail changed.
| Model | Task acc. | Pair acc. | Chance (task) | Visual sensitivity | Unparsed | Tasks | Tokens in/out |
|---|---|---|---|---|---|---|---|
| groq:qwen/qwen3.8-27b | 50% | 46% | 12% | 90% | 23% | 115 | 176,058 / 84,659 |
| baseline:blind-prior | 33% | 0% | 11% | 0% | 0% | 240 | 0 / 0 |
| blind:groq:openai/gpt-oss-120b | 3% | 0% | 12% | 40% | 8% | 120 | 17,640 / 35,505 |
Where groq:qwen/qwen3.8-27b breaks
Pair accuracy by family and difficulty level. Darker = more pairs fully correct. Hover for counts.
| Family | L0 | L1 | L2 | L3 | L4 | Breaks at |
|---|---|---|---|---|---|---|
| Report chart reading | 100% | 100% | 100% | 100% | 100% | not reached |
| Conjunction counting | 0% | 0% | 0% | 0% | 0% | L0 |
| Analog gauge reading | 100% | 50% | 50% | 0% | 0% | L3 |
| Route planning | 0% | 0% | 0% | 0% | 0% | L0 |
| Spatial relations | 100% | 50% | 50% | 50% | 50% | not reached |
| Wire tracing | 100% | 100% | 100% | 0% | – | L3 |
Difficulty curves
Pair accuracy as each family's difficulty knobs are turned up.
Report chart reading
Conjunction counting
Analog gauge reading
Route planning
Spatial relations
Wire tracing
Validity gauntlet
Every candidate must pass all gates before it can be shown to a model. Rejections per gate (all families):
| Candidates generated | 197 | |
| Rejected · Image carries sufficient evidence (sizes, margins, no ambiguous overlaps) | 55 | |
| Rejected · Independent pixel-level verifier recomputes the same answer from the rendered image | 0 | |
| Rejected · A minimal counterfactual twin exists: same question, different answer, also verified | 9 | |
| Rejected · Not a near-duplicate of an accepted image (perceptual hash) | 13 | |
| Accepted pairs (×2 tasks) | 120 |
Per-family breakdown: chart: 20/30 · count: 20/20 · gauge: 20/33 · maze: 20/23 · spatial: 20/43 · wires: 20/48
Task families
| Family | Real-world application | Visual skill probed | Difficulty knobs |
|---|---|---|---|
| Report chart reading read, rank, diff | Extracting numbers from charts in PDFs, dashboards and scanned reports. | Reading values against axis gridlines, ranking near-equal bars, arithmetic over read values | n_cats: number of categories; series: 1 = simple, 2 = grouped bars with legend; sub: value resolution relative to gridline step (1, 1/2, 1/4); label_every: label every k-th gridline; min_gap_px: smallest gap between ranked bars |
| Conjunction counting count | Counting parts on trays, items on shelves, defects on boards (inventory & QC). | Exact enumeration under clutter; binding colour+shape (conjunction search) | targets: number of target objects; distractors: number of distractors; conjunction: distractors share colour OR shape with target; size: object radius range; overlap: background distractors may overlap; grid: background clutter grid |
| Analog gauge reading read | Reading plant instruments from site photos / CCTV (pressure, temperature, flow). | Fine angular estimation + interpolation between tick marks + scale arithmetic | n_major: number of labelled major intervals; minor: minor ticks per major interval (1 = none); half_steps: value may sit half-way between minor ticks; setpoint: red set-point needle as distractor; sparse_labels: label only every other major tick; offset_zero: scale starts below zero; noise: sensor noise sigma; needle_w: needle width (px) |
| Route planning shortest | Floor-plan / warehouse navigation: shortest route between two marked locations. | Multi-step spatial planning with global constraints (walls), path length counting | grid: maze size n x n; loops: fraction of extra openings (multiple routes); min_dist: minimum shortest-path length |
| Spatial relations nearest, count_rel, order | Layout checks: planogram compliance, site-plan / floor-plan questions, robot pick targets. | Metric distance comparison, relational counting and ordering by position | n_objects: objects in scene; ratio: 2nd-nearest / nearest distance ratio; margin_px: min clearance from decision boundaries; radius: object size |
| Wire tracing trace | Verifying wiring / piping diagrams, patch panels and control-cabinet photos. | Continuous contour following through crossings (a known weak spot of patch-based vision) | n_wires: number of wires; waypoints: bends per wire; coloring: distinct / paired / mono; width: wire thickness (px) |
Failure gallery · groq:qwen/qwen3.8-27b
Verified tasks the model got wrong (57 of 115). Truth comes from the generator and was re-derived from pixels.
How many pink triangles are in the image?
Truth 15 · model None
Twin: truth 13, model None ✗
How many pink triangles are in the image?
Truth 13 · model None
Twin: truth 15, model None ✗
How many teal squares are in the image?
Truth 10 · model None
Twin: truth 8, model None ✗
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
Truth 135.0 ± 1.125 · model 140.0
Twin: truth 45.0, model 60.0 ✗
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
Truth 45.0 ± 1.125 · model 60.0
Twin: truth 135.0, model 140.0 ✗
What value does the black pointer on this temperature gauge indicate, in °C? Ignore the red set-point marker.
Truth 3.0 ± 0.09 · model None
Twin: truth 4.8, model 4.0 ✗
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
Truth 20 · model None
Twin: truth 22, model None ✗
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
Truth 22 · model None
Twin: truth 20, model None ✗
Moving one cell at a time (up, down, left or right) and never crossing a wall, what is the minimum number of moves needed to get from the green cell to the red cell?
Truth 32 · model None
Twin: truth 20, model None ✗
How many objects have their centre above the centre of the brown star?
Truth 6 · model None
Twin: truth 5, model 5.0 ✓
How many objects have their centre above the centre of the pink triangle?
Truth 8 · model None
Twin: truth 9, model 9.0 ✓
Follow the wire that starts at terminal C on the left. Which numbered terminal on the right does it end at?
Truth 7 · model 3.0
Twin: truth 3, model 3.0 ✓
How many objects have their centre left of the centre of the purple star?
Truth 2 · model 5.0
Twin: truth 3, model 5.0 ✗
Honest limitations
- Synthetic, clean renders. Images are procedurally drawn; real photos add perspective, blur and glare. The factory measures a lower bound on real-world difficulty.
- The verifier knows the rendering conventions (palette, dial geometry, terminal positions). It is independent of the evaluated model and of the generator's answer, but not of the drawing style. Changing a renderer means updating its verifier; the test-suite enforces agreement.
- Prompt sensitivity. One fixed prompt and answer format per task; results can move a few points with other prompts. Unparsed answers are reported separately and count as wrong.
- Sample sizes. Per-cell pair counts are small in the demo run; treat single cells as indicative, use the generator to scale up (cost is linear and cached).
- Difficulty levels are ordinal, designed per family; L3 in one family is not calibrated to equal L3 in another.
Reproduce: pixelproof run --seed 7 --pairs 4. Audit: pixelproof audit demo re-derives every answer from pixels and checks image hashes.