LLM Benchmark

megane can build a pipeline from a natural-language request. bench/llm/ is how that gets measured: a fixed set of prompts, a rubric per prompt, and — for every one of them — a reference pipeline and the picture it draws.

Why a picture

The scorer grades the shape of a generated pipeline: which node types exist, how they are wired, what their parameters say. It never draws anything, so on its own it cannot tell an answer from its opposite. Two examples it could not see:

• A bare filter selects; it does not change what is drawn. Selecting 8 carbons and selecting 1006 oxygens+nitrogens produce pictures that differ from each other by 0.002 % of pixels — the same view, both scoring full marks.
• Atoms and bonds are independent viewport streams. A pipeline that fades a molecule's atoms and leaves its bonds at full opacity looks, to a shape-only rubric, exactly like one that fades both.

So each case carries a rendered reference, and each reference carries at least one counterexample: a named wrong pipeline that must not draw the same picture. A comparison that accepts everything is indistinguishable from one that works.

What a case is made of

Every case is a prompt, a pipeline, and an image, stored together:

bench/llm/dataset.ts                              the prompt and its rubric
bench/llm/golden/<case id>/pipeline.megane.json   a pipeline that answers it
bench/llm/golden/<case id>/expected.png           what that pipeline draws
bench/llm/golden/<case id>/meta.json              fixture, expectation, capture source

The folder name is the case id. Nothing registers a case: the loader discovers the directory and fails loudly if a folder names a prompt that does not exist, and the unit tests fail if any of the three files is missing.

The pipelines are captured, not written. Each one is store.serialize() taken from a graph built in the editor, from a fresh boot. Hand-authoring them is how the first attempt shipped a bond query megane rejects (resname is not a bond field), a bondSource no fixture loads, and a parameter that lives on a different node type — each of which scored full marks against the rubric while drawing nothing.

What the images do and do not prove

A picture is ground truth only if the same pipeline draws it again, so each case records what repeated renders actually produced:

• asserted — the reference reproduced byte-for-byte across independent boots, and the runner compares against it pixel-for-pixel.
• recorded, not asserted — the same pipeline produced one of two renders depending on the session. Across independent boots the store's viewport state is bit-identical (same atom count, same position checksums, same bond count, same mesh vertex and coordinate checksums), the camera returns the same position, target and zoom to the last float, and the visible subsystems agree — so what flips is below the scene, in how coincident geometry rasterises. The image is kept for review; widening the pixel budget until it passed would make the suite green without making it true. The counterexamples are still checked, against a reference rendered in the same page.
• rubric only — no pipeline can draw a different picture for this prompt. A SerializedPipeline does not carry trajectory frames or loaded force vectors, and applying one clears them, so load_trajectory and load_vector draw nothing after the round trip. The product behaves the same way, so grading against a picture it cannot produce would be grading the wrong thing.

Running it

npm run build:app            # the runner renders the built webapp
npx playwright test --project=bench-golden

The runner renders whatever ground truth it finds and asserts both directions: the reference draws its picture, and the counterexamples do not. Scoring a real model against the rubrics is separate and opt-in — the llm-eval label runs .github/workflows/llm-prompt-eval.yml, which makes real, paid API calls and posts a before/after comparison on the pull request.

The cases

case	prompt	fixture	image
molecule-basic	Show a molecule with its chemical bonds.	caffeine_water.pdb	asserted
molecule-no-bonds	Just display the atoms of my structure, no bonds at all.	caffeine_water.pdb	asserted
molecule-trajectory	Load a protein and play its MD trajectory with bonds.	caffeine_water.pdb	rubric only
crystal-polyhedra	Visualize a perovskite crystal with coordination polyhedra around the metal atoms.	perovskite_srtio3_3x3x3.xyz	recorded, not asserted
crystal-distance-bonds	I have a crystal XYZ file without bond records. Show it with distance-based bonds and the unit cell.	perovskite_srtio3_3x3x3.xyz	recorded, not asserted
crystal-polyhedra-exclude	Show coordination polyhedra for an oxide but skip titanium (Z=22) as a center, keep only the others.	perovskite_srtio3_3x3x3.xyz	rubric only
filter-carbon	Display only the carbon atoms of my structure.	caffeine_water.pdb	asserted
filter-residue	Show only the alanine residues (resname ALA).	1ubq.pdb	recorded, not asserted
filter-oxygen-nitrogen	Highlight just the oxygen and nitrogen atoms.	caffeine_water.pdb	asserted
modify-scale	Show the molecule with the atoms drawn at half their normal size.	caffeine_water.pdb	asserted
modify-transparent	Make the structure semi-transparent so I can see through it.	caffeine_water.pdb	asserted
labels-element	Display the structure and label every atom with its element symbol.	caffeine_water.pdb	asserted
vectors-forces	Load forces from a file and draw them as arrows on each atom.	caffeine_water.pdb	rubric only
multistep-filter-bonds	Show my structure with bonds, but only the carbon atoms, and label them with their element.	caffeine_water.pdb	asserted
multistep-water-transparent	I have a caffeine molecule dissolved in water, where the water residues are named HOH. Keep the caffeine fully visible, but make the water molecules semi-transparent.	caffeine_water.pdb	asserted
representation-water-line	I have a caffeine molecule dissolved in water, where the water residues are named HOH. Render the water molecules as a line representation, but leave the caffeine in its normal style.	caffeine_water.pdb	asserted
hide-water	I have a caffeine molecule dissolved in water, where the water residues are named HOH. Hide the water so only the caffeine is shown.	caffeine_water.pdb	asserted
molecule-basic-ja	分子を結合付きで表示してください。	caffeine_water.pdb	asserted
filter-carbon-ja	炭素原子だけを表示して。	caffeine_water.pdb	asserted
solid-supercell	Show a 2x2x2 supercell of my crystal structure.	perovskite_srtio3_3x3x3.xyz	recorded, not asserted
color-by-element	Color the atoms by their element.	caffeine_water.pdb	asserted
representation-cartoon	Show my protein as a cartoon.	1ubq.pdb	recorded, not asserted
surface-molecular	Show the molecular surface of my structure.	quartz_sio2_2x2x2.xyz	recorded, not asserted
volumetric-isosurface	Load a cube file and render its isosurface.	caffeine.sdf	rubric only