Picture a gel. Lane 1 is the marker, lane 3 is the original plasmid, and lanes 4 through 10 are colonies you just picked. Your job is to say which colonies carry the new construct. The AI splits the bands into two groups perfectly. Then it picks the wrong group. If you trusted it, you’d spend the next day purifying the plasmid you were trying to get rid of.

That wrong answer came from GPT-6 Astra, OpenAI’s top-tier model and the best performer on BioBasicsBench, a benchmark OpenAI released on November 6. It’s a useful place to start, because the headline score matters less than the question behind it. AI can summarize papers. Can it be trusted with the messy, half-failed data that actually piles up on a lab bench?

A test built from failed experiments, not paper figures

OpenAI’s pitch is simple. Most science benchmarks are built from charts and images that were cleaned up for publication. Real wet-lab work doesn’t look like that. It’s failed runs, blurry images and observations that don’t agree with each other. BioBasicsBench tries to put that everyday mess into a test.

OpenAI built it with Ligo, an enzyme-design biotech company. Ligo scientists turned their own experiments into 100 tasks and wrote the grading rubrics. Each task went through three reviews: a Ligo scientist who didn’t write it, a PhD-level scientist from an outside wet lab, and a final internal pass. The tasks span five areas: protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning.

The input data tells you what kind of test this is. Here’s a selection from OpenAI’s breakdown (tasks can mix several types, so the counts add up to more than 100):

Data type Tasks Examples
Gels and blots 31 Agarose DNA gels, SDS-PAGE, Western blots
Protocols, records and specs 27 Lab notebooks, photos of handwritten notes, instrument specs
Tables and instrument exports 20 CSV and Excel files, raw flow cytometry (FCS) files
Bench, device and reagent photos 18 Gel rigs, PCR plates, pH strips
Microscopy 14 Brightfield and fluorescence images, z-stacks, time-lapses
Protocol videos 7 Streaking, serial dilution, weighing

Instrument screen photos (13), raw sequencing files (12) and agar plates (7) round out the set.

The grading is where it gets interesting. Answers aren’t just right or wrong. Each rubric gives points for the conclusion, the numbers, the reasoning behind them, and for recognizing when the evidence simply isn’t enough to conclude anything. Saying “I can’t tell from this” is something the benchmark actually rewards.

One scope note. Every task is about reading data and returning an answer. Nobody’s asking the model to pipette or run an instrument. Headlines will say “AI scientist,” but what’s being measured is closer to an AI that reads your results.

The scoreboard, and why token counts matter

OpenAI tested seven of its own models across 37 model and reasoning-effort combinations, with each combination running all 100 tasks five times. That’s 37 × 100 × 5 = 18,500 graded results. A few summaries read this as “7 models × 37 settings” and flagged the total as inconsistent. It isn’t. The 37 is the total number of settings across all models.

Here are the results at “xhigh,” the highest effort level reported for all seven models:

Model Mean reward 95% CI Output tokens per task
GPT-6 Astra 66.7% 60.3–73.1% 4,704
GPT-6.1 Sol 62.0% 55.6–68.4% 5,238
GPT-5.6 Sol 43.8% 37.6–50.0% 9,099
GPT-6 Luna 38.3% 32.3–44.5% 13,120
GPT-5.5 35.4% 29.7–41.0% 11,135
GPT-5.4 33.2% 28.3–38.4% 13,717
GPT-5.2 31.2% 26.2–36.3% 27,620

Run the numbers yourself and a few things jump out. Astra beats GPT-5.6 Sol by 22.9 points while using about 52% of the output tokens. Against GPT-5.2, it uses roughly 17% of the tokens and scores 35.5 points higher. So Astra didn’t win by thinking longer. It got better at reading the images in the first place. OpenAI notes that Astra pulled ahead even at low effort and says stronger native multimodal ability “may be key.” That’s a hypothesis, not a finding.

Scatter plot of output tokens per task vs. mean reward on BioBasicsBench at xhigh effort. GPT-6 Astra scores 66.7% with 4,704 tokens, GPT-6.1 Sol 62.0% with 5,238, and GPT-5.2 31.2% with 27,620

The confidence intervals cut the other way at the top. Astra (60.3–73.1%) and GPT-6.1 Sol (55.6–68.4%) overlap heavily, so with only 100 tasks the 4.7-point gap isn’t a clear win. The gap between Astra and GPT-5.6 Sol, on the other hand, doesn’t overlap at all.

Bar chart of mean reward with 95% confidence intervals for seven models. GPT-6 Astra (60.3–73.1%) and GPT-6.1 Sol (55.6–68.4%) overlap, while GPT-5.6 Sol (37.6–50.0%) sits clearly below

And newer doesn’t automatically mean better. GPT-6 Luna scored 38.3%, below the previous generation’s GPT-5.6 Sol, while burning more tokens. On this test, the tier (Astra, Sol, Luna) mattered more than the version number.

What Astra got better at: knowing what the data can’t say

The three improvement examples OpenAI picked all point the same way.

In a cloning task, GPT-5.6 Sol saw no colonies on the negative control plate and drew its conclusion from that. Astra read the handwriting on the plate’s rim, “LB Kan50.” That’s kanamycin, but the experiment used ampicillin. So Astra threw the plate out as a valid control.

In a fluorescence task, GPT-5.6 Sol labeled cells as dead based on a dead-cell dye. But the two dyes’ detection windows overlapped between 525 and 540 nm, so light from the other dye could have bled through. Astra flagged that and asked for better controls to separate the signals.

In a microscopy video where cells seemed to fade away, GPT-5.6 Sol went looking for a biological explanation. Astra recognized a z-stack, images taken at different focal depths.

None of these are about knowing more biology. They’re about not claiming more than the data shows, which is exactly the kind of mistake that eats weeks of lab time.

How it fails tells you more than the score

To OpenAI’s credit, it also published Astra’s failures. As R&D World quotes the post, “Higher benchmark scores do not yet mean models can make reliable judgments in the lab.”

The colony PCR case from the top is the clearest one. Astra grouped the lanes correctly, then assumed the larger PCR product meant successful cloning. It picked lanes 4, 6, 7 and 8. The reference answer, based on comparing against the controls, was 5, 9 and 10. Right observation, backwards interpretation.

The vacuole case is subtler. Astra looked at several regions and even noted mid-answer that contrast flips with focus. Its final answer still said vacuoles are darker than the surrounding cell. It saw the contradicting evidence and then dropped it.

On a colony count, Astra merged neighbors, missed visible colonies and marked empty agar, ending up about 12% over the reference. With volume and dilution fixed, the transformation-efficiency estimate inflates by the same 12%. One bad count quietly skews everything downstream.

The pattern is “right until the last step.” We saw the same thing in Google’s physical engineering benchmark last week, where the top setup satisfied 91% of individual constraints but only 41.3% of items passed every hard safety check. Different field, same lesson. Collecting partial credit and getting the final call right are different skills.

That’s also why 66.7% doesn’t mean “two out of three tasks solved.” It’s an average of partial-credit scores. An answer that picks the wrong colonies can still earn points for its observations. In a lab, the final pick is what counts, and a mean reward blurs that into one number.

How much should you trust this?

Some context before you quote the score. OpenAI built the benchmark with Ligo, and all seven models tested are OpenAI’s. There are no Claude or Gemini numbers. As of November 12, the announcement doesn’t say whether the tasks or rubrics will be released, so nobody outside can rerun it or test other models yet.

Still, publishing your best model’s mistakes in this much detail is worth something. The post reads less like a launch and more like a ruler for tracking progress. OpenAI also says tracking these skills helps it understand biosafety and biosecurity implications. An AI that reads lab results well can help with risky experiments as easily as useful ones, so it makes sense to measure both with the same ruler.

So what can you hand off today?

Right now, Astra looks like a decent second reader. Double-checking labels and control conditions, or spotting where the data can’t support a conclusion, it already does reasonably well. Decisions that drive the next experiment, like which colonies to grow or what your efficiency number is, still need a human looking at the raw data. If you do use a model this way, ask it to state which controls it based its call on, not just the call itself. That alone would have caught the colony PCR mistake.

The thing to watch is whether BioBasicsBench leaves OpenAI. Once the tasks are public and other labs’ models run under the same conditions, 66.7% stops being a launch-day number and becomes a real baseline.