OpenAI benchmark tests AI agents on raw wet-lab data

Overview

Updated 1 hour ago

OpenAI and the biotech company Ligo turned 100 messy moments from working laboratories, including smeared gel bands and colony plates labeled with the wrong antibiotic, into a test for AI agents. The benchmark, BioBasicsBench, asks a model to read raw experimental evidence and say what it does and does not support.

The best result came from GPT-6 Astra, which earned 66.7% of the available credit at high reasoning effort. GPT-5.6 Sol, the prior generation, earned 43.8%. OpenAI says better image and video handling explains part of the gain; its own examples show the model still misreads controls and overcounts colonies.

Why it matters

An AI assistant that cannot spot a mislabeled control or an unusable image is not ready to help decide which experiments to run next.

Questions about this story

Free account needed to ask — your question is kept and asked for you right after sign-up. Answers are public.

No questions yet — be the first to ask.

Key Indicators

100

Wet-lab tasks in the benchmark

Drawn from real experiments across five areas of laboratory work.

66.7%

Mean reward, GPT-6 Astra

Share of available rubric credit at the highest reasoning setting reported for all seven models.

43.8%

Mean reward, GPT-5.6 Sol

The prior-generation model's score at the same effort setting.

18,500

Graded model results

Seven models, 37 settings, five attempts on each of 100 tasks.

Laboratory areas covered

Protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning.

Voices

Curated perspectives — historical figures and your fellow readers.

People Involved

Sam Altman

Chief executive, OpenAI

Leads OpenAI as it expands from chat models into scientific tools

Organizations Involved

OpenAI

Artificial intelligence company

Released BioBasicsBench with Ligo

An AI company that builds large multimodal models and publishes evaluations of their capabilities.

Ligo

Biotechnology company

Co-developed BioBasicsBench; focuses on enzyme design

A biotech company focused on enzyme design that supplied real experimental data and expert review for the benchmark.

Timeline

3 events Latest: 2 days ago

November 2026
OpenAI and Ligo publish BioBasicsBench
Latest Release

The benchmark has 100 tasks drawn from real experiments across five lab areas.
2 days ago
GPT-6 Astra leads the model field
Result

It earns 66.7% of available credit, against 43.8% for GPT-5.6 Sol.
2 days ago
OpenAI documents remaining failure modes
Finding

Its examples show Astra misreading controls and overcounting colonies by about 12%.
2 days ago

Scenarios

External labs adopt BioBasicsBench as a standard screen

Possible Resolves by Q2 2027

Discussed by: OpenAI and Ligo, which present the benchmark as a measure of routine lab analysis

If the 100 tasks hold up under outside use, biotech teams could run models through the benchmark before trusting them with image review or colony counts. Adoption would show up as published scores from labs that did not build the test.

A model crosses 80% mean reward on BioBasicsBench

Possible Resolves by End of 2027

Discussed by: OpenAI, which reports GPT-6 Astra at 66.7% and says room remains

The score is an average of rubric credit across 100 tasks. A jump above 80% would mean fewer errors on controls, image interpretation and colony counts.

Benchmark expands from analysis to full protocol execution

Unlikely Resolves by End of 2027

Discussed by: RoboActu, which notes the benchmark covers the analysis layer rather than complete experiments

The current tasks stop at interpretation and recommendation. A follow-up could grade an agent on planning a protocol, running controls and handling a failed step.

Historical Context

3 moments from history that rhyme with this story — and how they unfolded.

2009-2012

ImageNet and the 2012 image-recognition competition (2009-2012)

Researchers built a large labeled image dataset and ran an annual competition on it. In 2012, a neural network cut the error rate sharply compared with prior methods.

Then

Teams shifted toward deep learning and the competition became the field's common scoreboard.

Now

Shared benchmarks became the organizing layer for measuring progress in AI.

Why this matters now

BioBasicsBench aims to do for messy lab data what ImageNet did for images: give researchers a common yardstick and a public leaderboard.

2018-2020

CASP protein-structure assessment and AlphaFold (2018-2020)

The Critical Assessment of Protein Structure Prediction is a blind, expert-graded test held every two years. DeepMind's AlphaFold entered in 2018 and 2020, scoring far above prior methods in the 2020 round.

Then

Protein structure prediction moved from a research problem to a practical tool.

Now

The case showed that blind benchmarks and expert grading can shift a scientific field quickly.

Why this matters now

BioBasicsBench uses expert-written rubrics and real experimental data, the same combination that made CASP a credible measure of AI progress in biology.

2004-2005

DARPA Grand Challenge (2004-2005)

The Defense Advanced Research Projects Agency offered a prize for autonomous vehicles to complete a desert course. No team finished in 2004; several teams finished in 2005.

Then

The challenge drew attention and funding to autonomous-vehicle research.

Now

It set a pattern for using concrete, scored tasks to measure machine capability.

Why this matters now

BioBasicsBench is a narrower task set than a desert race, but both convert a broad capability into a score that can be compared over time.