OpenAI and Ironclad evaluate contracting agents

Research covers 11 tasks.

Main report

OpenAI’s October 6 study with Ironclad reports a 55% average rubric score for Astra versus 41.6% for GPT-5.6 Sol across 11 research tasks. A rubric score is not a rate of completely correct contracts.

The comparison applies to the evaluated tasks. It does not establish hours saved or correct processing rates for an arbitrary business. Deployment requires testing the organization’s own rules, data and operating conditions.

Analysis

TBN Grow analysis: a contracting agent must preserve business rules throughout a workflow. Correct clicks and completed forms are insufficient if final approvals are configured incorrectly.

Consider a hypothetical purchase that needs approval above a spending threshold. A useful test includes requests below and above that threshold, plus incomplete submissions. Judge whether the completed process works, not merely whether individual actions look plausible.

What to watch

The reported time figures are simulated estimates, not measured customer savings. Keep that distinction when interpreting the study.

Start in a test environment, limit editing permissions and retain activity logs. Important decisions need a responsible human reviewer. We suggest measuring exception handling, recovery after errors and the effort required for human checking before extending autonomy.