Eval runs & evidence
Every number on the landing page comes from one canonical eval run or a dated live corpus count. This page holds the method, the full results, including the ones that didn't flatter us, and the limitations we know about.
How we evaluate
The geothermal15 suite runs 15 fixed questions against the production API: 10 corpus-grounded questions (Utah FORGE drilling facts, USGS resource estimates, BLM/NEPA records), 2 refusal traps for information deliberately absent from the corpus, and 3 domain-reasoning questions on scaling, sulfide stress cracking, and chloride SCC. It measures answer correctness, citation rate, grounding, refusal accuracy, and latency in one artifact.
Grading method: the run below was graded by automated pattern checks plus an LLM judge, not yet by an independent human evaluator. Our collaborating corrosion SME authored 15 of the corpus documents, so when SME adjudication is added we will disclose that relationship on every run it touches. The pilot exists precisely so your engineers can replace our grades with theirs.
What grounding measures: how closely an answer's numeric claims stick to the text of its cited sources. That is citation fidelity, not correctness. An answer can be well-grounded in a source and still judged wrong, and vice versa; that is why both numbers are published separately.
Canonical run: geothermal15, 2026-08-19
| Answers judged correct | 12/15 (80.0%) |
| Answers with page-level citations | 15/15 (100%) |
| Average grounding (citation fidelity) | 90% |
| Refusal traps (out-of-corpus questions refused) | 2/2 (n=2) |
| Latency | median 30.4s · P95 42.6s · mean 27.0s |
The three missed questions were factual-recall misses (a casing string enumeration, a bottom-hole temperature, and a NEPA-timeline median), not fabrications. Each failed answer still cited real sources. The suite's automated RAGAS judge malfunctioned in this run (it scored 0.0 across answers that passed every other check), so its numbers are excluded rather than reported.
End to end MVP run: geothermal-mvp-50, 2026-08-22
50 cases against the live production API covering all three agents: 45 Ask questions (including 5 out of corpus refusal traps), 4 Integrity Program generations, and 1 Survey Analyzer review. Dataset: tests/golden-dataset/geothermal-mvp-50.json. Full per case results: docs/evals/geothermal-mvp-50-results.json in the repository.
| Automated pass rate | 45/50 (90%) |
| After human review of the 5 fails | 47/50 (two were grader keyword artifacts; answers were correct) |
| Ask questions passed | 40/45 automated |
| Citation rate (non refusal answers) | 100% |
| Refusal traps caught | 5/5 |
| Ask latency | median 21.6s · P95 40.7s |
| Integrity Program | 4/4 valid programs, 9 to 11 mechanisms each, 108 to 150s |
| Survey Analyzer | 1/1: 10 findings, 7 anomaly flags, 58s on the synthetic UT survey |
| Hallucinations observed | 0 (every miss was an honest refusal or an alternative grounded answer) |
The three real misses were retrieval gaps, not fabrications: a NEPA timeline table, a heat exchanger conductivity table, and a Svartsengi workover detail were not retrieved, and in each case the answer said so plainly. Grading used automated keyword checks authored by the developer with human review of every failure; independent grading is what the pilot exists for.
Production end to end audit, 2026-08-18
A full production user-journey audit (uploads, negative upload cases, gold Q&A, all three agents) passed 12 of 15 checks. The Survey Analyzer read a synthetic casing UT survey (the fictional Obsidian Ridge test set, clearly labeled synthetic) and returned 10 findings and 7 anomaly flags in 56.6s. The same audit caught a production failure in the Integrity Program generator. It is published here because "built to be checked" has to include the misses; the fix is tracked ahead of any pilot.
Corpus & licensing
Live corpus count (2026-08-21): 65 geothermal documents (~1,944 source pages, 4,421 indexed chunks) plus a legacy oil & gas corrosion library, 115 documents and 6,317 chunks in total. The geothermal set breaks down as: 15 SME-authored knowledge documents, 32 open-access papers, 11 public-domain documents with full license provenance (Utah FORGE, USGS, BLM, DOE GeoVision), and the remainder open-literature technical reports (MIT Energy Initiative's Future of Geothermal, IDDP materials investigations, and other public studies). A per-document license manifest is in progress and will be published here.
The corpus contains no licensed AMPP/NACE/API standards texts. Answers reference the applicable clause from open literature and domain knowledge; they do not quote standards documents.
Known limitations
- Citations, grounding verification, and refusal behavior belong to the Ask pipeline. The Integrity Program and Survey Analyzer produce schema-validated structured output but do not emit citations or confidence scores on live runs. Confidence meters shown on those tabs in the demo are demo-mode previews.
- Retrieval is currently global across the whole corpus: uploaded documents join a shared index. Workspace isolation is a named phase-two pilot deliverable, and no proprietary documents are requested before it ships and passes a database-layer isolation test.
- The demo walkthrough replays pre-computed results instantly; live Ask answers take a median 30.4s.
Questions about any number here? Ask us directly →