Data and AI toolbox
The TheTechIdea.OilGas.DataAI package. Settings: toolboxes.dataai in the environment.
Legacy document extraction
Old well files - scanned reports, tally sheets, test summaries - hold casing strings, formation tops and pressure tests that never made
it into a database. IDocumentExtractor is the interface an extractor implements to read one document's text and propose values;
the toolbox ships none built in so that a company can plug in whatever suits it - a language model, an OCR reader, a pattern matcher.
The default, NullExtractor, proposes nothing.
An ExtractionProposal is one value read from one place: the kind of record (string, formation top or pressure test), the field, the
value and unit, the extractor's confidence, the document and page, and the text it matched. Proposals that belong to the same record
share a ProposalGroup (a string's name, top, bottom and grade are four proposals of one group).
Nothing is written until a person accepts it. Extracting alone changes nothing. LegacyExtraction.Apply(model, group, acceptedBy,
extractor) is the one place that writes: it creates the record from an accepted group, and logs a single audit entry naming the
person, the extractor, the document, the page and the exact source text - so every value stays traceable to where it came from. It
refuses mismatched groups, quantities without a unit, non-numeric values and unknown fields rather than guessing.
LabelledTextExtractor is a deterministic offline starting point for text that carries clear labels ("Top: 1250 ft", "Formation:
Upper Sand @ 3200 m", "Pressure: 4500 psi"); a language-model extractor will find far more. LegacyExtraction.ForReview(proposals) returns the proposals worth showing, grouped by record: a record is shown when its least
confident field reaches dataai.minimumConfidence (0.3), the most confident first; includeLowConfidence: true shows every record - nothing is
discarded, and showing a proposal never replaces the need to accept it.
Well identity reconciliation
The same well often sits in several systems under different names and identifiers. WellReconciliation.Propose(wells) finds candidate
duplicates - across sources and within one - and says why. WellIdentity.FromModel(source, model) builds the identity of a loaded well.
The evidence, each piece reported with the candidate:
- UWI / API number - equal once punctuation is ignored, or a 10-digit API number that is the start of a 14-digit one;
- name - equal once case, punctuation, spaces and leading zeros are ignored (
Cedar-07=CEDAR 7), or the same well number with a slightly different spelling (Westfield-3/Westfeld-3). The well number must match exactly:Alder-1is notAlder-12; - field - the same field is corroboration;
- location - within
dataai.locationToleranceis corroboration. There is no default tolerance (GPS, digitised-map and survey coordinates differ by company), so location is not used until you set one.
Conflicts count against a match: different UWIs, different fields, or locations beyond ten times the tolerance each multiply the score
down - "KW-0012" in two different fields with different UWIs is two wells, not one. Evidence combines as 1 - product(1 - e), so several weak
signs add up without any single one deciding. Candidates at or above dataai.minimumMatchScore (0.6 - a tuning parameter; confirm every
candidate either way) are listed best first. Blocking on name, UWI and location cells keeps it fast: 3,000 wells with 100 planted duplicates
run in well under a second.
Nothing is merged unconfirmed, and nothing is merged at all. A person's confirmations and rejections go in a WellCrosswalk - a
same-as table, not a data merge. Confirmations are transitive (A = B and B = C make A = C); a rejected pair is not proposed again;
confirming a link that would join two records someone rejected is refused, and rejecting what is confirmed is refused. The crosswalk saves
to JSON with who decided and when.
The labelled set in the tests (five true duplicates - a 10- vs 14-digit API number, two name formats, a spelling variant, a same-source pair, a UWI-only match - and look-alikes: the same name in another field, adjacent well numbers, the same name with different UWIs 7 km apart) is found with no misses and no false candidates. It is synthetic: it shows the method works, but the thresholds should be tuned on your own data before relying on the ranking.
OSDU ingestion
OsduIngest sends the records of the OSDU export (OsduExport.Records, valid against the official OSDU schemas) to a platform's Storage
Service: PUT <storage>/records, the service's upsert, with the headers data-partition-id and Authorization: Bearer <token> and a JSON
array of records as the body; the platform answers with recordCount, recordIds, skippedRecordIds and recordIdVersions. The
caller needs the platform's service.storage.creator permission. Because it is an upsert, sending again is safe.
A dry run is the default. OsduIngest.Plan(records) builds the requests without any network access and checks what the platform would
refuse: no storage address, no data partition, no ACL groups or legal tags, a record id that does not start with the data partition, an
unusable kind, a duplicate id. plan.Describe() shows every request (the token appears only as Bearer <token>). Only
OsduIngest.SendAsync sends, and it refuses a plan that has problems. It sends one request at a time in the records' order - a well's
Wellbore before its trajectory and logs - and stops at the first failure, reporting the platform's error and how many requests were not
sent.
Settings (toolboxes.dataai): osduStorageUrl - the full address of your platform's records endpoint (no default: the path prefix
differs between deployments; https is required, since a token travels with every request, except for a local address) - and
osduBatchSize (100: below every Storage Service batch limit that could be verified - patch 100, metadata update 500; the limit of PUT
itself was not verified, so raise it only against your platform's documentation). The partition, ACL groups and legal tags are the core
osdu section the export already uses. The access token is never part of the configuration: the caller passes it in for the call,
and the command line reads it from the OSDU_TOKEN environment variable.
wellbore-config ingest-osdu environment.json workbook # dry run: shows the requests, sends nothing
OSDU_TOKEN=... wellbore-config ingest-osdu environment.json workbook --send
This was built against the platform's published API documentation and tested with a stand-in server; it has not been run against a live OSDU platform (see the master to-do).
Matching weights
How strongly each kind of evidence counts when two well records are compared is a setting, not a number in the code: toolboxes.dataai.matching holds uwiEqual (0.95), uwiPrefix (0.8),
uwiPrefixMinLength (8), nameEqual (0.6), nameSimilar (0.4), nameSimilarityMinimum (0.85), fieldEqual (0.15), locationWithin (0.5), the three conflict factors
uwiConflictFactor (0.25), fieldConflictFactor (0.5), locationConflictFactor (0.25), and locationFarFactor (10 location tolerances). The defaults are starting points chosen so that the
labelled examples come out right; they are tuning parameters of the method, not facts. Tune them on your own data - run the proposals over wells you already know are duplicates or not, and
adjust until the list a person has to review is short and misses nothing.