TheTechIdea OilGas - Developer guide User guide →

TheTechIdea.OilGas.DataAI

20 public types. Summaries come from the XML documentation of the compiled assembly.

TheTechIdea.OilGas.DataAI

TypeSummary
CrosswalkLinkclassA person's confirmation that two records are the same well.
CrosswalkRejectionclassA person's decision that two records are different wells (they are not proposed again).
DataAISettingsclassSettings of the data and AI toolbox (environment section toolboxes.dataai).
DataAIToolboxstatic classData and AI toolbox: legacy document extraction, well identity reconciliation, OSDU ingestion.
ExtractedKindenumKind of record a proposal fills.
ExtractionProposalclassOne value an extractor read from a document, proposed for one field of one record - never applied until a person accepts it. A tubular string's top and bottom, for instance, are two separate proposals (ProposalGroup links the ones that belong together).
IDocumentExtractorinterfaceA source of extraction proposals (a language model, an OCR/regex reader, or a company's own tool) behind an interface so none is built into the toolbox; NullExtractor is the default and proposes nothing. Extractors never write to a model - they only propose; LegacyExtraction is what applies an accepted proposal.
LabelledTextExtractorclassA deterministic, offline extractor for text that already carries clear field labels (e.g. "Top: 1250 ft", "Formation: Upper Sand @ 3200 m") - a starting point and a fallback when no language model is configured; a real document extractor (a language model behind IDocumentExtractor) will find far more than this can.
LegacyExtractionstatic classApplies accepted extraction proposals to a well model: creates the record of a group's proposals, sets each field, and logs one audit entry per record naming the extractor, the document, the page and the source text - so every value stays traceable to where it came from. Nothing here decides what to extract; it only ever applies what a person has already accepted.
MatchingSettingsclassHow strongly each kind of evidence counts when two well records are compared (toolboxes.dataai.matching). The defaults are starting points chosen so that the labelled examples in the tests come out right - they are tuning parameters of the method, not facts: tune them on your own data.
MergeCandidateclassA proposed duplicate: two records, a score, and every piece of evidence for and against - never merged until confirmed.
NullExtractorclassThe default extractor: proposes nothing (a host supplies a real one - a language model or a pattern reader).
OsduBatchOutcomeclassThe outcome of one request.
OsduIngeststatic classIngestion of OSDU records (OsduExport.Records) to a platform's Storage Service: PUT {storage}/records, the upsert of the OSDU Storage Service API (headers data-partition-id and Authorization: Bearer; body a JSON array of records; response recordCount / recordIds / skippedRecordIds / recordIdVersions; needs the service.storage.creator permission). JsonObject}) builds the requests without touching the network - that is the default, and what a dry run shows; only CancellationToken) sends, and only for a plan with no problems. The access token is passed in for the call and never stored, logged or part of any result.
OsduIngestPlanclassWhat an ingestion would do - built without touching the network.
OsduIngestResultclassThe outcome of an ingestion.
OsduRequestclassOne request the ingestion would make: PUT to the Storage Service records endpoint with a batch of records.
WellCrosswalkclassThe record of what people decided: confirmed same-as links (transitive - A = B and B = C make A = C) and rejected pairs. No well data is merged or deleted; the crosswalk says which records are the same well so a front end or loader can treat them as one. Confirming a link that would join two records a person rejected is refused, never silently applied.
WellIdentityclassA well as one source lists it.
WellReconciliationstatic classFinds candidate duplicate wells across (and within) sources, and explains why. Evidence: UWI (equal after removing punctuation, or a 10-digit API number that is the start of a 14-digit one); name (equal once punctuation, case and leading zeros are ignored, or the same well number with a name spelled slightly differently - the well NUMBER must match exactly: Alder-1 is not Alder-12); field; and location within the tolerance. Conflicts (different UWIs, different fields, far apart) reduce the score. Evidence combines as 1 - product(1 - e), and every conflict multiplies the result down. Blocking keeps it fast on large sets.

A person's confirmation that two records are the same well.

MemberSummary
string A { get; set; }Key of the first record (source|id).
string B { get; set; }Key of the second record.
string ConfirmedBy { get; set; }Who confirmed.
List<string> Reasons { get; set; }The evidence the person saw.
double Score { get; set; }The proposal's score at confirmation.
DateTime Utc { get; set; }When (UTC).

CrosswalkRejection class

A person's decision that two records are different wells (they are not proposed again).

MemberSummary
string A { get; set; }Key of the first record.
string B { get; set; }Key of the second record.
string RejectedBy { get; set; }Who decided.
DateTime Utc { get; set; }When (UTC).

DataAISettings class

Settings of the data and AI toolbox (environment section toolboxes.dataai).

MemberSummary
Measure LocationTolerance { get; set; }Well reconciliation: two records closer than this count as agreeing in location (evidence); farther than ten times it count as conflicting. None = location is not used (a tolerance is a data-quality judgement of each company - GPS, digitised map and survey coordinates differ).
MatchingSettings Matching { get; set; }Well reconciliation: the weight of each kind of evidence.
double MinimumConfidence { get; set; }Minimum confidence an extraction proposal must have to be shown at all (below it, a proposal still exists but is filtered by default views); does not change whether acceptance is still required.
double MinimumMatchScore { get; set; }Well reconciliation: lowest score listed as a candidate duplicate (a tuning parameter - lower finds more, higher is stricter; confirm every candidate either way).
int OsduBatchSize { get; set; }OSDU ingestion: records per PUT request. 100 is below every Storage Service batch limit verified (patch 100, metadata update 500); the limit of PUT itself was not verified, so raise it only against your platform's documentation.
string OsduStorageUrl { get; set; }OSDU ingestion: the full address of the platform's Storage Service records endpoint (e.g. https://host/api/storage/v2/records - the path prefix differs between deployments, so there is no default). Empty = ingestion is not configured. The access token is never stored here.

DataAIToolbox static class

Data and AI toolbox: legacy document extraction, well identity reconciliation, OSDU ingestion.

MemberSummary
static DataAISettings Settings { get; }The toolbox's settings in the current environment.
const string Name = "dataai"Section name in the environment's toolboxes.
static void Register()Registers the toolbox with the environment (idempotent); hosts call it at start-up, the toolbox's own entry points call it too.

ExtractedKind enum

Kind of record a proposal fills.

MemberSummary
TubularStringA casing or tubing string (tally).
FormationTopA formation top.
PressureTestA pressure test.

ExtractionProposal class

One value an extractor read from a document, proposed for one field of one record - never applied until a person accepts it. A tubular string's top and bottom, for instance, are two separate proposals (ProposalGroup links the ones that belong together).

MemberSummary
double Confidence { get; set; }The extractor's confidence (0..1); not a guarantee - the person still decides.
string Document { get; set; }Document the value was read from.
string Field { get; set; }Field of that record (e.g. "Top", "Name", "Depth").
ExtractedKind Kind { get; set; }What kind of record this fills.
int Page { get; set; }Page within the document (1-based; 0 = not paginated / not applicable).
string ProposalGroup { get; set; }Groups proposals that belong to the same record (e.g. a string's name, top, bottom, grade): accept or reject together.
string SourceText { get; set; }The text the extractor matched, for the person to check against the document.
string Unit { get; set; }Unit of the value, where the field is a quantity (e.g. "ft", "m"); null for text fields.
string Value { get; set; }The value as text (parsed to the field's type on acceptance).

IDocumentExtractor interface

A source of extraction proposals (a language model, an OCR/regex reader, or a company's own tool) behind an interface so none is built into the toolbox; NullExtractor is the default and proposes nothing. Extractors never write to a model - they only propose; LegacyExtraction is what applies an accepted proposal.

MemberSummary
string Name { get; }Extractor name, recorded against every accepted value for provenance.
IEnumerable<ExtractionProposal> Extract(string documentName, string text)Proposals read from one document's text (and an identifying name for it, e.g. a file name).

LabelledTextExtractor class

A deterministic, offline extractor for text that already carries clear field labels (e.g. "Top: 1250 ft", "Formation: Upper Sand @ 3200 m") - a starting point and a fallback when no language model is configured; a real document extractor (a language model behind IDocumentExtractor) will find far more than this can.

Inherits / implements: IDocumentExtractor

MemberSummary
string Name { get; }Extractor name, recorded against every accepted value for provenance.
IEnumerable<ExtractionProposal> Extract(string documentName, string text)Proposals read from one document's text (and an identifying name for it, e.g. a file name).

LegacyExtraction static class

Applies accepted extraction proposals to a well model: creates the record of a group's proposals, sets each field, and logs one audit entry per record naming the extractor, the document, the page and the source text - so every value stays traceable to where it came from. Nothing here decides what to extract; it only ever applies what a person has already accepted.

MemberSummary
static void Apply(WellModel model, IReadOnlyList<ExtractionProposal> group, string acceptedBy, IDocumentExtractor source)Applies one group of accepted proposals (same ProposalGroup, same Kind) to the model as a single new record, audited once.
static List<List<ExtractionProposal>> ForReview(IEnumerable<ExtractionProposal> proposals, bool includeLowConfidence = false)The proposals worth showing a person, grouped by record: a record is shown when its least confident field reaches dataai.minimumConfidence, the most confident first. Nothing is discarded - pass includeLowConfidence to see every record, lowest last.

MatchingSettings class

How strongly each kind of evidence counts when two well records are compared (toolboxes.dataai.matching). The defaults are starting points chosen so that the labelled examples in the tests come out right - they are tuning parameters of the method, not facts: tune them on your own data.

MemberSummary
double FieldConflictFactor { get; set; }The score is multiplied by this when both records name a field and the fields differ.
double FieldEqual { get; set; }Evidence when both records name the same field.
double LocationConflictFactor { get; set; }The score is multiplied by this when the wellheads are far apart.
double LocationFarFactor { get; set; }How many location tolerances apart two wellheads must be to count as far apart.
double LocationWithin { get; set; }Evidence when the wellheads are within locationTolerance of each other.
double NameEqual { get; set; }Evidence when the names are equal ignoring case, punctuation, spaces and leading zeros.
double NameSimilar { get; set; }The most evidence a slightly different spelling of the same well number can give (scaled by how close the spellings are).
double NameSimilarityMinimum { get; set; }How alike two spellings must be (1 = identical) to count at all.
double UwiConflictFactor { get; set; }The score is multiplied by this when both records have UWIs and they differ.
double UwiEqual { get; set; }Evidence when the two UWIs are equal once punctuation is ignored.
double UwiPrefix { get; set; }Evidence when one UWI is the start of the other (a 10-digit API number and its 14-digit form).
int UwiPrefixMinLength { get; set; }The shortest UWI that may count as the start of another (and the length of the prefix used to find candidates).

MergeCandidate class

A proposed duplicate: two records, a score, and every piece of evidence for and against - never merged until confirmed.

MemberSummary
WellIdentity A { get; set; }First record.
WellIdentity B { get; set; }Second record.
List<string> Conflicts { get; }Evidence that they are not (each reduces the score).
List<string> Reasons { get; }Evidence that they are the same well.
double Score { get; set; }Score 0..1 (evidence combined, reduced by conflicts).

NullExtractor class

The default extractor: proposes nothing (a host supplies a real one - a language model or a pattern reader).

Inherits / implements: IDocumentExtractor

MemberSummary
string Name { get; }Extractor name, recorded against every accepted value for provenance.
IEnumerable<ExtractionProposal> Extract(string documentName, string text)Proposals read from one document's text (and an identifying name for it, e.g. a file name).

OsduBatchOutcome class

The outcome of one request.

MemberSummary
int Batch { get; set; }Request number (1-based).
string Error { get; set; }The platform's error (truncated) when the request failed.
bool Ok { get; set; }Whether the request succeeded.
int RecordCount { get; set; }recordCount of the response.
List<string> RecordIds { get; }recordIds of the response (created or updated).
List<string> SkippedRecordIds { get; }skippedRecordIds of the response (unchanged records).
int StatusCode { get; set; }HTTP status code.

OsduIngest static class

Ingestion of OSDU records (OsduExport.Records) to a platform's Storage Service: PUT {storage}/records, the upsert of the OSDU Storage Service API (headers data-partition-id and Authorization: Bearer; body a JSON array of records; response recordCount / recordIds / skippedRecordIds / recordIdVersions; needs the service.storage.creator permission). JsonObject}) builds the requests without touching the network - that is the default, and what a dry run shows; only CancellationToken) sends, and only for a plan with no problems. The access token is passed in for the call and never stored, logged or part of any result.

MemberSummary
static OsduIngestPlan Plan(IReadOnlyList<JsonObject> records)The requests that would ingest these records, and the problems that would stop it (never sends anything).
static Task<OsduIngestResult> SendAsync(OsduIngestPlan plan, HttpClient http, string accessToken, CancellationToken cancellation = null)Sends a plan, one request at a time, stopping at the first failure (the PUT is an upsert, so running it again is safe). Refuses a plan with problems. The token is used for the Authorization header only.

OsduIngestPlan class

What an ingestion would do - built without touching the network.

MemberSummary
bool CanSend { get; }Whether there is nothing to stop the plan being sent.
List<string> Problems { get; }Why this cannot be sent (missing partition, ACL or legal tags, wrong ids ...); empty when it can.
int RecordCount { get; }Records in all requests.
List<OsduRequest> Requests { get; }The requests, in order (records keep their order, so a well's Wellbore is sent before its trajectory and logs).
string Describe()A readable account of the plan; the access token is never shown.

OsduIngestResult class

The outcome of an ingestion.

MemberSummary
List<OsduBatchOutcome> Batches { get; }Per-request outcomes (up to and including the first failure).
int NotSent { get; set; }Requests not sent because an earlier one failed.
bool Succeeded { get; }Whether every request was sent and succeeded.
List<string> Warnings { get; }Things to look at even when it succeeded (a response that does not account for every record).

OsduRequest class

One request the ingestion would make: PUT to the Storage Service records endpoint with a batch of records.

MemberSummary
string Body { get; set; }The request body: a JSON array of records.
Dictionary<string, string> Headers { get; }Headers other than the access token (data-partition-id, Content-Type).
string Method { get; set; }HTTP method (PUT: the Storage Service upsert).
List<string> RecordIds { get; }Ids of the records in the body, in order.
Uri Url { get; set; }The records endpoint.

WellCrosswalk class

The record of what people decided: confirmed same-as links (transitive - A = B and B = C make A = C) and rejected pairs. No well data is merged or deleted; the crosswalk says which records are the same well so a front end or loader can treat them as one. Confirming a link that would join two records a person rejected is refused, never silently applied.

MemberSummary
List<CrosswalkLink> Links { get; set; }Confirmed links.
List<CrosswalkRejection> Rejections { get; set; }Rejected pairs.
bool AreSame(string a, string b)Whether two records are confirmed to be the same well.
string CanonicalKey(string key)The key every record of a confirmed group maps to (the first in order); a record in no group maps to itself.
void Confirm(MergeCandidate candidate, string confirmedBy)Records a person's confirmation of a candidate. Throws when it would join records a person rejected.
static WellCrosswalk FromJson(string json)Reads a crosswalk from JSON.
List<List<string>> Groups()Equivalence classes of confirmed records (size 2 or more), each ordered, classes ordered by their first key.
bool IsRejected(string a, string b)Whether a person rejected this exact pair.
void Reject(MergeCandidate candidate, string rejectedBy)Records a person's decision that a candidate is not the same well.
string ToJson()The crosswalk as JSON.

WellIdentity class

A well as one source lists it.

MemberSummary
string Field { get; set; }Field.
string Id { get; set; }Id within the source.
string Key { get; }Unique key of the record across sources.
double? Latitude { get; set; }Wellhead latitude (deg).
double? Longitude { get; set; }Wellhead longitude (deg).
string Name { get; set; }Well name.
string Operator { get; set; }Operator.
string Source { get; set; }Source system / data source name.
string Uwi { get; set; }UWI / API number as recorded.
static WellIdentity FromModel(string source, WellModel m)The identity of a loaded well model.

WellReconciliation static class

Finds candidate duplicate wells across (and within) sources, and explains why. Evidence: UWI (equal after removing punctuation, or a 10-digit API number that is the start of a 14-digit one); name (equal once punctuation, case and leading zeros are ignored, or the same well number with a name spelled slightly differently - the well NUMBER must match exactly: Alder-1 is not Alder-12); field; and location within the tolerance. Conflicts (different UWIs, different fields, far apart) reduce the score. Evidence combines as 1 - product(1 - e), and every conflict multiplies the result down. Blocking keeps it fast on large sets.

MemberSummary
static string NameKey(string name)The key two names are compared by: letters upper-cased, punctuation and spaces dropped, leading zeros of each number dropped ("Cedar-07" and "CEDAR 7" are both CEDAR7).
static List<MergeCandidate> Propose(IReadOnlyList<WellIdentity> wells, WellCrosswalk crosswalk = null)Candidate duplicates at or above dataai.minimumMatchScore, best first; pairs already decided in the crosswalk are skipped.
static MergeCandidate Score(WellIdentity a, WellIdentity b, double? toleranceMetres)The score and evidence for one pair (null when there is no evidence at all).