Home · Work

Systems Work · No. 25

Contract Portfolio Extraction

Rajesh Ranjan Mahapatra

A pipeline that reads hundred-plus-page long-form legal documents across a large back-book, extracts terms, obligations and rights with a page-level citation for every field, and routes anything uncertain to a human exception queue.

Start reading
Systems WorkCase study

© 2026 Rajesh Ranjan Mahapatra. 5 min read

Contract Portfolio ExtractionAll Systems Work
Home · Work

Contract Portfolio Extraction

A pipeline that reads hundred-plus-page long-form legal documents across a large back-book, extracts terms, obligations and rights with a page-level citation for every field, and routes anything uncertain to a human exception queue.

Challenge

A large regulated enterprise held a large back-book of facility agreements, security documents and legal side letters, many running past a hundred pages, most existing only as scanned paper.

Architectural move

Extraction is anchored to structure before models run.

Outcome

The pipeline was taken through build and assurance by the programme at a large regulated enterprise, and the extracted portfolio store became the substrate for obligations monitoring and portfolio-wide term queries that were previously manual archaeology.

FIG.00 // SYSTEM MAPSTORE / RECORDHUMAN AUTHORITYAGENTDETERMINISTICSIGNAL / DATAIN · 01Back-book documentsfacility agreements,security documents, sideletters; mostly scannedDET · 02Managed textextractionlayout and table awareDET · 03Clause segmentationclause-and-schedule spanindexAGT · 04Extraction agentsclause by clause againsttyped field schemasDET · 05Checklistverificationrequired fields,cross-field rules,plausibility rangesDET · 06Citation bindingdocument · page · spanon every fieldREC · 07Portfolio storeverified terms,queryableBOUNDED SPANSCITED FIELDSNOT SUREHUM · EXCHuman exception queuelow-confidence or uncited fields;corrections flow back as labelleddataSYS/CPEX · FIG.00 · SCHEMATIC · NOT TO SCALE
FIG.00 // SYSTEM MAPSIGNAL / DATADETERMINISTICAGENTHUMAN AUTHORITYSTORE / RECORDIN · 01Back-book documentsfacility agreements, security documents, sideletters; mostly scannedDET · 02Managed text extractionlayout and table awareDET · 03Clause segmentationclause-and-schedule span indexBOUNDED SPANSAGT · 04Extraction agentsclause by clause against typed field schemasDET · 05Checklist verificationrequired fields, cross-field rules,plausibility rangesDET · 06Citation bindingdocument · page · span on every fieldCITED FIELDSREC · 07Portfolio storeverified terms, queryableNOT SUREHUM · EXCHuman exception queuelow-confidence or uncited fields;corrections flow back as labelled dataSYS/CPEX · FIG.00 · SCHEMATIC
FIG.00 System map. Documents are bounded into spans before any model runs; only verified, cited fields reach the portfolio store.

The problem

FIG.01 // THE PROBLEMREJECTED PATHGATESIGNAL / DATABK-00Large back-bookFacility agreements, security documents,legal side letters; many past a hundredpagesBK-01Mostly scanned paperMargins, obligations and review clauseslived inside that paper and nowhere elseBK-02One document at a timeA simple portfolio question meant aperson opening documents by handBK-03Whole book: effectively impossiblePortfolio-wide questions went unansweredSYS/CPEX · FIG.01 · SCHEMATIC · NOT TO SCALE
FIG.01 // THE PROBLEMSIGNAL / DATAGATEREJECTED PATHBK-00Large back-bookFacility agreements, security documents, legalside letters; many past a hundred pagesBK-01Mostly scanned paperMargins, obligations and review clauses livedinside that paper and nowhere elseBK-02One document at a timeA simple portfolio question meant a personopening documents by handBK-03Whole book: effectively impossiblePortfolio-wide questions went unansweredSYS/CPEX · FIG.01 · SCHEMATIC
FIG.01 The problem. The terms that matter lived only in scanned paper, so portfolio-wide questions were effectively unanswerable. Widths are indicative.

A large regulated enterprise held a large back-book of facility agreements, security documents and legal side letters, many running past a hundred pages, most existing only as scanned paper. The terms that matter, obligations, obligations, margins, review clauses, lived inside that paper and nowhere else. Answering a simple portfolio question meant a person opening documents one at a time, and answering it across the whole book was effectively impossible.

I designed the architecture for a portfolio extraction pipeline: key terms, obligations and obligations pulled from every document, verified against a per-type checklist, with every extracted field citing the page it came from and anything uncertain routed to a human exception queue. The programme took it through build and assurance.

Why it was hard

FIG.02 // WHY IT WAS HARDREJECTED PATHGATESIGNAL / DATATRAP · APlausible but wrongextracted from the wrong clause;enters a system of recordwearing the costume of a factREQUIREMENTCheckable in secondsper-field confidence as a first-classoutput, citation to the source pageTRAP · BSmall error rate, hugebookat back-book volume, thousandsof wrong fields→ WORSE THAN MISSINGSCALE ←DOCUMENTS ENGINEERED AGAINST NAIVE EXTRACTIONHZ · 01Side letters amendthe operative clause is changedelsewhereHZ · 02Defined terms shiftmeaning moves between schedulesHZ · 03Three occurrencesthe number appears three times; onlyone bindsSYS/CPEX · FIG.02 · SCHEMATIC · NOT TO SCALE
FIG.02 // WHY IT WAS HARDSIGNAL / DATAGATEREJECTED PATHTRAP · APlausible but wrongextracted from the wrong clause; enters asystem of record wearing the costume of afact↑ WORSE THAN MISSINGREQUIREMENTCheckable in secondsper-field confidence as a first-classoutput, citation to the source page↓ SCALETRAP · BSmall error rate, huge bookat back-book volume, thousands of wrongfieldsDOCUMENTS ENGINEERED AGAINST NAIVEEXTRACTIONHZ · 01Side letters amendthe operative clause is changed elsewhereHZ · 02Defined terms shiftmeaning moves between schedulesHZ · 03Three occurrencesthe number appears three times; only one bindsSYS/CPEX · FIG.02 · SCHEMATIC
FIG.02 Why it was hard. Plausible-but-wrong is worse than missing, and scale turns a small error rate into thousands of wrong fields.

Legal documents are engineered against naive extraction. The operative clause is amended by a side letter, the defined term shifts meaning between schedules, and the number you want appears three times with only one occurrence binding. A model that reads a facility agreement the way it reads a news article will extract something plausible from the wrong clause, and plausible-but-wrong is worse than missing, because it enters a system of record wearing the costume of a fact.

Scale changed the failure economics too. At back-book volume, even a small error rate is thousands of wrong fields, so per-field confidence had to be a first-class output, not an afterthought. And the consumers were credit and legal teams who would trust nothing they could not check in seconds, which made citation to the source page a functional requirement rather than a nicety.

Key decisions

FIG.03 // KEY DECISIONSSTORE / RECORDGATEDETERMINISTICSIGNAL / DATAD · 01Anchor to structureClause-and-schedulespan indexEvery questionasks a boundedspanNot a hundredpagesD · 02Bind to evidenceDocumentidentifierPageSpanD · 03Checklist, not a vibeRequired fieldsCross-field rulesPlausibilityrangesD · 04Confidence routesworkHigh confidenceflows throughEverything elsequeues for apersonCorrectionsbecome labelleddataBOUNDED SPANSCITED FIELDSCHECKLIST STATEMANAGEDScaling theback-book isa capacitydecisionSYS/CPEX · FIG.03 · SCHEMATIC · NOT TO SCALE
FIG.03 // KEY DECISIONSSIGNAL / DATADETERMINISTICGATESTORE / RECORDD · 01Anchor to structureClause-and-schedule span indexEvery question asks a bounded spanNot a hundred pagesD · 02Bind to evidenceDocument identifierPageSpanD · 03Checklist, not a vibeRequired fieldsCross-field rulesPlausibility rangesD · 04Confidence routes workHigh confidence flows throughEverything else queues for a personCorrections become labelled dataBOUNDED SPANSCITED FIELDSCHECKLIST STATEMANAGEDScaling the back-book is a capacitydecisionSYS/CPEX · FIG.03 · SCHEMATIC
FIG.03 Key decisions. A field that cannot cite its page is never written; it goes to the exception queue.

Extraction is anchored to structure before models run. Documents are segmented into a clause-and-schedule span index, so every downstream question is asked of a bounded span, not of a hundred pages.

Every extracted field is bound to its evidence: document identifier, page, span. A field that cannot cite its page is not written to the portfolio store; it goes to the exception queue instead.

Verification is a checklist, not a vibe. Each document type carries an explicit checklist of required fields, cross-field rules and plausibility ranges, and a document is complete only when its checklist passes.

Confidence thresholds route work rather than decorate it. High-confidence fields flow through; everything else queues for a person, and the person's correction is captured as labelled data.

The pipeline runs on managed services throughout, object storage, managed text and layout extraction, workflow orchestration and managed model endpoints on the institution's AWS estate, so scaling the back-book was a capacity decision rather than an engineering project.

Architecture

FIG.04 // ARCHITECTURESTORE / RECORDHUMAN AUTHORITYAGENTDETERMINISTICSIGNAL / DATAAWS · 01Object storagedocuments landAWS · 02Text and layoutextractiontables preservedAWS · 03Span index builtcarried more weight thanany model choiceAWS · 04Extraction agentstyped field schemasAWS · 05Checklist verifierjudges each documentAWS · 06Citation binderpage-level evidencebefore any writeAWS · 07Portfolio storepassing fieldsFAIL / LOW CONFIDENCEEXCException queuefailures and low-confidencefields; resolved corrections flowback inSYS/CPEX · FIG.04 · SCHEMATIC · NOT TO SCALE
FIG.04 // ARCHITECTUREDETERMINISTICAGENTHUMAN AUTHORITYSTORE / RECORDAWS · 01Object storagedocuments landAWS · 02Text and layout extractiontables preservedAWS · 03Span index builtcarried more weight than any model choiceAWS · 04Extraction agentstyped field schemasAWS · 05Checklist verifierjudges each documentAWS · 06Citation binderpage-level evidence before any writeAWS · 07Portfolio storepassing fieldsFAIL / LOW CONFIDENCEEXCException queuefailures and low-confidence fields;resolved corrections flow back inSYS/CPEX · FIG.04 · SCHEMATIC
FIG.04 Architecture. A bounded clause is a tractable question with a checkable answer; a hundred pages is an invitation to confident error.

Documents land in object storage and pass through managed text extraction with layout and table awareness. The span index is built, extraction agents work clause by clause against typed field schemas, and the checklist verifier judges each document. The citation binder attaches page-level evidence to every field before anything is written. Passing fields land in the queryable portfolio store; failures and low-confidence fields route to the exception queue, and resolved corrections flow back in.

The span index carried more weight than any model choice. Asking a model to find the margin ratchet inside a bounded clause is a tractable question with a checkable answer; asking it to find the same thing in a hundred pages is an invitation to confident error.

Trade-offs

FIG.05 // TRADE-OFFSREJECTED PATHGATEDETERMINISTIC▼ PAYSDESIGN CHOICE▲ BUYSFront-loaded effort astraight-to-model designskipsT-01Span index and per-typechecklistsNew document types arrive aschecklists, not pipelinesReal work for real people inthe queueT-02Routing on confidenceEach exception: a correctedrecord and a training exampleStorage and plumbingT-03Page-level citationA claim a legal reviewerchecks in secondsSYS/CPEX · FIG.05 · SCHEMATIC · NOT TO SCALE
FIG.05 // TRADE-OFFSDETERMINISTICGATEREJECTED PATHT-01Span index and per-type checklists▼ PAYSFront-loadedeffort astraight-to-modeldesign skips▲ BUYSNew documenttypes arrive aschecklists, notpipelinesT-02Routing on confidence▼ PAYSReal work forreal people inthe queue▲ BUYSEach exception:a correctedrecord and atraining exampleT-03Page-level citation▼ PAYSStorage andplumbing▲ BUYSA claim a legalreviewer checksin secondsSYS/CPEX · FIG.05 · SCHEMATIC
FIG.05 Trade-offs. Each cost buys the property the credit and legal consumers actually trade in.

Building the span index and per-type checklists front-loads effort that a straight-to-model design skips, and it pays back the moment the document mix broadens, because new document types arrive as new checklists rather than new pipelines. Routing on confidence means the exception queue is real work for real people; the design treats that as the point, since every resolved exception is both a corrected record and a training example. Page-level citation costs storage and plumbing and buys the only currency that matters with legal reviewer: a claim they can check in seconds.

Outcome

FIG.06 // OUTCOMEDETERMINISTICSIGNAL / DATAPORTFOLIO-WIDE TERM QUERIESmanual archaeologyQueryablePORTFOLIO STORE AS SUBSTRATEOBLIGATIONS MONITORINGOn the storeBUILT ON EXTRACTED, CITED TERMSCERTIFIED PATTERN, REUSED ON OTHER BACK-BOOKSSpan indexChecklist verificationCitation bindingException queueSYS/CPEX · FIG.06 · SCHEMATIC · NOT TO SCALE
FIG.06 // OUTCOMESIGNAL / DATADETERMINISTICPORTFOLIO-WIDE TERM QUERIESmanual archaeologyQueryablePORTFOLIO STORE AS SUBSTRATEOBLIGATIONS MONITORINGOn the storeBUILT ON EXTRACTED, CITED TERMSCERTIFIED PATTERN, REUSED ON OTHERBACK-BOOKSSpan indexChecklist verificationCitation bindingException queueSYS/CPEX · FIG.06 · SCHEMATIC
FIG.06 Outcome. The extracted store became the substrate for obligations monitoring and portfolio-wide queries.

The pipeline was taken through build and assurance by the programme at a large regulated enterprise, and the extracted portfolio store became the substrate for obligations monitoring and portfolio-wide term queries that were previously manual archaeology. The pattern, span index, checklist verification, citation binding, exception queue, shipped as a certified pattern with its evaluation pack and control mapping, and teams I worked with have reused it on other document-heavy back-books.

What I learned

FIG.07 // WHAT I LEARNEDHUMAN AUTHORITYDETERMINISTICAX-01Bound the questionMost extraction failures wereretrieval failures wearing extractionclothes.AX-02Quality is a stateChecklists turn quality from anopinion into a state a document is in.AX-03"Not sure" earns trustReviewers forgave "not sure" far morereadily than a confident mistake.SYS/CPEX · FIG.07 · SCHEMATIC · NOT TO SCALE
FIG.07 // WHAT I LEARNEDDETERMINISTICHUMAN AUTHORITYAX-01Bound the questionMost extraction failures were retrievalfailures wearing extraction clothes.AX-02Quality is a stateChecklists turn quality from an opinion into astate a document is in.AX-03"Not sure" earns trustReviewers forgave "not sure" far more readilythan a confident mistake.SYS/CPEX · FIG.07 · SCHEMATIC
FIG.07 What I learned. Three lessons from the engagement.

Bounding the question is worth more than upgrading the model; most extraction failures I saw were retrieval failures wearing extraction clothes. Checklists turn quality from an opinion into a state a document is in. And the exception queue is where the system earns trust: reviewers forgave the pipeline for saying "not sure" far more readily than they would ever have forgiven a confident mistake.