IntellectAI · 2022–2024
First production LLM in regulated finance
I wrote the bid that won the ESG data contract for a sovereign wealth fund with over $2 trillion under management. At month ten, with the contract at risk, I proposed and proved the LLM approach that saved it.
Congratulations for delivering the first Production outcome using LLMs.
What happened
- $2.5Mn contract saved at month ten.Two weeks of LLM R&D proved the approach, the first week alone and the second with a colleague, then eight of us took it to production in five days.
- All 9,000 portfolio companies covered in depth,up from the 60 to 80 a year the fund's analysts could manage.
- Likely the first production LLM in regulated finance anywhere.A large language model running live for a regulated client. The CTO announced it to the whole company in August 2023.
The arc
Machine learning took ten months to reach its limit. The replacement took nineteen days from that limit to running live.
- Sep 2022
Bid won
I wrote the tender for a sovereign wealth fund with over $2 trillion under management. Four competitors. The public procurement record names the client.
The public record of the $2.5Mn award, with the client named
- Month ten
The BERT models reach their limit
Ten months of hand-labelled examples, and accuracy was still short of what the fund needed. The contract was at risk.
- Mid-2023
Two weeks of LLM research
A week alone, then a week with Satish Kandru: prompts, retrieval and checks tried and measured against the same accuracy targets as the machine learning approach.
- Aug 2023
Five days to production
A team of eight shipped the replacement.
- 15 Aug 2023
CTO writes to the company
Intellect's first production outcome using LLMs.
-
The fund's first commendation
Twelve of thirteen questions taken into production.
- 2024 on
Starting point for how Purple Fabric reads documents
The rescue framework shaped how the platform finds answers in documents.
The problem
Most of the portfolio was skimmed or never reviewed at all.
The client is a sovereign wealth fund with over $2 trillion under management. It owns shares in 9,000 companies and has to judge how each one behaves on the environment, on its people and on how it is run. That is what ESG stands for. The evidence sits in thousands of pages of company reports. The brief was to read those reports for every company in the portfolio and answer the fund's own set of questions about each one, with each answer pointing to the page it came from. The public procurement record on the timeline above names the fund.
Ratings agencies could not fill the gap. Their standard products did not answer the fund's own questions, on topics like water and biodiversity, and where they had data it did not cover all 9,000 companies. The numbers show how much of the portfolio went unread.
The turn
The old models had to be taught each question from hundreds of examples. GPT-4 and Claude 2 could read the report and answer it, and with the right harness and flow around them, do that for all 9,000 companies.
For ten months the pipeline ran on BERT models, an earlier kind of language model that has to be taught each question from hundreds of examples labelled by hand. That works for questions with a clear pattern. It failed on greenwashing, a claim that sounds green but is not, because a greenwashed passage uses the same words as a real one. Those were the questions the fund cared about most. After ten months, accuracy was still below what the fund required. The table shows the two approaches side by side.
Then, in 2023, GPT-4 and Claude 2 arrived. They could read a report they had never seen and answer a question about it directly. On their own they were not enough. They needed a harness: the software around the model that gives it the right passages, asks the question the right way, checks the answer, and stops it guessing. With that harness, the same question could be asked of every report from all 9,000 companies. I proposed we build it.
- To learn one question
- Hundreds of passages pulled from company reports, each labelled by hand: answers the question, looks like it does but does not, or off the point
- The question in plain English, plus the relevant passages from the report. The hand-labelled answers are now used only to check the model.
- Judgement calls
- We could tell a real answer from greenwashing. The model could not, because a greenwashed passage uses the same words as a real one, and we never cracked that.
- The model reads the passage the way an analyst would, when the harness feeds it the relevant passages and checks the answer.
- Time
- Eighteen months of building
- Two weeks to prove, five days to ship
- Accuracy
- Below the accuracy the fund required. No questions in production.
- Met the accuracy the fund required. Questions in production across the whole portfolio.
- People to run it
- 20 data scientists and 18 ESG analysts
- 18 ESG analysts and 2 data scientists. The rest of the data science team moved on to build Purple Fabric.
The rescue
One week alone, one week with a colleague, then eight of us shipped it in five days.
In a regulated delivery, with a client this careful, the idea met scepticism at first, and that was fair. So I asked for one week, not a project. The first week showed promise, the second held up under a second pair of eyes, and the CTO gave us the team to ship it.
The proof
The CTO wrote to the whole company the week we shipped. The client wrote nine months later.
Almost all of the questions meet or exceed our expectations. We are already working on putting 12 of 13 questions into production.
In production
All 9,000 companies in depth, and every answer traced back to the report it came from.
Every answer links to its source page
Click a data point and the company's report opens at the paragraph the answer came from.
Scores for the whole portfolio, down to one company
Environmental, social and governance scores per company, down to the individual metric.
- 10M
documents, 60 billion searchable passages
Every question runs against all of them, and each answer names the report it came from.
What it became
A contract rescue became the starting point for how Purple Fabric finds answers in documents.
Purple Fabric is IntellectAI's platform for building AI agents inside banks and insurers. Before an agent can answer anything, the platform has to find the right passage in the right document. The question-answering framework from the rescue was the initial seed for that layer. It didn't turn into that layer directly, but it set the direction.
In January 2024, four months before Purple Fabric launched, I ran Intellect's first Prompt-a-thon: three days in which 18 ESG analysts learned to build with LLMs from scratch.
