FICO Focused Foundation Model Delivers Trustworthy, Responsible GenAI
FICO® Focused Language Model and FICO® Focused Sequence Model set global standards for Responsible AI, delivering measurable enterprise value
If you follow my Forbes Tech columns, FICO blogs or LinkedIn posts, you’ll know I have been writing about—and working on—focused language models (FLMs) for quite some time. I am delighted by the recent release of the FICO Focused Foundation Model [FFM] for Financial Services, a domain-specific, task-focused, rigorously trained small language model (SLM) solution that produces consistent answers, a critical requirement in heavily regulated industries like banking.
Importantly, FICO FFM can be accompanied by a trust score alongside every response, giving users a risk-based approach to monitoring and stopping hallucinations.
In this blog I’ll talk about the “why” and “how” of both FLMs and trust scores and illustrate how FICO’s approach to responsible, trustworthy use of generative AI (GenAI) is a critical step toward operationalizing GenAI.
Why Do We Need Focus?
There’s no need to dwell on describing GenAI large language models’ (LLM) tendency to hallucinate—you may have experienced it yourself! Hallucinations are a key reason that many organizations have been reluctant to deploy GenAI because LLM-based solutions can’t be operationalized in a consistent manner.
SLMs have cropped up as an industry response. Smaller and less complex than LLMs, they’re designed to efficiently perform specific language tasks and are built with fewer parameters and often less training data. Like LLMs, SLMs are available from multiple providers and come with many of the same challenges as LLMs.
My approach to achieving Responsible GenAI concentrates LLM applications further into a "focused language model" (FLM), a new concept in that SLM development is focused around a very narrow domain and a specific task. A fine level of specificity ensures the appropriate data is chosen, and the model painstakingly task-trained to further ensure its correctness on a narrowly defined task. This is the “secret” to the success of FICO Focused Foundation Model—our FLM focuses on both the domain and the task to be resolved in each instance.
The FLM approach is distinctly different from commercially available LLMs and SLMs, which offer no control of the data used to build the model. This is what causes hallucinations and harm.
Ultimately, financial organizations will have collections of highly valuable task FLM models to solve problems such as compliance, underwriting, AML, collections of interactions, hardship, and many more.
Why Do We Need Trust Scores?
It’s an obvious question that may pop into LLM users’ minds every time they send a prompt into the ether: How can they trust if any given language model output is correct? Since I’m a scientist who loves nothing more than solving problems, my mind started firing on all cylinders on this one. I theorized that the associated data science solution would need to do two things:
Leverage sanctioned answers to questions the language model is designed to respond to
Score the alignment of associated GenAI outputs with these sanctioned answers
Armed with the one-two punch expressed in a derived risk-based score, users could then assess whether specific GenAI output aligns with the model’s purpose and sanctioned responses––in other words, judging whether the response is reliable and trustworthy.
Thus, I developed the trust score, an analytic construct produced by a secondary model running alongside the FLM. This sidecar model produces a trust score from 1 to 999, reflecting the likelihood that the response covers data the task-specific FLM was trained on (such as a bank’s corporate policies for handing, say, hardship requests), and the response’s alignment with the sanctioned answers known as knowledge anchors (more on those ahead). In this way, users can refer to the score to decide if they will trust the FLM’s answer as being supported.
Trust scores really are a breakthrough—finally, we have a way to quantify the trustworthiness of language model outputs, so they can be operationalized in enterprise use cases.
How Does the FICO Focused Foundation Model Catalyze GenAI ROI?
One of the first popular domains for FICO’s FLM solution is regulatory compliance, a critically important concern. We often hear about big fines related to financial institutions’ failure to comply with anti-money laundering (AML) failures, but in some parts of the world, banks are heavily regulated as to how they can interact with customers around specific issues such as financial hardship.
For example, in April 2024, the UK Financial Conduct Authority (FCA) issued policy statement PS24/2, requiring “mortgage and consumer credit firms to provide enhanced support to their customers who are both in and at risk of financial difficulty.”
Assessing customers for hardship and offering remedies involves a complex interplay of real-time conversation, historical conversation, and transaction data. Here’s how to put a focused language model with trust scores on tasks.
Define the focus: The data science team works with financial institution’s business owners and domain experts to define the new capability. This includes defining the specific problem to be solved (here, financial hardship), requisite criteria, the internal data sources to be leveraged, and the length of historical data considered. Importantly, this phase also yields a heavily scrutinized set of seed examples of correct and incorrect ways that customers are treated by contact center agents based on the broad data made available to the FLM.
Build the knowledge anchors: Those who will monitor the FLM outputs work with the data science team to define knowledge anchors associated with the tasks performed by the FLM: the questions the FLM is designed to answer and the aligned responses with sanctioned answers. There may be 100 knowledge anchors, a thousand, or even more, depending on the scope of the FLM. Equipped with knowledge anchors, the data scientists concurrently develop the trust model, independent of the FLM, that works with the specialized domain vocabulary. This model tokenizes the relevant vocabulary of the knowledge anchors and creates latent space representations of them, allowing comparisons between subtle responses and the degree of their differences (from the sanctioned responses) to be quantified.
Train the FLM with synthetic data: The data science team then utilizes the experts’ input and provided seed examples to create huge volumes of synthetic language data—for instance, hundreds of customer treatment examples can translate into millions of synthetic language data examples, which are used to train an FLM task model. Synthetic training data is imperative because it allows a large data corpus to be created that does not expose the model to personal identifying information (PII) and is focused on heavily curtailed seed data produced by the financial institution’s top experts.
Give the agent real-time guidance: As the agent communicates with the customer, the FLM, now trained and in production, provides an appropriate script in real-time. In the background, as the agent asks questions of the FLM, the trust model removes non-domain vocabulary, computes a latent space representation, and measures how close this vector is to the knowledge anchors. If it’s close within the knowledge anchor latent vector space, the question is aligned with the “what” and the “how” the model is designed to answer, producing a high trust score. Low alignment yields a low trust score. An analogous process occurs with the response. The agent, assured of acting compliantly, can focus on conducting an empathetic conversation with the customer, which may include hardship accommodation options such as a three-month payment holiday, a lower credit card interest rate or balance transfer offer, a second charge mortgage (the UK term for home equity line of credit [HELOC]), or any number of other individualized actions.
Laser-focused FLMs Drive Better Decisions
For financial institutions, this FLM application yields a focused solution to an ambiguous challenge: monitoring customers needing personalized hardship modifications and delivering them in an empathetic, fair, consistent, and compliant manner. The model scales a few very specialized human experts’ knowledge to permeate all operations.
The trust model is an equally powerful breakthrough because it enables language model answers to be audited and provides a risk-based score to manage GenAI hallucinations. Banks can raise and lower risk tolerances to hallucinations and suboptimal answers by choosing different thresholds of the trust score.
Looking at the bigger picture, focused language models drive critical business value for the bank in two ways:
First, FLM models are built with a focus that indicates the domain data used and are task-trained using seed data examples that are heavily scrutinized. This focus, on only what is required to build the task model, translates into highly accurate models that outperform the largest language models in the world, while enabling their audit and regulation.
Second, the trust score allows FLM models to be trusted, the key to operationalizing exciting GenAI technology. Accurate, trustworthy, auditable, and transparent––these are necessary requirements to achieving Responsible Generative AI, and FICO FFM meets them all.
How FICO Can Help You Advance in Responsible AI
Read about FICO’s new State of Responsible AI for Financial Services report
Download FICO's AI Playbook: A Step-by-Step Guide for Achieving Responsible AI
Popular Posts
Business and IT Alignment is Critical to Your AI Success
These are the five pillars that can unite business and IT goals and convert artificial intelligence into measurable value — fast
Read more
Average U.S. FICO Score at 717 as More Consumers Face Financial Headwinds
Outlier or Start of a New Credit Score Trend?
Read more
FICO® Score 10T Decisively Beats VantageScore 4.0 on Predictability
An analysis by FICO data scientists has found that FICO Score 10T significantly outperforms VantageScore 4.0 in mortgage origination predictive power.
Read moreTake the next step
Connect with FICO for answers to all your product and solution questions. Interested in becoming a business partner? Contact us to learn more. We look forward to hearing from you.