We Need Defensive AI to Protect Us From AI Attacks
My recent work at FICO to make AI better has drawn upon my background in theoretical physics to create what we call defensive AI. This is needed, because AI-based attacks are not s…
Note: This is an edited version of an article I wrote for RT Insights.
I’m convinced we are entering the Golden Age of artificial intelligence (AI); with so much promise and potential in front of us, I am feeling a little like Neo in The Matrix as he swallows the red pill. (Alert: more Matrix analogies ahead.)
However, rather than science fiction, my recent work at FICO to make AI better has drawn upon my background in theoretical physics to create what we call defensive AI. This is needed, because AI-based attacks are not science fiction — they are happening today.
Why We Need Defensive AI
Businesses have relied on AI to fight fraud and financial crime for more than 25 years. Before these neural network defenders came along, banks used rule-based systems to try to prevent fraud.Today, AI is considered “superhuman” at executing certain tasks, fraud detection being one of them. Criminals have had a harder time testing and understanding the complex inter-relationships between all the data elements that the anti-fraud neural networks combine, compared to the fraud rules they used to test and then evade.
But fraudsters, like New York City, never sleep. As business and society, overall, become more and more dependent on AI, we must assume that criminals will want to reproduce the AI models. Once learned, criminals find the secret paths to commit fraud without detection, based on the unique behaviors of the AI system.
To find those paths, fraudsters are again testing and attacking, targeting the bank’s AI defenders with their own offensive AI systems. But now, instead of trying to figure out the thresholds that would trigger an anti-fraud rule set to stop the transaction—such as a new grocery store, for an amount greater than $250 dollars, between the hours of 3-6 p.m.—the attackers are trying to learn the AI model.
If the attackers can learn the AI defender and how it responds, they have the ability to anticipate its move, like a fight scene from The Matrix. The attacker will learn the model and determine what they can get away with. The fraudster could then run millions of transaction perturbations in a cloud testing environment, find those that look the most likely to succeed, and launch attacks based on expert, learned knowledge of the likely AI system.
We’ve long understood this risk. As such, FICO models incorporate adaptive components that adjust scores, making it harder for offensive AI systems to learn the neural network response.
How Fraudsters Create an AI Attack Model
Criminals can learn the AI we rely on by directing fake test data at banks’ AI systems. They could get access by pretending to be a new merchant, and/or by compromising a merchant system to gain access to a payments channel, as well as banks’ testing and governance partitions.Criminals may even try to steal the AI models outright through cyber breaches. If stolen, these obfuscated and encrypted models may still respond to testing transactions.
Once they have access to the defender AI, fraudsters would likely send batches of testing transactions, millions at a time. The criminals would get a fraud score for each transaction and attempt to map out likely transaction sequences, monitoring the behaviors of the model.
In other words, using AI of their own, criminals can create an AI model to produce the same score response to their testing. This is the offensive AI model, constrained by the quality and effectiveness of the criminals’ testing and the volume of testing transactions.
With their offensive AI tech nailed down, the criminals could steal expertly, working to circumvent the bank’s AI model response with unique transactions that the bank’s anti-fraud system might have not been seen before. Banks that depend on anti-fraud AI systems need to keep them protected, and monitor data streams that may be pointed at the model to ensure that they are legitimate.
But what can the AI do itself to prevent being learned? This is defensive AI, and FICO has recently applied for a patent for this technique.
Defensive AI Outsmarts Criminals
Defensive AI models selectively deceive or return incorrect outputs, if the models believes they are being monitored. They might return scores that are backwards, or create patterns that make the adversary modeling data set inaccurate and consequently the attacker’s AI less effective. Clever score responses could even guide the defensive AI to create artificial patterns in a learned offensive AI, making the criminal’s use of the offensive AI model easier for the bank to detect.The defensive AI system could also bias the responses for the attacker AI to be misled, to generate data of a particular form. For example, the AI may decide it will give much reduced (non-fraud) scores to the attacker’s test data for electronic purchases between $300-$500. Later, in production, the defensive AI system can determine if the attacker took the bait, and then rapidly isolate the new transactions of this behavior and the sources, to turn over to law enforcement.
Just Like Neo
Defensive AI thwarts criminals’ attempts to measure it. As such, criminals and their AI will find it much harder to determine which responses from defensive reactions are legitimate. The criminals will wonder, “Is it real, or is it The Matrix?”I’m going to catch up with my old friend Neo now, on Netflix. Send me your favorite Matrix moments on Twitter, where I’m @ScottZoldi.
Popular Posts
Average U.S. FICO® Score at 716, Indicating Improvement in Consumer Credit Behaviors Despite Pandemic
The FICO Score is a broad-based, independent standard measure of credit risk
Read more
The GSE Data Is Public. Here's What It Shows About Credit Score Modernization.
Independent Milliman analysis of GSE historical data confirms what anyone can now verify directly: FICO® Score 10T is the most predictive score.
Read more
FICO® Score High Achievers: Is Age the Only Factor?
<p>FICO’s latest research on <a href="http://www.fico.com/en/blogs/risk-compliance/us-credit-quality-rising-the-beat-goes-on/" target="_blank">national FICO® Score distribution</a> shows that US credit quality continues to trend upwards, with an increasing number of consumers scoring in the highest scores ranges. Given that, we decided it was time to refresh our study on “<a href="http://www.fico.com/en/blogs/risk-compliance/infographic-what-does-a-high-credit-scorer-look-like/" target="_blank">FICO® Score high achievers</a>,” where we examine the credit behavioural profiles of consumers with higher credit scores.</p> <p>As expected, our latest study showed that older people generally have higher scores, as has been the case in our <a href="http://www.fico.com/en/blogs/risk-compliance/age-before-beauty-credit-worthiness-before-youth/" target="_blank">past research</a>. This is largely because they have trade lines that have been open longer, which leads to higher scores (assuming the trade lines are in good standing and all other things equal). The most credit savvy among you will remember that length of credit history accounts for roughly 15% of the overall <a href="http://www.myfico.com/CreditEducation/WhatsInYourScore.aspx" target="_blank">FICO® Score calculation</a>.</p> <p>But for younger consumers, it isn’t too helpful to say, “just wait until you are 50 years old, and that will fix things!” Thus, I decided to dive deeper with our analysis, including looking at the credit behavioural characteristics of younger consumers with high FICO® Scores.</p> <p>In this analysis, I looked at credit characteristics for the two highest score bands, 750-799 and 800-850, which have about 16% and 23% of the population, respectively. I further broke this down by looking at the characteristics of the youngest 15% in each group, which occurs at age 29 for the 750-799 group and 43 for the 800-850. We used FICO<strong>®</strong> Score 9 for our analysis, and examined typical variables of the categories that go into the make-up of the score.</p> <p><a href="http://www.fico.com/en/blogs/wp-content/uploads/2016/10/FICO-Score-High-Achiever-table.png"><img class="alignnone size-full wp-image-29367" src="http://www.fico.com/en/blogs/wp-content/uploads/2016/10/FICO-Score-High-Achiever-table.png" alt="fico-score-high-achiever-table" width="715" height="747" /></a></p> <p>Since there’s a lot to dissect in this chart, I’ll highlight a few findings that struck me: <ul> <li><strong>Credit history.</strong> Higher scores are undoubtedly associated with a longer time on file. What’s striking is the higher average time on file for the 800+ group, both overall and for revolving accounts (the most common trade line). Clearly, it helps to start credit use early in life – but that’s not the sole determining factor for high scoring. The younger cohorts in both the 800+ and 750-799 groups have time on file medians significantly less than their older counterparts.</li> <li><strong>New credit.</strong> High credit scorers do, in fact, pursue new credit. We see little variation in months since last trade line opening and last inquiry among all groups observed.</li> <li><strong>Pay on time.</strong> As you might expect, the further in the past the delinquency, the higher the score – and that’s <u>if</u> a delinquency ever happened. 95% of the 800+ people and 72% of the 750-799 group have never been delinquent, which is a striking contrast to the 57% for the total population. The percentages for the younger groups are even higher, especially so for the 750-799ers. Collection items are also scarce for these high scorers. While 23% of the national population have collection items greater than $99, the figures are only 0.5% for the 800+ group and 3% for the 750-799 group. It is clear that, in conjunction with starting a credit file early, paying on time is a key trait of the FICO® Score high achiever.</li> <li><strong>Balance and utilization.</strong> Finally, we look at how credit is being used, focusing on revolving product use. (Since most people only have one mortgage or one auto loan, variation in instalment product figures is minimal and driven by payment history, which is near perfect for high scorers, as we’ve seen.) FICO® Score high achievers tend to have more open revolving accounts. While the % active trades and utilization are generally lower, by no means do high scorers avoid the use of credit products. What drives the difference here is the amount of <u>available</u> revolving credit utilized. The 800+ group is roughly 1/4 of the national median, both in total and on their highest utilized card; they also have lower balances and more cards<em>. </em>The 750-799 group is more active and higher utilized, but still only at 2/3 of the national figures. The young 750-799 group compensate with lower revolving balances and, as a result, lower utilization, which is in line with what we see for the 800+ group.</li> <li><strong>Benchmarks.</strong> I’d like to point out that while this study provides good benchmarks of the typical credit traits of the FICO® Score high achiever, individual credit profiles will vary widely. And keep in mind that when we sort groups of consumers as a whole by different behaviours, some results may look superficially odd. For example, the 800+ band has a longer median average open account time (128 months) than average revolving months on file (140 months).</li> </ul> So, what do we conclude? A huge takeaway is that high scorers don’t have too much in the way of serious delinquencies and negative public record data.</p> <p>In most respects, the young 800+ people are like their peer group. They really only differ markedly in their shorter credit history and higher non-mortgage balance, both of which are a function of life stage. Our results show that you don’t have to be a credit product hermit; these high scorers have more credit cards. The takeaway is to start credit activity early and maintain well-managed and moderate usage.</p> <p>In many ways, the young 750-799 people are more interesting because their age profile is much younger. Their credit characteristics differ more from their peer group, particularly in terms of length of credit history. They compensate by having almost pristine credit records and lower balances.</p> <p>Ultimately, 39% of US consumers are FICO® Score high achievers, which they accomplish by following common-sense credit practices. It ain’t rocket science!</p>
Read moreTake the next step
Connect with FICO for answers to all your product and solution questions. Interested in becoming a business partner? Contact us to learn more. We look forward to hearing from you.