A Moment of Science: Polyglot Everything
By Josh Hemann Last month, I wrote about the commoditization of analytics. Actually, what I wrote was that although there were some trends in this area, I do not think it will go…
By Josh Hemann
Last month, I wrote about the commoditization of analytics. Actually, what I wrote was that although there were some trends in this area, I do not think it will go far because of the increasingly eclectic nature of tools and skills needed to work with Big Data. In this post, I want to go a little deeper into what I was referring to by “eclectic”…
Polyglotism
If you can speak several languages fluently, you are a polyglot. I am not a polyglot, but much of the world’s population is. For example, people in Sudan might speak Arabic in a public setting, English when at a government office, and a tribal language when with family.
Polyglotism exists in the domain of software development as well. I would argue that like human language, much of the world’s analytic programmers will need to be software language polyglots, if they are not already.
This notion of polyglot programming is not new. Neal Ford seems to have coined the phrase in 2006, and the idea has been popularized in recent years with respect to polyglot persistence. The idea is that with increasing instrumentation and digitization of life, multiple types of databases are needed to store (persist) information and use it effectively.
Of course, multiple databases have been used in business settings for years. In my experience working with retailers, it is almost certain that there is a mix of Oracle, Microsoft SQL Server, and IBM DB2 in use over different parts of operations. There is some polyglotism here in that SQL Server has query functionality that is distinct from Oracle.
But this type of polyglotism-by-accident is very different from purposefully and carefully choosing different databases for different parts of operational needs. These days, the area where a lot of purpose is needed is with respect to how NoSQL databases might be included in the mix. Key-value stores, document databases, graph databases, and scientific databases, for example, all make compromises with traditional database requirements (i.e., ACID) in exchange for flexibility and scalability.
Running NoSQL databases alongside traditional relational databases management systems (RDBMS) is increasingly common, especially in web-focused companies where there are large volumes of structured and unstructured data. And here is a key reality to understand: it is not as if these companies can afford to have a team of SQL developers at one end of the hall and a team of NoSQL developers at the other end. The same developers must often times be fluent in RDBMSs and SQL, as well as in data models, query engines, and query semantics found in NoSQL flavors that are completely different.
This is a concrete example of what polyglot persistence looks like: writing queries on relational data in a functional language (i.e., SQL) using well-established tools, as well as writing queries on hierarchical and unstructured data using imperative languages (e.g., Javascript) using more specialized tools (e.g., Hadoop). All in the same day.
Why polyglotism matters
Travis Oliphant (a demi-God in the Python scientific computing community) once told me something that has really stuck: shared abstractions are powerful. To add some color here, we rely on abstractions every day in our analytical work. For example, we use higher-level programming languages that allow us to focus on what we want to do, rather than on low-level machine instructions for how to change memory state. The SQL language is an abstraction for how to execute relational algebra on stored data. After being in use for more than three decades, no one can argue that this abstraction is widely shared and powerful. (For an interesting read on the database language that lost out, see here.)
A couple of weeks ago, I was honored to speak at the PyData NYC 2012 conference, and one topic I touched on in my talk was this idea of polyglot everything:
Analytic practitioners may need to develop software in one or more of these verticals that make up the end-to-end process of turning data into information affecting decision making. I spend most of my time in Persistence, Analytics and Visualization, but also dabble in the other two.
No matter what the setting, I find myself having to mentally switch between abstractions throughout the day. Sometimes this is personally rewarding, since I feel like I am learning and seeing new ways of tackling old problems.
Other times, though, it is exhausting. Each time I switch between an abstraction (e.g., MongoDB and Vertica; matplotlib and ggplot2), there is a bit of a cost (my wife would say a large cost, since I am incapable of multitasking).
Such is life in the current age of Big Data, and I don’t see this getting better any time soon. In coming posts, I’ll dive into what polyglotism looks like in some of these other verticals, how it affects business analytics, and where the new shared abstractions seem to be converging.
Popular Posts
Average U.S. FICO® Score at 716, Indicating Improvement in Consumer Credit Behaviors Despite Pandemic
The FICO Score is a broad-based, independent standard measure of credit risk
Read more
The GSE Data Is Public. Here's What It Shows About Credit Score Modernization.
Independent Milliman analysis of GSE historical data confirms what anyone can now verify directly: FICO® Score 10T is the most predictive score.
Read more
FICO® Score High Achievers: Is Age the Only Factor?
<p>FICO’s latest research on <a href="http://www.fico.com/en/blogs/risk-compliance/us-credit-quality-rising-the-beat-goes-on/" target="_blank">national FICO® Score distribution</a> shows that US credit quality continues to trend upwards, with an increasing number of consumers scoring in the highest scores ranges. Given that, we decided it was time to refresh our study on “<a href="http://www.fico.com/en/blogs/risk-compliance/infographic-what-does-a-high-credit-scorer-look-like/" target="_blank">FICO® Score high achievers</a>,” where we examine the credit behavioural profiles of consumers with higher credit scores.</p> <p>As expected, our latest study showed that older people generally have higher scores, as has been the case in our <a href="http://www.fico.com/en/blogs/risk-compliance/age-before-beauty-credit-worthiness-before-youth/" target="_blank">past research</a>. This is largely because they have trade lines that have been open longer, which leads to higher scores (assuming the trade lines are in good standing and all other things equal). The most credit savvy among you will remember that length of credit history accounts for roughly 15% of the overall <a href="http://www.myfico.com/CreditEducation/WhatsInYourScore.aspx" target="_blank">FICO® Score calculation</a>.</p> <p>But for younger consumers, it isn’t too helpful to say, “just wait until you are 50 years old, and that will fix things!” Thus, I decided to dive deeper with our analysis, including looking at the credit behavioural characteristics of younger consumers with high FICO® Scores.</p> <p>In this analysis, I looked at credit characteristics for the two highest score bands, 750-799 and 800-850, which have about 16% and 23% of the population, respectively. I further broke this down by looking at the characteristics of the youngest 15% in each group, which occurs at age 29 for the 750-799 group and 43 for the 800-850. We used FICO<strong>®</strong> Score 9 for our analysis, and examined typical variables of the categories that go into the make-up of the score.</p> <p><a href="http://www.fico.com/en/blogs/wp-content/uploads/2016/10/FICO-Score-High-Achiever-table.png"><img class="alignnone size-full wp-image-29367" src="http://www.fico.com/en/blogs/wp-content/uploads/2016/10/FICO-Score-High-Achiever-table.png" alt="fico-score-high-achiever-table" width="715" height="747" /></a></p> <p>Since there’s a lot to dissect in this chart, I’ll highlight a few findings that struck me: <ul> <li><strong>Credit history.</strong> Higher scores are undoubtedly associated with a longer time on file. What’s striking is the higher average time on file for the 800+ group, both overall and for revolving accounts (the most common trade line). Clearly, it helps to start credit use early in life – but that’s not the sole determining factor for high scoring. The younger cohorts in both the 800+ and 750-799 groups have time on file medians significantly less than their older counterparts.</li> <li><strong>New credit.</strong> High credit scorers do, in fact, pursue new credit. We see little variation in months since last trade line opening and last inquiry among all groups observed.</li> <li><strong>Pay on time.</strong> As you might expect, the further in the past the delinquency, the higher the score – and that’s <u>if</u> a delinquency ever happened. 95% of the 800+ people and 72% of the 750-799 group have never been delinquent, which is a striking contrast to the 57% for the total population. The percentages for the younger groups are even higher, especially so for the 750-799ers. Collection items are also scarce for these high scorers. While 23% of the national population have collection items greater than $99, the figures are only 0.5% for the 800+ group and 3% for the 750-799 group. It is clear that, in conjunction with starting a credit file early, paying on time is a key trait of the FICO® Score high achiever.</li> <li><strong>Balance and utilization.</strong> Finally, we look at how credit is being used, focusing on revolving product use. (Since most people only have one mortgage or one auto loan, variation in instalment product figures is minimal and driven by payment history, which is near perfect for high scorers, as we’ve seen.) FICO® Score high achievers tend to have more open revolving accounts. While the % active trades and utilization are generally lower, by no means do high scorers avoid the use of credit products. What drives the difference here is the amount of <u>available</u> revolving credit utilized. The 800+ group is roughly 1/4 of the national median, both in total and on their highest utilized card; they also have lower balances and more cards<em>. </em>The 750-799 group is more active and higher utilized, but still only at 2/3 of the national figures. The young 750-799 group compensate with lower revolving balances and, as a result, lower utilization, which is in line with what we see for the 800+ group.</li> <li><strong>Benchmarks.</strong> I’d like to point out that while this study provides good benchmarks of the typical credit traits of the FICO® Score high achiever, individual credit profiles will vary widely. And keep in mind that when we sort groups of consumers as a whole by different behaviours, some results may look superficially odd. For example, the 800+ band has a longer median average open account time (128 months) than average revolving months on file (140 months).</li> </ul> So, what do we conclude? A huge takeaway is that high scorers don’t have too much in the way of serious delinquencies and negative public record data.</p> <p>In most respects, the young 800+ people are like their peer group. They really only differ markedly in their shorter credit history and higher non-mortgage balance, both of which are a function of life stage. Our results show that you don’t have to be a credit product hermit; these high scorers have more credit cards. The takeaway is to start credit activity early and maintain well-managed and moderate usage.</p> <p>In many ways, the young 750-799 people are more interesting because their age profile is much younger. Their credit characteristics differ more from their peer group, particularly in terms of length of credit history. They compensate by having almost pristine credit records and lower balances.</p> <p>Ultimately, 39% of US consumers are FICO® Score high achievers, which they accomplish by following common-sense credit practices. It ain’t rocket science!</p>
Read moreTake the next step
Connect with FICO for answers to all your product and solution questions. Interested in becoming a business partner? Contact us to learn more. We look forward to hearing from you.