Brian Romanko - Software architect, product strategist, entrepreneur and total geek.
What did you find compelling about Earnest?I got involved with Earnest a year ago today. I spent a lot of time talking to Louis and Ben [Earnest's founders], prior to deciding to join. I'm always drawn to industries where the status quo is just not great.
To become a bank is a massive undertaking. The existing products and services that exist for consumer finance are really not great. A lot of financial institutions don't really care about forming a life-long relationship with customers. Louis and Ben, they proposed to me the idea of a financial institution that did care about all that stuff and really, deeply cared about doing the right thing by customers.
It seems like a very noble kind of mission overall. The existing state of banking product and services in general is really terrible. In my mind, making a bank that doesn't suck isn't too much of a challenge. What do you think it takes to make a bank that's great?We approached everything from the ground up. We built an alternative product and an alternative service that asked, "If you knew nothing about the way existing loans and profiles are calculated, how would you judge risk? What would you look at to determine financial responsibility?" Let's build models that do that instead and not necessarily assume some existing status quo way of doing it.
Traditional credit scores look at how you have historically behaved with lending products. Whether you had a credit card, made these payments on time, or have you had any bankruptcies, how many types of credit that you applied for. It's looking at one aspect of who you are as an individual. What we're doing is we're saying, okay, that's still valid and useful information, but it's not a complete picture. There are other things about a person who's looking to borrow money, like, What's your education level? What's the industry that you work in? How many gaps in your employment have you had in your career? What's the trajectory or the expected growth for the industry you're in and the unemployment rate?
We also look at spending and savings patterns. If you're constantly contributing to a 401K or a savings account, that's a really good indicator of how financially responsible you are and how much you plan for the future. A credit score doesn't look at that type of information.
Then, because we're collecting a whole lot of information from other sources, in order for us to actually make credit decisions on it, we have to a lot of DTS and ETL work. We have a distributed task manager that's also written in Node and sits on top of Amazon's SQS. It allows us to take what is very raw information and do transformations, do aggregations on it and then produce key-value pairs and other decision-making values that we can then use in the data science side. For example, for fraud detection, for underwriting, etc. And then we have other internal tooling which is also Angular.
There have been problems there from a code-making standpoint. We've developed a set of patterns and practices, number one, to ensure consistency, so we use Phabricator for code reviews and security checks that force all code through unit tests, linking, and validation, prior to reviewing. Then you have peers that are reviewing your code.
Node is really focused on building small modules that you build together, that you piece together, so we apply a lot of those same principles to our own application development. We have certain small repositories that are each responsible for one thing, and it helps maintain re-factoring to a more limited area. For instance, the ORM and persistence layer lives in a separate Node module.
Alex Lockwood - Full-Stack Software Engineer, former Wall Street-er, bicycle racer.
What's your background, and how did you end up at Earnest?My original career path was actually in finance. I ended up in New York for a few years and then found out that I wasn't getting to create stuff as much as I would like. I moved back to California where I grew up, and did this, that and the other. I started dabbling with code because I could and I had some experience with that, and one thing led to another.
Here at Earnest, I was introduced through a random family relation to Louis about a year ago. We started chatting, we got along great. That was actually before our CTO Brian was part of the group.
How has your testing experience been with Angular?So, that is one of the nice things of using Angular that its testable, and it integrates nicely with testing frameworks. Theoretically, tests come first, but there's a balance between speed of development, limited resources, and whatnot. In the past six months, we've moved towards getting very high test coverage. It's very important to us to not break things. We use Protractor tests for end-to-end testing, and Karma and Mocha for our unit testing. Most things are now unit tested and end-to-end tested, although it depends on what part of the application we're talking about.
With everything you've learned, is there anything you would do differently next time?Well, I would be aware that it's more of a process. That is, a lot of development is not just coding. It's about, and not even just infrastructure, but about whole management of the process, and working together as a team. We've learned a lot about how to mitigate some of the procedural risks in terms of who's responsible for QA, and who's responsible for determining test coverage.
We're moving forward integrating a more robust QA plan, and we brought somebody to help us do the product management. He's helping ensure that we run our tests, until we have a dedicated QA team. Sometimes we take for granted that people have gone through and tested a bunch of corner cases. Now we're documenting all of that. Part of the process is building the development and engineering parts of the organization.
Austen Head - Founding Data Scientist, Stanford PHD in Statistics, kite flyer.
What drew you to Earnest?I was at Stanford, and I'd done several consulting projects. I put up a personal website with my research interests and coursework, and gave my email out for people to contact me about stats consulting.That's actually how Louis reached out to me. I saw that his email address was an @a16z. I thought, "Oh, this is really cool. Maybe I can be an in-house stats consultant for Andreessen Horowitz or something like that," and so I agreed to meet him.
When we first met, he asked me, "How can you give loans to people and have zero percent defaults?" I said, "That's not how statistics work." [We laugh]. "We need some people who default, some people who don't. Then, in the future, don't give loans to people who are similar to those who defaulted." There were a number of statistical questions that came up pretty immediately.
I asked him, "Why do you think that this is possible? Because from what you described initially, it doesn't seem possible." He said that if he looked at all the data by hand, if he looked through all the data that he wanted for, let's say, 100 people, then he would be able to pick out five of them that would be in the top 10 or 20 of some objective least likely to default. He thought if he could do this, then surely a machine could do it.
Then, I realized, "This is actually related to a hot topic in machine learning called Active Learning." There, the idea is that you have three different costs you want to minimize. Together you have false positives (giving a loan to somebody who defaults), false negatives (failing to give a loan to someone who would not have defaulted), and the cost of actually labeling the data point - that is, have a human underwriter look at this and ask, is this person going to default? I got excited about this, and thought, "This is cool. I can turn this into a paper and get a publication out of it, and then I'll go off and do my next thing." But the more I talked to Louis about it, the more I realized this was something that I could get behind. I never imagined myself going into finance.
That problem sounds really cool, and it sounds like what ended up driving you into, well, finance.I see so many unsavory examples of financial companies that I didn't see myself going in there. I saw what Earnest is trying to do is really great. I feel like being in finance, I as an individual can have a much bigger impact to try to make things better than if I were in some industry that was already doing things right.
In finance, I can make a larger dent in things, because there is so much already broken. I feel like this is a very mission-driven company, and it's a mission that I very much agree with. Our team is absolutely phenomenal, and the work that I get to do every day is cool.
You've mentioned several times you're looking at novel features in building your models - could you elaborate a bit on those, without revealing anything that's confidential?We collect data from three sources. A client will connect LinkedIn, their financial accounts, and we will do a credit pull on them so we have a credit report. The reason for that is because then we see balances and a transaction history for the same amount of time as if you were to connect it to, say, Mint. You can get a time-stamp, an amount, and a string that describes it. We have that data. From LinkedIn, it's anything that is on LinkedIn is what we're getting. And of course, we have the standard credit report data.
Early on, one of the more surprising insights we got was that the number of LinkedIn connections ended up being very important in our model for predicting financial responsibility. That was a pretty big surprise to us, even though in retrospect, it's obvious why.
Well, it seems like there could be confirmation bias there?Right. We dug into this a lot to figure out, "OK, what's going on here?" We saw that it's not the number of LinkedIn connections, but rather, does this person have zero LinkedIn connections, or do they have more than zero LinkedIn connections? Once we had that information, we realized the people who actually connected LinkedIn and already had that profile, tended to be people who we rated as more financial responsible. We ended up making a business decision to actually requiring LinkedIn for all of our clients.
Cool! In terms of the actual set-up you have, how does your set of models and your feature extraction pipeline fit in the rest of the infrastructure?We ended up with a tough decision between R and Python. In the end we wound up using R - I'm more familiar with it, but also, Python feels like it was built for doing engineering stuff ; data science feels like an afterthought. And on the exploratory side, there is just so much that R dominates Python on, particular in plotting.
As far as where it fits in, in some sense it is relatively independent. Part of the reason for that is it's easier to iterate quickly and then once I'm confident something is a good model, we run it by our risk team and our underwriters and our legal team. Since we're finance company, we need to make sure that we aren't doing things that have disparate impact.
So you have to be very careful to avoid variables that are unwittingly disciminatory?Yes, exactly. We do need to make sure that we're being fair in the way we do this. It's not enough to just do a good job of making predictions - we need to do a good job of making predictions in an ethical and fair manner.
Posted by Vaibhav Mallya on October 19, 2014. Don't forget to subscribe to our list to stay up-to-date!
Earnest is looking for engineers and data scientists. sponsored