Liberty Mutual Property Inspection, Winner's Interview: Qingchen Wang
The hugely popular Liberty Mutual Group: Property Inspection Prediction competition wrapped up on August 28, 2015 with Qingchen Wang at the top of a crowded leaderboard. A total of 2,362 players on 2,236 teams competed to predict how many hazards a property inspector would count during a home inspection.
This blog outlines Qinchen's approach, and how a relative newbie to Kaggle competitions learned from the community and ultimately took first place.
The Basics
What was your background prior to entering this challenge?
I did my bachelor’s in computer science. After working for a few months at EA Sports as a software engineer I felt the strong need to learn statistics and machine learning as the problems that interested me the most were about predicting things algorithmically. Since then I’ve earned master’s degrees in machine learning and business and I’ve just started a PhD in marketing analytics.
Qingchen's profile on Kaggle
How did you get started competing on Kaggle?
I had an applied machine learning course during my master’s at UCL and the course project was to compete on the Heritage Health Prize. Although at the time I didn’t really know what I was doing it was still a very enjoyable experience. I’ve competed briefly in other competitions since, but this was the first time I’ve been able to take part in a competition from start to finish and it turned out to have been quite a rewarding experience.
What made you decide to enter this competition?
I was in a period of unemployment so I decided to work on data science competitions full-time until I found something else to do. I actually wanted to do the Caterpillar competition at first but decided to give this one a quick go since the data didn’t require any preprocessing to start. My early submissions were not very good so I became determined to improve and ended up spending the whole time doing this.
What made this competition so rewarding was how much I learned.As more or less a Kaggle newbie, I spent the whole two months trying and learning new things. I hadn’t known about methods like gradient boosting trees or tricks like stacking/ blending and the variety of ways to handle categorical variables. At the same time, it was probably the intuition that I developed through previous education that set my model apart from some of the other competitors so I was able to validate my existing knowledge as well.
Do you have any prior experience or domain knowledge that helped you succeed in this competition?
I have zero prior experience or domain knowledge for this competition. It’s interesting because during the middle of the competition I hit a wall and a number of the top-10 ranked competitors have worked in the insurance industry so I thought maybe they had some domain knowledge which gave them an advantage. It turned out to not be the case. As far as data science competitions go, I think this one was rather straightforward.
Histogram of all fields in the dataset with labels. Script by competition participant, Rajiv Shah
Let's Get Technical
What preprocessing and supervised learning methods did you use?
I used only XGBoost (tried others but none of them performed well enough to end up in my ensemble). The key to my result was that I also did binary transformation of hazards which turned the regression problem into a set of classification problems. I noticed that some other people also tried this method through the forum thread but it seems that they didn’t go far enough with the binary transformation as that was the best performing part of my ensemble.
I also played with different encodings of categorical variables and interactions, nothing sophisticated, just the standard tricks that many others have used.
Were you surprised by any of your findings?
I’m surprised by how poor our prediction accuracies were. This seemed like a problem that was well suited for data science algorithms and it was both disappointing and exciting to see such high prediction errors. I guess that’s the difference between real life and the toy examples in courses.
Which tools did you use?
I only used XGBoost. It’s really been a learning experience for me as I entered this competition having no idea what gradient boosted trees was. After throwing random forests at the problem and getting nowhere near the top of the leaderboard, I installed XGBoost and worked really hard on tuning its parameters.
XGBoost fans or those new to boosting, check out this great blog by Jeremy Kun on the math behind boosting and why it doesn't overfit
How did you spend your time on this competition?
Since the variables were anonymous there wasn’t much feature engineering to be done. Instead I treated feature engineering as just another parameter to tune and spent all of my time tuning parameters. My final solution was an ensemble of different specifications so there were a lot of parameters to tune.
What was the run time for both training and prediction of your winning solution?
The combination of training and prediction of my winning solution takes about 2 hours on my personal laptop (2.2ghz Intel i7 processor).
Words of Wisdom
What have you taken away from this competition?
One thing that I learned which I’ve always overlooked before is thatparameter tuning really goes a long way in performance improvements. While in absolute terms it may not be much, in terms of leaderboard improvement it can be of great value. Of course, without the community and the public scripts I wouldn’t have won and may still not know about gradient boosted trees, so a big thanks to all of the people who shared their ideas and code. I learned so much from both sources so it’s been a worthwhile experience.
Click through to an animated view of the community's leaderboard progression over time, and the influence of benchmark code sharing. Script by competition participant, inversion
Do you have any advice for those just getting started in data science?
For those who don’t already have an established field, I strongly endorse education. All of my data science experience and expertise came from courses taken during my bachelor’s and master’s degrees. I believe that without already having been so well educated in machine learning I wouldn’t have been able to adapt so quickly to the new methods used in practice and the tricks that people have talked about.
There are now a number of very good education programs in data science which I suggest that everyone who wants to start in data science to look into. For those who already have their own established fields and are doing data science on the side, I think their own approaches could be very useful when combined with the standard machine learning methods. It’s always important to think outside the box and it’s all the more rewarding when you bring in your own ideas and get them to work.
Finally, don’t be afraid to hit walls and grind through long periods of trying out ideas that don’t work. A failed idea gets you one closer to a successful idea, and having many failed ideas often can result in a string of ideas that work down the road. Throughout this competition I tried every idea I thought of and only a few worked. It was a combination of patience, curiosity, and optimism that got me through these two months. The same applies to learning the technical aspects of machine learning and data science. I still remember the pain that my classmates and I endured in the machine learning courses.
Just for Fun
If you could run a Kaggle competition, what problem would you want to pose to other Kagglers?
I’m a sports junkie so I’d love to see some competitions on sports analytics. It’s a shame that I missed the one on March Madnesspredictions earlier this year. Maybe one day I’ll really run a competition on this stuff.
Editor's note: March Machine Learning Mania is an annual competition so you can catch it again in 2016!
What is your dream job?
My dream job is to lead a data science team, preferably in an industry that’s full of new and interesting prediction problems. I’d be just as happy as a data scientist though, but it’s always nice to have greater responsibilities.
Bio
Qingchen Wang is a PhD student in marketing analytics at theAmsterdam Business School, VU Amsterdam, and ORTEC. His interests are in applications of machine learning methods to complex real world problems in all domains. He has a bachelor’s degree in computer science and biology from the University of British Columbia, a master’s degree in machine learning from University College London, and a master’s degree in business administration fromINSEAD. In his free time Qingchen competes in data science competitions and reads about sports.