Thank you so much for your interest in SVDS. We have been acquired and no longer provide consulting services, but have left our blog posts here as reference material for those who may benefit from their content. We thank our customers, partners, investors, friends and family for their support over the years. And most importantly, we thank our employees for their hard work and dedication to building a great company! Silicon Valley Data Science April 2013 – December 2017   The post Thank You appeared first on Silicon Valley Data Science ..read more

We look forward with anticipation to what 2018 will bring—and we wish you peace, prosperity, and happiness this season and in the year ahead.   The post Happy Holidays from SVDS appeared first on Silicon Valley Data Science ..read more

Editor’s note: Welcome to Throwback Thursdays! Every third Thursday of the month, we feature a classic post from the earlier days of our company, gently updated as appropriate. We still find them helpful, and we think you will, too! You can find the original post here. Many project teams have found themselves in the situation where it seems that they are “so close” to completing a product rollout, but can never quite seem to get there. It’s as if there exists an invisible canyon between development and production that only heroic feats can overcome (images of tightropes, rickety bridges, and j ..read more

Editor’s Note: This is a transcript of the Q&A portion of a webinar we held earlier this year, called “What it Means to be Data-Driven”. These questions have come up repeatedly at other talks we’ve given on the topic, and so we decided to present them in blog format in the hopes that they will reach and help even more people. Have additional questions? Leave them in the comments below. Q: What is the minimum viable platform infrastructure I need to play with data? It’s not easy to transition an entire organization’s infrastructure, and in order to convince people to do that, I need to show ..read more

An aptitude for data is truly fundamental in today’s business world. Business moves fast, and technology moves even faster. In an ever-shifting landscape, how can you guide your business into managing the resulting uncertainty at the same time as tapping some of the growth potential? Being data-driven is the best way to manage uncertainty—but achieving that is about far more than bringing a bunch of numbers to your latest meeting. It’s about having your entire organization set up in a way that can experiment itself in a competitive marketplace. At the end of the day, data and its associated te ..read more

Many of us here at SVDS rely on Caltrain, Silicon Valley’s commuter rail line, to commute to and from the office every day. As part of an ongoing R&D effort, we have been collecting and analyzing various sources of Caltrain-related data with the goal of determining where each train is, and how far behind schedule it’s currently running. In addition to video cameras and microphones, we use social networking platforms as sources of signal to detect service disruptions or delays as they’re reported on by the affected passengers. Twitter is a popular place for people to vent their frustrations ..read more

Editor’s note: Welcome to Throwback Thursdays! Every third Thursday of the month, we feature a classic post from the earlier days of our company, gently updated as appropriate. We still find them helpful, and we think you will, too! You can find the original post here. If you’re fresh from a machine learning course, chances are most of the datasets you used were fairly easy. Among other things, when you built classifiers, the example classes were balanced, meaning there were approximately the same number of examples of each class. Instructors usually employ cleaned up datasets so as to co ..read more

I recently spoke with Paco Nathan, Director of the Learning Group at O’Reilly Media. In the interview below, we discuss making life more livable, AI fears, and more. I want to start by just asking what are you up to and what are you working on in the data space right now? My job kind of changed a bit back in February. We were out at our own AI conference about a year ago in New York and talking to people and recognizing that some of what we were showcasing had a lot of applicability for our own products and services. So we started up an effort to do AI applications in media. Mostly for Safari ..read more

Business leaders cannot afford to ignore their organization’s data—rather, that data should be used to make informed decisions. In this post, Principal Data Scientist Tom Fawcett and Professor of Data Science Foster Provost discuss how businesses can make the most of their analytical teams. Tom and Foster are the authors of Data Science for Business. What aspect of data science do you feel business folks most miss/misunderstand? Tom: I’ll list four: Data science techniques need data. (You’d be surprised how often this is forgotten.) Evaluation should be one of the first things you t ..read more

Small files can be a pain when working with Hadoop installations. When using HDFS, they can cause contention and significant memory utilization on the NameNode due to having to keep track of the metadata for each of the files. While dealing with small files is a common problem across all Hadoop distributions distributions, MapR-FS has unique difficulties. In this post, we will discuss how dealing with small files is different if you are using MapR-FS rather than the traditional HDFS installation. Working with MapR-FS MapR-FS is a ground up rewrite of the Java-based HDFS in C/C++. It focuses on ..read more