Apr 30, 2015

From Data Analyst to Predictive Modelers to Data Scientists

In an earlier posting we argued that much of what is holding companies back from adopting predictive analytics is a lack of understanding about how to recognize opportunities and create value with these tools.  It’s equally as true that once the opportunity is realized there is also a knowledge gap in the nuts and bolts of the process and where and how to acquire the talent to produce and maintain these models.  Developing predictive analytic models is a different process from responding to requests for charts and graphs based on existing information.  It is much more intuitive and creative and requires the mastery of different tools.
The mastery of how to use predictive analytic tools is not common and not easy.  You are unlikely to find competence in predictive analytics anywhere in your management team unless you have specifically hired for it.  Once business leaders recognize the opportunity, they follow one of two paths, or a combination of the two.
Grow Your Own:  Some organizations have set out to hire fully trained and competent predictive analysts.  .  There is a large and growing shortage of people with these skills.  A recent survey of individuals actually performing these tasks estimated that it took between two and eight years to achieve competence.  The shortage of educated and experienced practitioners is just now beginning to be addressed by higher education.  This can be a productive approach but requires investment, sponsorship, and an environment in which these folks can learn to apply their skills to each company’s unique problems.
On the side of the technology industry developing predictive analytic suites there has been a concerted effort to make them simpler to use.  To some extent this has been successful but not to the level these application developers would like you to believe.  These are not tools that your middle managers are going to be able to learn quickly.  Proficiency can only be gained by repeated use, preferably with the coaching and mentoring of highly skilled analysts.  There are still too many ways that these tools can produce the wrong answer.  Be very careful of going down this path.
Outside Consultants:  Much as we would like to avoid expensive outside consultants, this is one area where paying for competence and experience is a good idea.  There are some elements of this practice that really are rocket science.  Using a consulting data scientist as the core of your start-up analytics group where that person serves as instructor and mentor to your in-house dedicated staff is an approach that has proven successful at many companies and grows internal capacity while limiting cost.
Finally, the Complication of Big Data:  In a later post we will address this in more detail, but predictive analytics is now even more complex than it was before, made so by the three horsemen of Big Data:  volume, variety, and velocity.  You can see this in massive data bases in new non-traditional storage formats (volume: Hadoop, NoSQL, and the like), in the new capability to merge and match disparate types of data, some structured, some unstructured, some semi-structured (variety), and in the fast flowing data from sensors and web logs and the desire to take advantage of this in real time (velocity).
From Data Analyst to Predictive Modeler to Data Scientist: Two decades ago the folks who prepared our reports, graphs, and visualizations were ‘data analysts’ who knew how to extract data from relational data warehouses and run it through reporting and visualization tools like Crystal Reports.  Ten years ago, predictive models were built by ‘predictive modelers’ who understood both the extraction and preparation of the data as well as the specialized predictive analytic tools like SAS and SPSS that allowed them to prepare predictive models.  In the last few years, Gartner now declares that we need ‘data scientist’ who have all the above skills but also understand the complexities of the new NoSQL databases like Hadoop and can marry data from many sources and types together to produce useful and profitable predictive models.  The requirement for broader and deeper skills is real and must factor into any business decision to build in-house capacity, as well as vetting potential consultants.
Business sponsors recognize the great value and quick ROI that is possible from predictive analytic programs.  More importantly, they recognize that their competitors also see this and are probably hard at work capitalizing on these competitive advantages right now.  Organizing for the successful implementation of Big Data and predictive analytics can follow several models.  There are challenges to be overcome but thinking through where you are in the development process is worth the effort.

No comments: