Get Ready for the Next Wave of Disruption in Data and Predictive Analytics
In the last article (Predictive Analytics Ain’t No Crystal Ball), we introduced some fundamental concepts on predictive analytics and how powerful it can be to drive business values. With the increasing computing power and the rapid development of artificial intelligence (AI), the combination of AI and predictive analytics on big data is the next big wave, enabling businesses to realize business values deeper, faster, and at larger scale.
In this article we will talk about what disruptive innovations in the data and analytics world consists of, the next wave of disruption where AI is combined with predictive analytics, and what it means for your business. We will provide examples of the benefits in taking advantage of this upcoming innovative disruption and briefly touch upon how to better prepare your business for it.
Just like the rise of Uber disrupting the traditional taxi business by completely changing people’s commute behavior – disruptive innovations are also happening in the data and predictive analytics realm today. “Disruptive Innovation” is defined as technologies that emerge to challenge established incumbent businesses, i.e., the traditional way of people doing things. It is not a marginal improvement to make things a little better, but a fundamental change that make the old things obsolete.
The first wave of disruption in data and analytics arrived about 10 years ago, when coding-based data analytics platform is transformed into visual-based platforms. And with that comes the era of modern Business Intelligence, where data are visualized in an interactive and code-free environment and have since then been realizing true business values.
Now we are sitting on the horizon of another market disruption – “Augmented Analytics” where predictive analytics is combined with the power of artificial intelligence (AI). When the power of predictive analytics is enhanced by booming development in machine learning and AI, it will be capable to drive real business value at scale and speed that precedents our wildest imagination. 10 years ago, we struggled to find a handful of machine learning/AI enabled business applications. In 10 years, we will struggle to find any that don’t.
Disruptions are happening in the data and predictive analytics market, starting with the emergence of Business Intelligence – a new way for data visualization.
As someone who has been conducting data science research and predictive analytics for a while, I have lived through the time where all analysis was done through hardcore coding. To conduct an accurate business intelligence analysis and understand the patterns in the data, I need to write codes to slice-and-dice the data in different ways, and then write codes to visualize the results from the segmentations. To conduct predictive analytics and uncover hidden insights from the data, I need to write codes to formulate the regression model and test different variables to discover which are the significant ones and quantify the impact. With all the potential variables that might have significance, it usually takes weeks, or even months, combined with good luck, to achieve a statistically meaningful and business sense-making discovery from the data.
These deficiencies: the lack of scale, speed and application are indeed the factors that have been stopping traditional business intelligence and predictive analytics from driving real business value. The current business world is discovering bigger and more complex data and is in need for strategic analytics that demands faster time to reach actionable insights. The gap between business needs and reality is the reason that data analytics still has limited applications in the business world despite the length of time it has been around.
But the bright side is on the horizon. With the increasing computing capability and the power of AI, analytics will see enormous improvements on scale, speed and application. And along with these innovative changes, there comes the disruption in the data and analytics market. The first wave of disruption happened about 10 years ago, where coding-based data analysis platforms start shifting towards visual-based data discovery platforms. With that, tools like Tableau, Qlik and Microsoft Power BI emerged to disrupt the market and completely change the way analysts visualize and present their data. Because of this disruption, we officially entered the world of widely implemented modern Business Intelligence.
Of course, there is a learning curve to these new tools but after basic use, the return is enormous. The visualization enabled by these tools exceeds the wildest imagination of coding-based platforms. Being code-free, fast, and interactive – this disruptive innovation brought data visualization and BI to another level. With this power, firms start to realize the benefit of BI and you would be pressed to find a large enterprise who does not have a business intelligence group, crunching data and developing dashboards to help the business. In Gartner’s 2017 survey for “Magic Quadrant for Business Intelligence and Analytics Platforms” more than 65.4% mid-size enterprises reported ‘excellent/good’ when answering how well various business benefits were achieved from implementation of analytics and BI adoption.
The disruptive innovation does not stop at Business Intelligence. The next wave of disruption is going to be in the next level: Predictive Analytics.
Although the breakthrough in BI has been proved to be great in helping companies understand the question “What is happening?” with their data, the journey does not stop here. Since there are so much more can be done with data, especially through predictive analytics. Using predictive analytics, we would be able to answer much deeper questions beyond “what happened?” into questions like “Why it happened?” or “What could happen?” and “How can I improve it?” – In other words, predictive analytics looks into the future and provides actionable insights to directly drive business values.
The lack of speed and scale is again the roadblocks keeping predictive analytics from being practical and driving real value. It usually takes weeks, if not months, to run millions of regression models with different combinations to identify key influencers and develop hypothetical scenarios to predict the impact. This is again where AI can come in and help. Compared to the BI breakthrough, we are now at a point where we are expecting another market disruption with “Augmented Analytics” where predictive analytics is combined with the power of AI. Gartner uses the term “Augmented Analytics” to address that the idea is not to eliminate human from the analytics loop, but to provide deeper information to the person designing the analytics methodology, therefore to assist better decision making.
So, what does this next wave look like, and how is it different? The augmented analytics means an augmentation in all the key elements in the data and predictive analytics workflow: data preparation, data modeling to find insights, and the sharing of the insights to make it actionable.
- Data Preparation. The old process of manual data preparation, investing heavily on human effort to format data, detect data error and irregularity will be eliminated. Under the scheme of augmented analytics, algorithms will automatically detect data quality issues, catalog and recommend enrichment, and build data lineage and metadata.
- Data Modeling. The old way to conduct data modeling and extract insights is through manual exploration of data using interactive visualization, or through manual engineering and model building to find patterns in the data. Augmented analytics introduces the concept of “Natural-Language Query”, where algorithms find all relevant patterns in data and bring it to you. The proper models are auto-selected and validated to find the best way to depict the data, and code is auto-generated.
- Sharing and Operationalizing Findings. In the world of BI and traditional predictive analytics, dashboards, and interpreted storytelling are the way to go. It heavily depends on users to interpret results, and it further requires users to build scenarios and forecast to make actionable recommendations. With augmented analytics, insights are narrated in natural language or visualizations to focus user on what is important and actionable. It also automatically builds scenarios to make predictions, and then provide concrete recommendations with measurable outcomes.
Pioneering vendors are eagerly building augmented analytics (AI + Predictive Analytics) into their capability; and early adopters are seeing real return of investments in their businesses.
In the next 2 to 5 years, augmented analytics would bring yet another transformation in the data and analytics market. Some vendors are already on the forefront of this wave. A few examples include Salesforce Einstein Discovery (a $1.1 million acquisition of the startup Beyondcore), ThoughtSpot, Microsoft Power BI (Quick Insight feature), IBM Watson analytics, SAP Cloud analytics, and much more, infusing AI and machine learning to automate data science modeling. It’s not only the Tech-giants in the data analytics market that are innovating their tool, there are also numerous start-ups in this young and uprising arena, such as SparkBeyond, DataRobot, H2O, and Tellius. This indicates that augmented analytics has vibrancy and promising future. These cutting-edge analytics platforms are already being used by Fortune 500 companies in the Finance, Manufacturing, Life-Sciences, Energy, e-Commerce, Internet and Healthcare industries.
These early adopters are seeing real returns from using “Augmented Analytics” in their business. A large US bank has partnered with one of the AI analytics platform to transform into a more customer-centric organization. With a large customer base, bankers and financial advisors knew they could work better together if they had a more complete picture of customer interactions. With the predictive tool, it has been able to reduce data siloes and better understand their customers. In the wealth management business, for example, analyzing client flows can be complex. Cash movement inside and outside the company could be completely normal between a customer’s accounts, or it could be an indicator that a client is broadening a relationship with another wealth provider. The partnership with augmented analytics helped U.S. Bank cut through the clutter and understand client flows more completely, so they could act on them appropriately.
The tool also provides the U.S. Bank team with more information and insights about all their customers and present opportunities to deepen relationships using customized approaches. As the 5th-largest commercial bank, the company wanted to better understand the overlap between their retail banking clients and their wealth management clients. They began by analyzing which retail banking customers are most likely to become new wealth management clients. They found that wealthy young clients between the ages of 20-35 are more likely to transition into wealth management. This was a surprising insight as it was previously believed that more mature clients were more likely to pursue wealth management. The US Bank team can now use this insight instead of long-believed intuition to develop more targeted marketing strategy to convert retail banking clients into wealth management.
Ten years ago we struggled to find a handful of machine learning/AI enabled business applications. Ten years from now we will struggle to find any that don’t. Disruptive innovations can be intimidating at first, but it won’t stop because you choose to ignore its approaching. The best strategy is to prepare and get ready for it – starting with baby steps. In the next article we will talk about how to evaluate your business’s maturity for augmented analytics, how to prepare your business for it, and how to strategize your analytics practice to take advantage of this technology of the future.
