Big Data can seem complicated and overwhelming to many in the enterprise, especially to non-technical business folks. Indeed, predictive analytics and machine learning are complex disciplines. That’s why data scientists—those mythical creatures with a harmonious blend of math skills, statistics expertise, business acumen and a knack for storytelling—are in such high demand. But the conceptual framework for operationalizing Big Data in the form of smart applications follows a fairly simple formula—or recipe—that almost any technical professional can grasp. It’s not for unicorns alone.
Smart applications are those applications that include data-driven features that deliver actionable insights to users in the right context, at the right time.
It’s important for application developers and data professionals alike to understand, at a conceptual level at least, this recipe because smart applications that deliver information in context are the key to gaining competitive advantage from Big Data. By understanding what makes smart applications smart, developers, data scientists and executives are better positioned to come up with new, innovative use cases that impact not just the bottom line, but also propels innovation and topline growth.
So let’s review the recipe for smart applications, which boils down to a handful of ingredients (with data being the secret to the sauce), and five fundamental steps.
Step 1: Engage.
The key ingredient of any smart application is usage data (i.e., data created by users engaging with your application.) This includes data that users create explicitly (such as transaction data created when a user makes an order in an application) and implicitly (such as click-stream data users leave behind as they navigate throughout an application.) Effective smart applications use this type of data to understand their users’ behaviors. So step one is simply to develop an application in such a way that it encourages continuous and frequent user engagement in order to create large volumes of usage data.
If you’re thinking that’s harder than it sounds, you’re right. Despite the multitude of apps on the average user’s smartphone, typically only a handful are used heavily. Make sure your apps are easy to use and deliver immediate utility. Utility leads to use, which leads to useful data.
Step 2: Capture
The next step is to capture, store and process all that usage data, combining it with other data sources as needed. This requires building data pipelines to move all that usage data to some type of operational persistence layer in real- to near-real time. The data must also land in an analytical data store, though that process does not necessarily need to take place in real-time. Batch loading of data to analytical stores is sometimes acceptable. The type of operational persistence layer (in-memory data grid, key/value store, relational database, etc.) and analytical data store (MPP analytical database, Hadoop, etc.) will vary depending on use case and data structure.
Step 3: Model.
Note that steps 3 and 4 don’t necessarily need to take place in linear or synchronous fashion. Once usage data is captured in an analytical data store, the next step is to analyze the data to develop predictive models that generate actionable insights. In most cases, this process happens offline (i.e. not in real-time.) Data scientists often enrich application usage data with other data sources that contain predictive value prior to building their models to improve the accuracy and utility of insights. Other data sources could include historical application usage data (that is, data past application users created, either implicitly or explicitly), customer data, demographic data, weather data, financial markets data, or product data, depending on the use case in question. The important point in step 3 is that the end results are working predictive models.
Once you predict an event, then you can optimize a decision. For example, once you can predict how long it will take for various vehicles in a fleet of cars or trucks (or boats for that matter!) will reach a certain destination, you can optimize which vehicle should be routed to that point. Here you are potentially optimizing that arrival time against other factors, such as load capacity and operator history. Nonetheless, this optimization exercise is actually a second model that needs to be built, particularly to move to the next stage.
Step 4: Suggest.
During step 4, predictive and/or optimization models are applied to live, discreet application usage data in real-time to generate actionable insights. Remember, some input data has already landed in real or near-real-time in some type of operational persistence layer (step 2.) By discrete application usage data I mean data associated with a particular user session as it is happening. Once the models are applied and actionable insights are created, insights are surfaced back to the application in a form that is useful for and actionable by the user. This could take the form of a suggested action for the user to take or an estimate on when an upcoming event will take place. Again, the actual insight and resulting suggestion will depend on the particular use case. Finally, the user takes an action based on the suggestion.
Step 5: Repeat.
Now that the user has acted on the suggestion surfaced in the application, the process repeats itself again and again. Unlike a recipe for bundt cake or tuna casserole, the recipe for smart applications has no end. In fact, the more cycles the smarter the application becomes as predictive models get refined on ever large volumes of data.
To apply some context to this admittedly conceptual recipe, some smart applications you may already be familiar with include Netflix, Uber and Amazon. Netflix follows its version of the smart applications recipe to serve up personalized movie and TV recommendations. Uber accurately forecast fares and predict when your car will arrive to pick you up. And Amazon’s “Customers Who Bought This Item Also Bought” feature is the result of its take on the recipe for smart applications.
This recipe is, of course, a simplified version of the steps required to develop and refine smart applications. But it hopefully gives you a better understanding of what makes smart applications smart and serves as the basis for brainstorming with both your business-side and more technical colleagues about how to put Big Data to work for your enterprise.
To learn more about smart applications and operationalizing Big Data, join me and Constellation Research’s Doug Henschen for a live webinar, Insights to Action: How to Transform Your Business with Smart Applications, on August 17, 2016 at 11am PT. Register here.
