If you’re reading this blog post, you’re probably tired of wandering into meetings where your colleagues debate the proper definition of “important” terms such as enterprise performance management (EPM), business intelligence (BI), business analytics, master data management (MDM), data scientist…and, my favorite, big data. While confusion reigns in your working group about what these terms mean, you’re undoubtedly also experiencing requests for different kinds of analysis and reporting from people in your organization who never used to care about this stuff! And they, too, are asking you about this elusive “big data.”
So what’s really going on here? Think back (if you’re old enough) to the shift to the Internet; or a little further back to client server; or even earlier, to the shift from mainframe to other varieties of computing. We’re in the middle of one of those tectonic shifts in the technology world that comes along fairly infrequently, and causes a lot of discomfort until things shake out! And that annoying, elusive, catch-all term, “big data” is at the heart of this change.
The philosopher Aristotle once said, “He who thus considers things in their first growth and origin … will obtain the clearest view of them.” So how did the big data concept evolve?
As described in Wikipedia’s big data “Definition” description, “…analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional…increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Gartner, and now much of the industry, continues to use this “3Vs” model for describing big data. In 2012, Gartner updated its definition as follows: “Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.”
Another big transformation is afoot that only adds to the confusion: the proliferation of various technologies to address the need to get some of these new kinds of data into an organization’s analytic environment. What does this mean for your technology environment intended to support your organization’s data discovery, analysis, and reporting needs?
A picture’s worth a thousand words, so let me use Performance Architects’ very own business analytics architectural stack to illustrate how quickly – and how much – the business analytics architecture has had to evolve to address the big data revolution.
This view of a “recommended” business analytics environment from 2012 shows an architecture familiar to any of us who have participated in business analytics initiatives in our organizations over the past several years:
Now look at the same chart, updated in 2015:
See any differences (hint: everything in purple)? This is where I arrive at my five recommendations for how to evolve your business analytics environment to address the big data revolution!
Recommendation #1. Consider all data in your organization to have equal importance (until proven otherwise).
This mind shift is the most important element you’re personally going to need to make, and that you’ll need to socialize with your colleagues, if you’re going to actually implement anything “big data” in your organization.
Transactional or more structured data is what we’re used to dealing with in our work today. This is the data you have to cleanse for reporting and/or regulatory reasons…and it’s always historical. Examples include financial data in your general ledger (GL) and customer data in your customer resource management (CRM) system. The issue is that the BI and EPM software categories grew up to analyze just transactional data, because frankly that’s all organizations were collecting in a digital, analyzable format 10+ years ago! Most other data sets were considered “garbage” because they were too “dirty” (aka: disorganized) or too difficult to access and store.
“All other data” outside of transactions is where big data mostly comes into play (although there are rightfully organizations that collect such massive amounts of transactional data that it can be considered “big data”…but that’s a topic for another blog post). I bucket this information in two categories: 1. Unstructured data (really raw data streams that aren’t created to be “officially” analyzed, such as personal documents or videos, etc.) and 2. Semi-structured data (machine and sensor data gathered in large quantities that often need to be analyzed at almost near-real-time speed).
Notice anything about these buckets of data? This content has the power help you to influence the outcomes of the transactional data before it becomes a historical fact!
Think about the possibilities. What if your executive team could use a search engine to determine the sentiment of each sales and customer service team about prospective and current customers with deals in the pipeline for that quarter, allowing leadership to intervene if the team’s consensus is that a deal is going sour? How about mining files on a student or an employee to see trends in behavior that predict an important outcome, such as quitting school or work as a result of financial or familial pressures?
Now do you start to see why all data matters until proven otherwise?
Recommendation #2. Stop thinking you need to know the answer before you analyze the data (look for correlations first).
Many of us think we need to design a business analytics environment from the “top down,” meaning we need to understand what information we want to glean from the data in order to set up the environment. That’s why most BI + EPM or business analytics projects follow a predictable “waterfall” implementation path:
I recommend you shift your methodology to add a step in advance of your traditional project activities to address big data:
The “Discover Correlations” step means rather than assuming causation based on statistical analysis of your historical data, you can implement a variety of solutions on top of all of your data (because, remember, all data is important until proven otherwise!) to understand possible correlations between variables that you’ll then want to analyze further in your business analytics environment to make sure you’re focused on changing outcomes by influencing the behavior that causes these outcomes. This gets us to Recommendation #3…
Recommendation #3. Implement data storage and integration technologies that allow some kinds of data to remain “messy” (everything does not need to be stored in a database).
I already discussed the three “V’s.” The biggest issue right now with current technologies in place in most organizations is the variety of data, meaning that data is coming in a variety of formats that traditional BI and EPM solutions can’t handle. You need to make sure that your organization looks into and evaluates technologies such as Hadoop and NoSQL to access content that isn’t stored and retrieved using methods common to database technologies. Some of the larger and/or more thought-leading technology vendors are now starting to come up with technologies to unify these different storage methods, such as Cloudera’s Impala, IBM’s Big SQL, and Oracle’s Big Data SQL.
Recommendation #4. Make sure key terms are defined and measureable.
Get away from discussing “one version of the truth.” Instead, focus on knowing your data and data sources, and then making sure that your team agrees on how the data is defined depending on how it is used inside your organization. For example, a “region” may mean different things to your sales, customer service and legal teams…but they are all equally important and valid definitions for the organization. These definitions can be housed in something as simple as an internal Wiki or a high-powered master data management (MDM) solution.
And, of course, the first definition you need to get straightened out is…big data.
Recommendation #5. Evolve information delivery to meet current lifestyles.
Okay, answer truthfully here. Have you pulled over to the side of the road to send a note or report from your phone because your boss wants a real-time update on how your team’s performance is proceeding against plan? Have you attended one of your kid’s sporting events but spent the whole game in frustration trying to access an important report that crashes on your phone? Have you snuck away secretly at a reception, play, or movie to check work emails and attachments?
Let’s be honest…sure you have. We all have. And that’s the point. Our work lives are evolving to overlap with personal time, and are allowing us more freedom to do what we want, when we want, as long as we can access the information we need to do our jobs!
So rather than rely on traditional BI solutions that were designed when everyone had desktop or very heavy laptop computers, architect a reporting and analysis environment to address the modern worker’s needs. This means mobile-friendly solutions that are thin (or no) client; that can be consumed “on the fly;” and that are useful with little to no instruction (think Amazon or Facebook user interface design).
If you’d like help thinking through alternatives, or if you want to learn more about how Performance Architects can help in this arena, please fill in the contact form here and we will follow up with you: as soon as possible!
Author: Kirby Lunger, Performance Architects