Tapping Into Non-Relational Data

Modern BI and Analytics programs use non-reatlional data management for six key functions. You should be cognizant of these functions when you add non-relational technology to your data architecture.

Kinds of Non-Relational Storage

Most IT professionals are familiar with the RDBMS. Relational databases store data in tables that are defined in advance. The definition specifies the columns that comprise the table, their data types, and so forth. The design is referred to as a data model or schema.

Relational storage is immensely useful, but it is not the only game in town. There are several alternative types of data storage including:

• Key-value stores store data in associative arrays comprised of sort keys and associated data values. These flexible data structures can be distributed across nodes of commodity hardware, and are manipulated using distributed processing algorithms (map-reduce). Hadoop functions as a key-value store.

• Document stores track collections of documents that have self-defining structure, often represented in XML or JSON formats. A document store may be built on top of a key-value store. MongoDB is a document store.

• Graph databases store the connections between things as explicit data structures (similar to pointers). These are stored separately from the things they connect. This contrasts with the RDBMS approach, where relationship information is stored within the things being associated (keys within tables). Neo4j is a graph database. 

Reasons for Non-Relational Storage

In the age of big data, organizations are tapping into these forms of storage for a variety of reasons. Here are just a few:

• No model: not requiring a predefined schema enables storing new data that has yet to be explored and modeled.
• Low cost: the cost of data management can be significantly lower than RDBMS storage.
• Better fit: some business use cases are a natural fit to alternative paradigms.

But don’t simply add a box to your data architecture for non-relational data. You must plan for specific usage paradigms, and make sure your architecture and processes support them.

Uses for Non-Relational Storage

There are six major use cases for non-relational data in modern data architectures:

• Capture  Non-relational storage facilitates intake of raw data.  New data sets are captured in non-relational data stores, where they become “raw material” for various uses. A non-relational data store such as Hadoop can be used to capture data without having to model it first. This is often referred to as a data lake. I prefer to call it a landing zone.

• Explore  Non-relational storage facilitates exploration and discovery. Exploration is the search for value in new data sets. Exploration applies analytic methods to captured data, often combining it with existing enterprise data. The goal is to find value in the data, and to identify things that will be worth tracking on a regular basis. 

• Archive  Non-relational storage serves as an archive. Data for which immediate value has not been identified is moved to an archive. From here it can be fetched for future use. Archiving data helps ensure the data lake does not become the fabled “data swamp.” An alternative to archiving unused data is simply to purge it.

• Deploy  Non-relational storage supports production analytics. When value is found in data, it is transitioned to a production environment and processes are automated to keep it up to date. Deployments range from simple reports to complex analytic models.

• Augment  Non-relational storage serves as a staging area for the data warehouse. In many cases, the insights gained from exploration prove valuable enough to track on a regular basis. Augmentation is the process of adding elements to a relational data warehouse that come from non-relational sources or analytic processes. A credit score, for example, might be incorporated into a customer dimension.

• Extend:  Non-relational storage expands what can be maintained in the data warehouse. Sometimes there is lasting value in non-relational data, but is not appropriate to migrate it to relational storage. In such cases, the non-relational data is moved to a non-relational extension of the data warehouse. Applications can link relational and non-relational data. For example, non-relational XML documents may made available for “drill down” from a dimensional cube.

In addition to these six primary use cases, non relational platforms may serve several utility functions. These include staging, data standardization, cleansing, and so forth.

Learn More

Join me for my course Data Modeling in the Age of Big Data, offered exclusively through TDWI. At the time of this writing, it is offered next at TDWI Chicago on May 11, 2017. You can also bring this course to your site through TDWI Onsite Education. For more information contact me.

Is Your Team Losing the Spirit of the Agile Manifesto?

As the adoption of agile BI techniques spread, it is easy to become wrapped up in methodology and lose site of the agile spirit.  

This year is the 15th anniversary of the Agile Manifesto. Mark the occasion by refocusing on the agile principles.

The Age of “NO”

Fifteen years ago, most business software was developed using rich but complex methodologies. Businesses had spent years refining various waterfall-based methods, which were heavily influenced by two decades of information engineering.

The result seemed to make sense from IT’s point of view, ensuring a predictable process, repeatable tasks, and standard deliverables.

To the people who needed the systems, however, the process was inscrutable.¹ It seemed like a bewildering bureaucracy that was managed in a foreign language. They were always being told “no.”

Can we add a field to this screen?
No, you would have had to ask that before the requirements freeze.
Can we change the priority of this report?
No, that decision must be made by the steering committee.
Can we add a value to this domain?
No, that is the province of the modeling group.

The groups were looking at software development from completely different perspectives. They were not really collaborating.

Business people were asking for functionality. IT was beating back the requests by appealing to methodology.

Enter Agile

In 2001, a group of developers met in Colorado to talk about what was working and not working in the world of software development. They produced the Agile Manifesto:

Manifesto for Agile Software Development

We are uncovering better ways of developing

software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on

the right, we value the items on the left more.

Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas

© 2001, the above authors

this declaration may be freely copied in any form, but only in its entirety through this notice. 

The thrust of the manifesto is on collaboration between business and technical personnel, with an emphasis on visible business results. That is, while methods are important, results are more important.²

The Agile Manifesto helped refocus software development on the product: functional business applications.

Agile Today: The Danger of Cooptation

Fifteen years later, it is safe to say that “Agile” has permeated the mainstream. This is largely a good thing. But whenever something moves from being a new alternative to wide acceptance, there is a potential dark side. As new ideas spread, they can be misunderstood or corrupted; adoption becomes cooption.

I frequently see signs of back-sliding to The Age of “No.” Even among developers who follow agile-based approaches, there is a tendency to lose sight of the agile principles. Here are two examples from my recent experience:

A team had decided to implement an unusual data model. It could easily return incorrect results if not queried in specific ways. The recommendation was to write some extra documentation explaining the pitfalls, and a short tutorial for developers. The response: “We cannot do that. We are an Agile shop. We cannot produce documentation that is not auto-generated by our tools.”

On another occasion, a team was developing the scope for several work streams. One set of needs clearly required a three-week collaborative activity. This was expressly forbidden. “Our Agile approach requires everything be broken down into two-week sprints.”

In both cases, the response was couched in methodological concerns, with no focus on the business benefit or value.³ 

This is precisely the kind of methodological tunnel-vision against which the Agile Manifesto was a reaction.

Keeping the Faith

It is hard to disagree with the agile principles, regardless of your organization’s commitment (or lack thereof) to an agile process.

You can take one simple step to ensure that you are not losing touch with agile principles:

     Whenever you are tempted to say “no,” pause and reflect.

Why are you denying the request? Is it simply based on process? Think about what the person actually wants. Is there value? Is there a way to address the business concern?

Sometimes “no” is the right answer. But always be sure your evaluation places business capabilities and benefits ahead of process and procedure.


Learn More:

Read more posts about applying agile principles to BI and analytics:

• Create Social Documentation (July 8, 2015)
• Document Information Requirements Graphically with BDM Diagrams (February 10, 2014)

Notes

1. These people were often referred to as “users.” Over time, this became a derogatory term.
2. Agile is often misinterpreted as emphasizing speed.
3. Luckily, both of these teams saw fit to make exceptions to their processes, prioritizing business value over method.

Chris Adamson on Modeling Challenges

In a recent interview, the folks at WhereScape asked me some questions about data modeling challenges.

In Business Intelligence, modeling is a social activity. You cannot design a good model alone. You have to go out and talk to people.

As a modeler, your job is to facilitate consensus among all interested parties. Your models need to reflect business needs first and foremost. They must also balance a variety of other concerns — including program objectives, the behavior of your reporting and visualization tools, your data integration tools, and your DBMS.

It’s also important to understand what information resources are available. You need to verify that it is possible to fill the model with actual enterprise data. This means you need to profile and understand potential data sources. If you don’t consider sources of data, your designs are nothing more than wishful thinking.

When considering a non-relational data sources, resist the urge to impose structure before you explore it. You’ve got to understand the data before you spend time building a model around it.

Check out the video above, where I discuss these and other topics. For a full-sized version, visit the WhereScape page.

Create Social Documentation

Documentation is sometimes viewed as a necessary evil. But it doesn't have to be. Here's how to produce documentation that will be used.

Useful documentation gets used -- during all development phases, and by all interested parties.

Burdensome methodologies often expend precious hours producing documentation that is hard to use. Many projects leave behind fat binders of text that hardly anyone will ever open. These examples have given documentation a bad name.

The good news is that documentation can be done right. It does not have to be a drag on project time, it does not have to be a chore to read and review, and it does not have to be something we interact with alone.

Why we need documentation

Documentation is not an after-the-fact explanation of what has been built. Used properly, it is a central component of the entire lifecycle of a BI solution.

Important uses include:

1. Prior to development: Identify and validate requirements and designs
2. During development: Specify what to build
3. After development: Educate business people and support personnel

Of course, there are many other areas in which documentation has value (program planning, governance, change management, etc.). These three above are sufficient to illustrate the value of social documentation.

Social Documentation

Useful documentation should be easy to read and discuss. It should also not be burdensome to produce. Three principles shape social documentation.

Social documentation is the focus of collaboration. 

Whenever possible, I recommend to my clients that we use PowerPoint for documentation. Why? Word processors are tailor made for reading, which is a solitary activity. Presentation software is tailor made for collaboration.

Social documentation is easy to navigate. 

Support "random access" rather than "sequential access." Presentation software is great for this; we can easily sort and navigate slides by their titles. This can also be achieved using document maps or outlines.

Social documentation is not prose. 

Each slide in a presentation, or section in a document, should be set up to capture essential information in a consistent format.  This format may be tabular, diagramatic, or both. Your subject matter will dictate the appropriate format.

But here is the important part:

• No paragraphs
• No prose
• If using PowerPoint: No bullet lists. (They're just a back door to writing paragraphs.)

Uses for social documentation

I find the presentation format excellent for defining program priorities, defining project scope, capturing business requirements, developing top level information architectures, and a variety of other tasks. For specifications, a word processed document with multi-level headers and a document map typically fits the bill.

When documenting business metrics for a dashboard or scorecard, for example, set up a PowerPoint presentation with one slide per metric. Use a standard tabular format to document each metric. This documentation is easy to produce, review and revise, as I will discuss in a moment.


Where presentation software is not practical, word processors can be used in the same way. Divide the document into sections, activate the contents sidebar, and use a consistent tabular format.

Of course, not all documentation is captured in this manner. For example, we might use social documentation to capture a top level star schema design, then use a modeling tool to produce a detailed design.

Advantages of Social Documentation

This simple approach has numerous advantages.

Frictionless and Comprehensive

During requirements specification, social documentation allows you to capture the necessary information in frictionless and comprehensive manner. A standard tabular format, for example, ensures the same items are filled in. The presentation itself is easy to navigate via sections and slide titles.

Engages with the business

Social documentation invites collaboration. Give people a big fat binder and their eyes will cross. Show them 5 or 6 slides that capture the business metrics they care about, and they will give you feedback.

I always have my laptop with me, so if I happen to be in a room with a SME, I can pull it out, flip to the correct slide, and ask a question.

Incidentally, collaboration with the business is one of the cornerstones of the agile manifesto.

Reviewed together, rather than in isolation

Ever sent out a fat document for review? If you have, you know the results are not good. Most people will not review it by the deadline. When reminded, they will say, "it looks good." A precious few will provide detailed feedback.

Social documentation transforms this process. A review is conducted by bringing people into a room and reviewing the deck. Any agreed upon changes are made directly to the presentation slides.

The documentation is now ready for the next tasks: guiding development and then serving as the basis for education.

Learn More

Read more about documenting BI program activities in these posts:

• A great diagramming technique for information requirements: Document Information Requirements Graphically with BDM Diagrams (February 10, 2014.)
• Recommended approach to documenting dimensional designs: Dimensional Modelers Do Not Focus on Logical vs. Physical (July 5, 2011)

For more details on what to document, check out my book Star Schema: The Complete Reference. Detailed descriptions and examples can be found in Chapter 18, "How To Design And Document A Dimensional Model.”

I also discuss documentation of information requirements and business metrics in the course “Business Information and Modern BI.”  Check the sidebar for current offerings.

BI and the Path to Business Value

Managing BI services requires a consistent information architecture, even if different teams are responsible for data marts, performance management, and analytics.

Business Value From BI

Business Intelligence is the use of information to improve business performance.^[1] To improve business performance, we must do three things:

• Track business performance
• Analyze business performance
• Impact business performance

Each step on the path to business value is supported by two kinds of BI services, as shown in the illustration.

• Tracking performance requires understanding what is currently happening (Performance Management) and what has happened in the past (OLAP).
• Analyzing performance requires the ability to get to detail (OLAP), develop insight into cause and effect (Business Analytics).
• Impacting performance requires targeting a business metric (Performance Management) and taking a prescribed course of action (Business Analytics.)

Each of these steps leverages a pair of BI services, and each service shares a common interest in business information.^[2] Managing BI services therefore requires a consistent information architecture. This is true even when separate teams manage each area.

Tracking Performance

Understanding performance often starts with summarized data on dashboards and scorecards (Performance Management). The need investigate potential problems requires detailed data and history (OLAP and Reporting.)

As Wayne Eckerson demonstrated in Performance Dashboards, both these areas provide stronger business value when they are integrated. For example, a dashboard is more useful when someone can click a metric and bring up supporting detail in an OLAP cube.

To successfully link Performance Management and OLAP, the two domains must share common definitions for business metrics (facts) and associated reference data (dimensions). Metrics must be calculate in the same way, linked to reference data and different levels of detail, and synchronized (if managed separately).

Analyzing Performance

Analyzing performance is the process of breaking down what has occurred in an attempt to understand it better. Slicing and dicing an OLAP cube is a form of analysis, providing insight through detail. Analytic models provide a deeper level of analysis, providing insight into cause and effect, and extending this to the future through prediction.

OLAP is largely focused on exploring various aggregations of business metrics, while analytics is largely focused on the underlying detail that surrounds them. Our OLAP solutions provide historic detail to Business Analytics in the form of data from the data warehouse.^[3]

The exchange flows the opposite direction as well. Business analytics develop insights that suggest other things that should be tracked by OLAP services. For example, a particular set of behaviors may define a high value customer. This assessment is developed using Business Analytics, and applied to the customers in the OLAP data mart. For a fun example from the world of sports, check out the book Moneyballby Michael Lewis.^[4]

Improving Performance

All of this is somewhat academic if people in the business do not use all this information to make decisions. Business impact occurs at the intersection of Performance Management (which tells us what is important and how we are doing) and Analytics (which suggests the best course of action.)

Every analytic model targets a business metric or key performance indicator (KPI) from the world of performance management. That same KPI, in turn, can be used to measure return on investment of the analytic model.

For example, a direct sales manager of enterprise software wants to reduce cost of sales. An analytic model is developed that assesses the likelihood of a prospect to buy enterprise software.

The manager begins using the prospect assessment model to prioritize the work of the sales team. Less likely prospects are reassigned to a telesales force. Over the next two quarters, cost of sales starts falling. The same KPI that the analytic model targeted is used to measure its return on investment.

Information as an Asset

It is common to manage each of the pillars of Modern BI as a separate program. The path to business value, however, requires that these programs share a consistent view of business information. BI programs that are not centralized must carefully coordinate around a common information architecture.

Further Reading

1. For more on this definition of business intelligence, see Business Intelligence in the Modern Era (9/8/2014)
2. The three service areas are explained in The Three Pillars of Modern BI (2/9/2015).
3. Sometimes analytic modelers bypass the data warehouse, but there are steps you can take to make this important repository more useful. For tips on how to make your data warehouse more useful to analytic modelers, see Optimizing Warehouse Data for Business Analytics (9/25/13). Note that even with a well designed data warehouse, analytic models often augment this enterprise data with additional data sources.
4. The Oakland A's used analytics to re-evaluate the basic metrics used to assess the value of a baseball player. See Business Analytics and Dimensional Data (7/17/13).

** Interested in learning more about modern BI programs?  **

Check out my new course, Business Information and Modern BI: Evolving Beyond the Dimensional Data Mart.  Offered at TDWI conferences and onsite.  See the sidebar for upcoming dates.

The Three Pillars of Modern BI

Data marts are no longer sufficient to meet the demands of a modern BI program. This post lays out a framework for delivering BI value in the modern era.

The technologies and processes that help us deliver BI services have advanced by leaps and bounds over the last two decades. A modern BI program provides three perspectives on business performance, roughly corresponding to the past, present, and future.

OLAP and Reporting

OLAP and reporting services (or simply "OLAP") provide the "official record" of what has happened in the past--the canonical log of business activity.

This pillar of the modern BI program helps the business understand "where we've been." The typical information products provided in this service area include:

• Reports provide pre-built, parameterized access to business information
• Analysis provides the ability to explore the official record of business activity by slicing, dicing, drilling, and so forth (OLAP)
• Ad hoc query capabilities allow people to ask their own questions about the official record, even if a pre-defined report or analysis does not exist.

For people in the business, these kinds of information products come to define this pillar of the BI program. There is also a fourth important information product of which the business may have less direct awareness:

• The integrated record of business activities, aka "Data Marts." This record combines, standardizes and organizes information for business consumption.

Essential in delivering the first three kinds of information products, this component was the primary focus in the early years of BI, when we called the practice "data warehousing." Since then, the discipline has changed and expanded. But it is still essential that the BI program provide the ability to understand the past.

Performance management

Performance management services provide real-time status on key performance indicators, as well as performance versus goals.

KPI's and goals are carefully matched to the viewer's role and linked to business objectives. Goals communicate expectations, while KPI's communicate achievement of expectations.

If OLAP is about "where we have been," then performance management is about "where we are now." Typical information products in this BI service area include:

• Dashboards provide real-time or near-real-time status of KPI's
• Scorecards which communicate progress vs. goals

Information on dashboards and scorecards is carefully tailored for the user or functional area. Metrics are chosen for relevance and actionability, linked to business strategy, and balanced to reflect a holistic picture of performance.

While this service area can stand on its own, performance management solutions are more powerful when people can dig into the KPI's on their dashboards. This capability is enabled by integrating performance management services with OLAP services.

Business analytics

Analytic services probe deeply into data, providing insight into cause and effect, making predictions about what will happen in the future, and prescribing a course of action.

While analytic services draw on data from the past, their objective is to influence the future. Typical information products in this service area include:

• Analytic models that make sense of activities or predict future events
• Simulations that allow the manipulation of variables to study their potential impact on results
• Visualizations that communicate analytic insights
• Analytic metrics that assess current state and or future outcomes which are fed to OLAP, performance management, and OLTP applications

Like the other pillars of modern BI, analytic services can exist alone but are more powerful in the presence of the other pillars. Prescriptive metrics, for example, are best presented directly on operational dashboards; useful analytic metrics can be recorded and tracked over time in data marts.

Delivering Modern BI

In each area of the business, these capabilities should be balanced and tied together. Centralized management of all three pillars is not required, but they should be coordinated and integrated. A shared roadmap should lay out their planned evolution.

Your objective is business impact, and my next post shows how these services deliver it.

Learn More

• BI and the Path to Business Value (3/20/2015) explores how the three pillars of modern BI enable business impact.
• Business Intelligence in the Modern Era (9/8/2014) provides a definition of Business Intelligence for modern information assets.

For more on managing the Modern BI program, check out Chris's latest course: Business Information and Modern BI. Check the sidebar for upcoming dates.

Business Intelligence in the Modern Era

This post offers an updated definition for BI, and suggests that you don't have to think about it as a box on an org chart.

BI has changed a lot in the last two decades. Technologies and best practices have evolved, and we've found more ways in which a BI program can deliver value. Some of these innovations have occurred outside of IT or the BI Competency Centers that many businesses have established. At the same time, many organizations are moving to make business units autonomous.

These changes lead many people to ask what exactly is BI? Is it a box on the org chart? Does it include analytics that were never done by IT? How do data governance and master data management fit in?

Business Intelligence Defined

I define BI as follows:

Business Intelligence:

The use of information to improve business performance

- Chris Adamson

The first thing to note about this definition is that it does not address any specific technologies or methods. These aspects change over time, and they certainly influence what we may be able to achieve.  But the objective is always to provide business value.

Secondly, note that this definition is not beholden to the boundaries of a departmental structure. Regardless of who develops, supports or uses solutions, it's all considered BI.

Let's take a quick look at both these aspects.

BI Services and Activities

The reason we commit resources to BI programs is simple: we intend to use information to deliver some kind of business value.  The definition has been crafted to cover any activities that support this objective.  It can be used to describe a variety of activities that provide business value, both old and new.

Among the older activities it covers:

• Traditional reporting, OLAP and ad hoc functions
• Dashboards and scorecards
• Traditional data warehouses and/or data marts
• Data integration services

At the same time, some newer uses of information are covered:

• Business analytics and predictive analytic
• Master data management
• Data governance
• Virtualization and federation services

The definition also covers activities that some people think of as on "the other side of the fence" from BI:

• Transaction processing

That's intentional; transaction processing manufactures much of the "raw material" that BI programs attempt to leverage.  When we plan an operational solution, we should be thinking about these downstream uses.

BI and the Org Chart

While you may have a group responsible for BI program management, it is important to understand that the scope of BI reaches well beyond this group. The delivery of business benefit from information impacts the entire organization.

Some of the functional areas that participate in BI are:

• Business units  All of the value from BI happens within business areas that use information. This is where decisions are made and impacts are realized.  For many businesses, responsibility for development of BI solutions also lies in business areas.  This is particularly the case for analytics, but also increasingly for the traditional forms of BI.
• BI Competency Centers  Whether part of IT or external to it, many organizations have established a centralized resource for planning and overseeing the development of traditional forms of BI, such as data marts, dashboards or scorecards.  In some cases, these centers have become focused on providing advisory services to business units that create and manage their own solutions.
• Analytic Competency Centers  Business analytics often begins within business areas such as marketing or risk management.  Analytic competency centers are developed to help other areas of the business leverage information in a similar manner. Whether part of the BI competency center or distinct from it, this is also a core BI function.
• IT  At a minimum, IT has some responsibility for the technical infrastructure on top of which information systems are built -- networks, computers and the services that keep them up and running. IT may also have responsibility for some of the business applications and data management solutions.

Regardless of how your organization structure divvies up these responsibilities, BI is the sum total of these activities, and not the domain of a particular group or department. A business strategy to create value through information cuts across many departments.  It cannot be planned or executed in isolation.

The Future of BI

We're not far from an age where BI is not a separate part of our information architecture.  We're not there yet, but several trends have us on this path:

• Focus on the future value and re-use of data managed by operational applications
• Commitment to data governance
• Maturation of master data management solutions
• Technological advances in data management and information access

When we finally arrive at a unified information architecture, the definition of BI will still hold. We will be closer to delivering on its promise than ever before.

And, without a doubt, we will have come up with ways of using information to deliver value that have not even be thought of today.