Data warehousing and analytics is one of three main categories of solutions offered by pmOne. See below for more details about our experience and expertise in this area. The more you can relate to the challenges and lessons learned that are described here, the more likely it is that you will benefit from working with pmOne on a solution. Contact us for more information...
The data warehouse was and is, for many years, the center of many business intelligence projects and has been touted as the foundation for enterprise-wide business intelligence solutions. The data warehouse should serve as a "single point of truth" where important company knowledge is stored. All employees should be have consistent access to the information they need in the data warehouse.
Data warehousing projects often are initiated by the finance or controlling area. These areas often have a good overview and clear understanding of the requirements, which can usually be quickly incorporated into a data warehouse. Other areas with a different point of view, such as sales or marketing, may have more challenges with the integration of their information with the data warehouse.
Although much time and effort goes into harmonizing business processes and structure to ensure that the company's information is properly maintained in the data warehouse, these efforts often fail. Common causes include the following scenarios: business processes are too unstructured or inconsistent, departmental users are too undisciplined, too little time and attention to detail in the data warehouse design, etc.. In short, human factors can sabotage the idea - or rather the utopia - of the "single point of truth".
In retrospect, unsuccessful attempts have been made to change the way that people in the organization worked, enforcing structure and discipline as part of the "data warehouse ecosystem". Then the financial crisis of 2008/2009 led to the realization that a well structured and integrated data warehouse promotes rapid adoption and compliance with many new and mission-critical requirements. Without this important step, implementing a business intelligence application is far more cumbersome. The turmoil in the financial crisis and the economic reality of the situation demanded a faster response. The moral of this story is that technology has to align with the needs and behavior of people. Attempting to adapt the behavior of people to fit technology with a data warehouse solution was not practical.
Departments and IT are often disappointed with the benefits of a data warehouse. This can be seen very clearly in the results of a 2011 survey conducted by the independent BARC Institute on data warehousing in Germany, Austria and Switzerland. Based on over 200 responses from decision makers in IT departments and different departments, it is clear that the benefits of data warehouse projects remain below expectations. Below are some of the key results from the survey, which lead to this sobering conclusion:
Nearly all respondents (98 percent) assess the significance of the data warehouse as 'important' or 'critical' for their company.
"Trust in data" by far the most important goal for the implementation of a data warehouse. For 96 percent of the participants, this is "important" (14 percent) or "very important" (82 percent). Achievement of this goal in implementation of a data warehouse is disappointing: only 51 percent of respondents consider the target “trust in data” to be within reach. All other implementation goals had less than 50 percent achievement. There are significant differences between results reported by respondents. For example, users who access fully synchronized data reported 20 percent higher satisfaction than users working with non-synchronized data in the data warehouse. Many respondents see the greatest benefit from having a data warehouse in their own department.
Despite these issues, data warehouse use is very widespread: for 86 percent of respondents, data stored in a data warehouse is used for decision support. This is clearly the dominant approach, with 52 percent of respondents using informal information assets and 26 percent using established independent data marts.
In conclusion, Dr. Carsten Bange, CEO of BARC Institute commented about the results as follows. "The achievement of data warehouse projects can absolutely not be regarded as satisfactory and also the challenges mentioned in terms of data quality, agility, integration of key departments. Inevitably users who are more on likely to have occasional doubts can cause great headaches." In particular, the trust in the data corporate management utilizes can be supported very well by the data warehouse. "Despite the shortcomings and challenges of data warehousing systems, the alternatives, such as informal organization of data or the use of shadow systems is in many aspects far worse," writes Dr. Bange.
The results of the user survey can be requested here.
The Utopia of the "Single-Point-of-Truth"
For centuries, philosophy has taught us that the truth is in the eye of the beholder. Modern quantum physics goes one step further: The eye of the beholder even defines the truth. (http://de.wikipedia.org/wiki/Schr% C3% B6dingers_Katze). Applying this analogy to the data warehouse should not be attempted. However, a conclusion can be made: A "company-wide Single Point of Truth" in reality can not be both practical and theoretical.
However, implementing a data warehouse is not at all pointless. For many user requirements, especially in finance and controlling, a data warehouse is still the right solution. An understanding that all the requirements for a business intelligence solution can be covered in the underlying data warehouse is very important. A data warehouse is part of a larger "Business Intelligence Information Architecture". Along with this realization it is particularly important to take human factors into account such as user behavior.
Evolution and Complexity
There is often a sense that a data warehouse design is "on the drawing board." Requirements were collected and brought together (whiteboarding) and then implemented. Then they may change. At the end the hope is that the result is as expected.
Such a procedure can rely heavily on pre-defined processes where there is pre-defined structure such as is commonly found in the finance or controlling department. If the processes are proprietary or less transparent, as is the case for example in sales and marketing, such a procedure model often fails. One simple reason for this is that as soon as systems and processes have reached a certain level of complexity, it becomes more difficult to determine pre-defined requirements. Such factors can be complicated by technologies that can use large volumes of structured data (numbers) and unstructured data (text). Such technologies have recently been given the title "Big Data". When talking about Big Data, defining all known concepts in advance has no chance from the start.
The solution to this problem is learning from the evolutionary approach in nature. Instead of trying to define everything upfront, more complex requirements for business intelligence solutions can be handled by extending and validating as the solution evolves. There are good examples from the recent history of technology of how iterative and evolutionary approaches have proved more successful than working based on initial assumptions.
A few years ago a good photograph was planned and designed. The place was well chosen, the exposure was checked and if the right film is loaded, etc. then the photo was taken. After that the film was developed and then a few days later, the result was reviewed and verified to see if it satisfied the initial requirements. Those who invested no time in being a "photography expert" were often disappointed by the resulting photo.
With digital photography, one check the results immediately. Precise planning is no longer mandatory. If the photographer does not like the photo, it is easy to delete it and repeat the process until the picture is as expected. The technology allows for fast repetition of this "trial and error" or iterative procedure that does not involve significant additional expense or delay to take several photos. Good knowledge of photographic techniques certainly helps to achieve the perfect picture, but it's not necessary anymore. Now a photographic layman can create fantastic photos purely by trial and error.
For building a data warehouse, there are also advantages of using an evolutionary, iterative approach. Since there are also applications where a traditional approach to can still work it is important to consider the unique requirements for each solution. Taking a holistic view, it becomes clear that data warehouse technologies and processes need technology to support evolutionary and iterative approaches. The technology for this is now available.
The term and concept of "in-memory" is currently widely discussed. It is no secret that the new in-memory database technology at a very high speed can be reached at the database query. The hardware for in-memory technology has become affordable. In this respect nothing should stop the "in-memory"-revolution. There should be a very important factor in mind: simplicity of use. So at first no touch-enabled user interfaces are meant. It is worth considering what is currently by far the most widely used business intelligence solution in the world, and why it is used as widely.
The clear winner in the category "most widely used business intelligence solution in the world" is Microsoft Excel. Through its easy-to-follow operation of the standard for business intelligence. Even the numerous attempts by the various business intelligence vendors in recent years to prevent the triumph of Excel in companies as a tool for business intelligence. It is by far the most popular feature in any business intelligence tool, the "Export to Excel" button.
Obviously Excel offers especially in the creation and expansion of business intelligence solutions functions to users in other tools not readily accessible. Of course there are also some risks with regard to native Excel Security, auditability, and versioning.
The instinctive reaction of producers of business intelligence tools was to replace Excel as a whole. After this did not work in the past 15 years, comes slowly to realize that Excel is obviously essential for users. It makes more sense, Excel to modify the schedule in the elimination of the existing limitations and risks when working with Excel. Here again meanwhile the technological means are exist.
It is possible to combine the "world standard" of the data warehouse with the "iterative world" of Excel. It is even possible to develop a business intelligence solution in its beginnings as a small to accompany Excel solution to standardized, enterprise-wide and integrated into the data warehouse solutions with a single technical and semantic business intelligence framework.
Skills of change and agility of a business intelligence solution is critical to their success. Because the demands on any business intelligence solution are constantly changing. Modification is part of the system operation. This reality is to be found but in many data warehouse architectures, not again. The leaders will discuss only on static architectures and architectures always target the "perfect" final states. An Integrated Business Intelligence Information Architecture must also include the dimension of time. This means that the business intelligence information architecture solutions support during their entire life cycle. Especially when users can create solutions and to iteratively, which is an extremely important factor for success. As a basic rule: When there is a solution that provides end-users information, it must be for them in a holistic "BI Information Architecture" type place. Not when it was integrated into a data warehouse.
Consideration of changing requirements of users over time is a very important part of a business intelligence information architecture. In everyday corporate life arise countless questions whose answers require a work-up of data. Often is not foreseeable beforehand whether a quickly prepared solution is used only once. So if a question has been answered, and thus solved the problem or whether this analysis can also provide a wider audience on a regular basis information for decision support. Here again, the principle of evolution: What works and asserts itself, lives on, and the rest is dying.
Would a classic data warehouse try to respond to all queries that appear in a company, at the latest, after several years, a large part of the data warehouse would be irrelevant, but the operation can cause further costs. In reality, the data warehouse team is not nearly able to fulfill all the requirements, so the departments try to solve their problems on their own - if necessary, even secretly.
Today there are the technologies necessary to integrate the "iterative and agile business intelligence world with predefined, standardized world of the data warehouse. The temporal change and life cycle of a business intelligence solution can be from beginning to end in a technological world.
Frequently the IT sector is responsible for the implementation of a data warehouse. Special attention is paid on technical "thinness" and efficiency of the architecture, rather than on the actual end-user requirements. In this context, the data warehouse team has the concern that the end user receives too much space, and causes unpredictable or uncontrollable situations in the data warehouse. Reflexively, the data warehouse is limited functionally. It is the premise of the data warehouse team: "We control the data warehouse and manage what will be happen. We control and make sure that the end users do not break anything."
But this is a fallacy: Without the requested start and expected end user flexibility in many cases their own "solutions": Preferably using Excel and usually without the knowledge of really competent IT department. The resulting losses are lost in the original risk assessment. Applies to the data warehouse team is often the motto: "What I do not know will not hurt me."
A business intelligence information architecture has to fulfill the task, according to the needs of IT control and user demands for flexibility. Here, the following observation: "When a user needs certain data or analysis, then he gets this or created it, no matter what official rules, processes or systems are in place."
This empirical observation must accept those who are responsible for business intelligence as a whole or a data warehouse. It is impossible to prevent this by rigid architectures. Therefore, it should be ensured that users build their solutions in an environment where one can observe the IT least what happens. So IT managers can track so that data can be exported, or which user when accessing the solutions.
By "knowing" what happens, IT can respond much more to problems of the departments and actively support there. Furthermore, actions that could cause widespread security problems are suppressed quickly.
cMORE/Reporting is a solution that is easy to use and meets the requirements of the IT department.