Data warehousing can be made more efficient and errors in its processes can be greatly reduced by using Six Sigma methodology. Quality and improvement specialists recognize Six Sigma as the data-driven approach for measuring and eliminating defects and variation in a process. Six Sigma’s focus on processes, statistics and data make it appealing to industries with processes that are easily quantifiable such as engineering, finance and technology.
The Six Sigma methodology was originally adopted by companies like Motorola and General Electric to improve their manufacturing processes, allowing them to produce at Six Sigma’s benchmark – no more than 3.4 defects per million opportunities. However, it quickly became clear that Six Sigma principles apply equally well to areas that are strongly process-driven, including data warehousing.
Data warehousing is concerned with statically storing data on a server and dynamically managing the data so that it can be extracted, transformed, loaded and made available for use. Successful data warehousing is made up of a series of dynamic processes that are well suited to the application of Six Sigma-style analysis and improvement.
A key benefit to applying the Six Sigma process to data warehousing is through refining software during its development stage. During the Six Sigma Analyze Phase, developers can discover the root causes of the process defects through applying tools like sub-process mapping, process map analysis and cause and effect diagrams.
An example of an organization using Lean Six Sigma to evaluate and improve its data warehouse management processes occurred back in 2009 when Bank of America (BOA) used the methodology to track data warehouse load tasks on a time value map. BOA identified that more than 90 percent of its process involved non-value-added processing.
In the Analyze Phase, BOA used another Six Sigma tool, the Cause and Effect/Fishbone diagram, to dig deeper into the problem. This tool allowed the BOA team to identify, explore and graphically display in detail all of the possible causes related to the problem of a high-percentage of non-value added processing time in the data warehousing process.
In conjunction with the Fishbone diagram, BOA applied the Six Sigma technique of the 5 Whys. The 5 Whys Process doesn’t require advanced statistical tools, it only requires stating the problem and then asking “why.” When you have an answer to the first why, ask why again, and keep evaluating the answers by repeatedly asking why up to five times.
The 5 Whys tool helped BOA remove layers of symptoms and discover the root cause of their inefficient data warehousing problem. As a result of using Six Sigma methodology, BOA identified a significant flaw in its relational database management system and targeted a data warehouse appliance solution with the potential to increase data availability by 96 days per year.
As demonstrated by Bank of America, the rigorous, data-driven Six Sigma methodology is ideal to apply to the highly quantitative and process-oriented nature of data warehousing. When Six Sigma tools are used to examine and enhance data warehousing processes, organizations can clearly recognize defects, bring root causes of problems to the surface, identify opportunities for improvement and implement changes that will reduce defects and increase efficiency and productivity.
