Data Warehousing and Japanese Philosophy of Housekeeping: A Unique Perspective
Back when I was working as a production engineer in India, the company had implemented the concept of “7S-System of House Keeping”. The 7S is an extension of the Japanese philosophy of 5S.
The components of 7S are:
- Seiri - Sort the items
- Seiton - A place for every item and every item in its place
- Seiso - Remove unwanted items and arrange the necessary items in a orderly way
- Seiketsu - Consistently apply the cleanling practice
- Shitsuke - Train so that the system is maintained without explicit instructions
- Shikkari - Stabilize / Standardize
- Shitsukoku - Persistently maintain the system
Now working in the domain of data, I often think about this philosophy and how intertwined it is with the data warehousing. Although the factory warehouse and Data warehouse different purposes, the fundamental actions are the same.
For example, a factory warehouse
- receives inventory items from different providers,
- inspects it for correctness,
- ensures that no components are missing,
- sorts and standardizes them,
- stores them in a specific way and place for serving the end users.
A Data warehouse follows the same logic. It receives incoming data either in batches or in real-time streams, the incoming data is handled in the staging area, for their correctness, missing values and normalization. Once they are compliant with the specification, the facts and dimensions are modelled and stored in appropriate tables in the database for the end users to consume.
Here is a breakdown of how the 7S philosophy directly applies to a modern data warehouse:
While there is an order for the 7S principles, they are not necessarily sequential. That is to say, it does not follow the same order as the data warehousung process. However, the principles are all relevant and can be applied at different stages of the data warehousing process.
Seiketsu - Clean to achieve consistency
The Seiketsu principle emphasizes the importance of cleanliness and organization. In the context of data warehousing, this can be interpreted as the need for data cleansing and data quality management. Just like in a factory warehouse, where dirty or damaged items can cause problems, in a data warehouse, dirty or inaccurate data can cause problems for end users. This could be achieved through the use of data cleansing techniques such as data profiling, data validation, and data transformation, which help to ensure that the data is accurate and consistent. Additionally, it also emphasizes the importance of monitoring and maintaining the quality of the data over time, which helps to ensure that the data remains accurate and useful for end users.
Seiri and Seiton - Orderliness
The Seiri and Seiton principles are not dramatically different from each other. They both emphasizes the importance of organization and order. In the context of data warehousing, this can be interpreted as the need for a well-designed data model and schema. Just like in a factory warehouse, where each item has a specific place, in a data warehouse, each piece of data should have a specific place in the database/ schema or even in table. This could be achieved through the use of dimensional modeling techniques such as star schema or snowflake schema, which help to organize the data in a way that is easy to understand and query. Additionally, it also emphasizes the importance of metadata management, which helps to ensure that the data is properly documented and can be easily understood by end users.
Seiso - Stratification
The Seiso principle also emphasizes the importance of stratification, which is the process of organizing items into different layers or levels. In the context of data warehousing, this can be interpreted as the need for a well-designed data architecture that separates different types of data into different layers. For example, a common architecture for a data warehouse is to have a staging layer, where raw data is stored, a presentation layer, where cleaned and transformed data is stored, and a consumption layer, where data is made available for end users. It can also be compared to the popular medallion architecture, which separates data into bronze, silver, and gold layers.
This separation of layers helps to ensure that the data is properly organized and can be easily accessed by end users. While the Seiri and Seiton principles also emphasize the importance of organization, the Seiso principle specifically emphasizes the importance of stratification, that stratify the process with the operation layers in terms of architecture which is a crucial aspect of data warehousing.
Shitsuke - Training and Team culture
The Shitsuke principle emphasizes the importance of training and team culture. In the context of data warehousing, this can be interpreted as the need for a well-trained team that is familiar with the data warehousing process and the tools and technologies used to build and maintain the data warehouse. This could be achieved through regular training sessions, workshops, and knowledge sharing among team members. Additionally, it also emphasizes the importance of fostering a culture of collaboration and continuous learning within the team, which helps to ensure that the team is able to adapt to changes in technology and business requirements over time.
Shikkari - Standardization
Shikkari is the principle that emphasizes the importance of standardization. In the context of data warehousing, this can be interpreted as the need for a well-defined data warehousing process, strict adherence to defined data contracts. It is important to have a clearly defined schema, data quality rules, Service Level Agreements (SLAs) and data governance policies in place to ensure that the data is accurate and consistent.
Shitsukoku - Build for persistence
This last principle advocates that the system must be built for durability and persistence. In the context of data warehousing, this can be interpreted as the need for a well-designed data warehousing architecture that is scalable, reliable, and maintainable. Notably, the data engineers must build the data warehouses that ensures resillience by anticipating potential issues and implementing appropriate measures to mitigate them. For example, this could include a systematic approach to data backup and recovery, version control for data and code,and proactive technical debt management.
Concluding thoughts
By adopting a datawarehouse design based on the 7S philosopy, data engineers can build, operate and maintain a sustainable, high-quality and well-governed data warehouse that serves the needs of the business and end users effectively. The 7S principles provide a holistic framework for thinking about the various aspects of data warehousing, from data quality and organization to team culture and architecture.