“Leaders base decisions on Data, not Intuition.”  

— Avinash Vashistha  

​

​

“Big data” is one of the biggest challenges and an equally big opportunity that enterprises face today. According to Forbes, total digital universe of data has grown 10 times since 2015; there are over 6 billion mobile phone users and still less than 0.5% of all data is ever analyzed and used! The volume, complexity and variety of data available is growing at an astounding pace, which helps explain that last startling statistic: enterprises gain insights from less than 0.5% of this data.  

 

In today’s world, companies need to master the art and science of data transformation and management. Data transformation, is an ETL (Extract/Transform/Load) process, involving converting a raw data source into a sanitized, validated, and ready-to-use format. It can turn data into timely insights that positively impact businesses and provides the much-needed competitive edge. Data should be accessible, consistent, secure, and be trusted by the users, auditors and relevant regulatory authorities.  

 

Data transformation requires a conscientious data strategy that will deliver business value to all its stakeholders. According to Import.io’s paper, “Top 7 Best Practices for Data Transformation” published May 10th, 2018, key best practices evolved from experience across a diverse portfolio of enterprises are: 
 

1. Start with the end in mind – Design the target: Enterprises need to engage business users to understand the business processes that they are trying to analyze, and design the target format, before data transformation can deliver insights. The above “Dimensional Modeling” process, needs to deliver “Dimension Tables”, which provide the “Who, what, where, when, why and how” context for the data. The second element the above process delivers is “Fact Tables”, which store the results of the events being measured, and answer the “How many” questions.  
    

2. Speed date your data with data profiling: Knowing the business process you would like to examine typically leads to the source(s) of data to be turned. To evaluate market patterns, for example, you would need to enter the customer database, the inventory database, and then extract sales reports from a point-of-sale network. Once the root of the data is known, the raw data can then be converted to a usable format.  

3. Cleanse – When your data needs a bath: Equipped with data modeling tools, you can better understand how much and what kind of data engineering work you need to do on the data in order to make it useable. For example, if the date fields of the source data are in the format YYYY / MM / DD, and the target date fields are in the format MM-DD-YYYYY, you will have to convert the source date fields to suit the target format.
    

4. Conform data to the target format: The above three phases are the inceptions for data transformation into target format. Here the understanding of the source data from the data management department satisfies the need for data attributes for the consumers to evaluate a business process. Starting with mapping source columns to target columns, the data transformation team makes use of ETL tools to automate the data flow on successive data loads for those columns.  
    

5. Build dimensions then facts: Dimensions place meaning around the data; the details explain what happened within the dimensional context. For example, customers, products, and dates could be dimensions; sales results and measures could be facts. Secondly, the benefit of loading dimensional tables is that, freshly loaded fact records can then connect to relevant dimensional records. Sales data would not be very helpful if there were no ties to measurements of consumer, commodity, and time. Therefore, consumer, product and date dimensions must be updated with each data load first, preceded by the selling information sheet.  
    

6. Record audit and data quality events: During the cycle of data transition, audit tracking and data quality measures have huge benefits. Audit tracking measures the number of records enabled at each step of the process of transition and the period such changes took place. Capturing data quality test results, including in the data load audit documents, and connecting fact records to audit records provide the opportunity to trace the history of fact data and help to prove the validity of measurements measured from fact data. This approach allows analysts to "work backward" to react to specific stakeholder questions such as "Where did this data come from?" and “How do I know that those metrics are correct?”. Getting ready and accurate answers to these questions creates customer trust in the transformed data and offers solid ground for continued interaction with end users with the data transformation team.
    

7. Continually engage the user community: The true indicator of the importance of data transformation is to what degree the intended user community embraces the converted digital commodity and consistently utilizes it. But rendering freshly minted, conformed data available to end-users isn't the end of your data transformation; it's just the end of the process. Transformed data must be thoroughly tested for customer approval, and the data management team must quickly address faults discovered by business users "in the wild." The abundance of data today is a potential gold mine for businesses. And, like gold, to optimize its worth, such data must be extracted analyzed, refined and delivered to maximize its value. Understanding the fundamentals of data transformation, such as dimensional modeling, profiling cleaning conforming, testing and presentation, can allow you  to discover valuable insights from your data that can affect your company greatly.  

 

“Data Intimacy” Culture 

 

Data-centric leading companies need to maintain a tradition of "information trust" within their organizations. This requires a change in mindsets, attitudes and behaviors. In a Data Culture, people ask hard questions and challenge ideas. Through behavior, the leaders encourage judgments based on data, not instincts. Which needs leadership to create a "Digital Intimacy." According to Tableau, organization's five main tenants are – Trust, Commitment, Knowledge, Communication and Mindset.  

 

1. Trust: Leaders create a foundation of trust in people and data: Trust is the foundation of a strong Data Culture. Leaders are trusting their people; people are trusting the data and trust each other. Right model of data governance creates a single source of truth that breaks down silos around departments to build high-confidence, collaborative ties. The company provides data insights to discover impactful approaches.
    

2. Commitment: People treat data as a strategic asset: Organizations with effective Data Cultures, devote themselves entirely to understanding the importance of their information assets—and help people make better decisions through data. This dedication is apparent in all facets of the organization—from the framework of the company to daily operations.  
    

3. Talent: Organizations prioritize data skills in recruiting, developing, and retaining talent: Ultimately, Data Culture is made up of data people. Even with the best technology, system and processes, they can't be powered by data, if people don't understand how to deal with the data. As part of the talent strategy, managers must emphasize data skills for recruitment and training—clearly outlined in job descriptions and defined in the hiring process. Everyone in the company must feel confident that they will find the right  results, apply analytical methods to their job and communicate their findings.  
    

4. Sharing: Most data-solving challenges aren't limited to a single unit or business line. We need data from multiple platforms and various teams working together. People have a common interest in a data culture— using data to improve the enterprise. Together, people amplify the impact that data can have. The cooperative attitude builds "stewardship" around the data and analytics, and cultures.  
    

5. Mindset: The development of a data-first mentality is equally important as the development of data skills. In a Data Culture, data is given priority over intuition, anecdotes or rank. Data mindset exchanged by all, creates open discussions and ideas that contribute to exploration and innovation. In such an environment, data is seen as a source of personal growth and career development. People are curious and willing to challenge evidence with their own hypotheses— and are open to challenge from others. When data-driven practices become habits perceptions change and people begin to equate data with performance and development.  

 

__________________________________________________ 

A successful data migration platform - example, Advanced Data Migration (ADM) from Syniti, solves complex enterprise data transformation challenges with all eight stages of a successful migration built-in. Leading migration platform and solutions combine Intelligent Automation and Machine Learning with expert insights gained from thousands of data migrations. They provide project oversight, visibility; knowledge preservation for future migrations; speed and agility in implementation. A successful data migration engagement augmented by Intelligent Automation, should significantly automate the process >80%, save clients up to 50%, increase quality and provide a ROI in less than 9 months.  

__________________________________________________