25 Not-To-Dos of Data

Starting any journey is tough, and more so when it involves data! Since I decided to join the Blog community I have been thinking actively of what to write about and it suddenly dawned on me that maybe highlighting a few Not-To-Dos is am easy way to start this Blog experiment. After all, it is always easier to tell people what not to do even if what not to do is what you actually do yourself (and know you should not do!). Hope you are feeling me. Anyway, here comes the first Blog post on this site so, hope you agree with the contents. Happy to hear from you to improve this and future articles. Oh, and remember, 25 is an arbitrary number. Somewhat round, not too small and not too big but we could go to 50 or more, as what is definitely not in short supply is bad data habits:-).

Finally, this is meant to be a way for me to get back to heavy writing, find like-minded colleagues and get positive feedback to learn. Each item really scratches the surface. I am hoping to pick a few of these short articles in the next few weeks and months to explore further. Too early to say how but now you know what the plan is!

For now, here is a list of 25 "Not-To-Dos" of data that each of which, one way or another, affects effective Data Management:

1. Neglecting Data Governance

   1. What is it: Failing to implement a framework that outlines how data is managed, who owns it, and how it’s used.

   2. Potential Impact: Poor data quality, lack of accountability, and compliance issues.

   3. Best Practice Fix: Establish a formal data governance framework with clear roles, policies, and ownership for all data assets.

2. Ignoring Data Quality Management

   1. What is it: Not monitoring or enforcing the accuracy, completeness, and reliability of data.

   2. Potential Impact: Decisions based on inaccurate data lead to poor outcomes and financial losses.

   3. Example: JP Morgan’s London Whale incident where poor data quality in risk models led to $6 billion in losses (Google it!).

   4. Best Practice Fix: Implement regular data quality checks and establish clear KPIs around data integrity.

3. Allowing/Encouraging/Tolerating Siloed Data

   1. What is it: Storing data in isolated systems where it cannot be easily accessed or shared.

   2. Potential Impact: Difficulty in gaining cross-departmental insights, leading to fragmented decision-making.

   3. Best Practice Fix: Break down data silos by using centralized platforms or data lakes that integrate all data sources. Remember a centralized solution is not necessarily physical (hope to touch on this one in more details sooner, rather than later).

4. Over-Complicating Data Architecture

   1. What is it: Building overly complex data pipelines and systems that slow down the processing and decision-making.

   2. Potential Impact: Increased operational overhead and slower time to insights.

   3. Best Practice Fix: Design data architecture to be scalable and modular, focusing on simplicity and agility.

5. Relying on Manual Data Processes

   1. What is it: Using manual data handling methods such as data entry and validation.

   2. Potential Impact: Human errors, inefficiencies, and high labor costs.

   3. Example: Just Google for examples of financial services firms who lost millions due to manual data entry errors in transactions.

   4. Best Practice Fix: Automate data workflows and validations to reduce human intervention and error.

6. Lack of Standardization Across Data Sources

   1. What is it: Failing to standardize data formats, structures, and naming conventions across systems.

   2. Potential Impact: Inconsistent reporting, confusion, and difficulty in combining datasets. And maybe I should really say: too many reports for the same topics! I feel many know exactly what I mean here.

   3. Best Practice Fix: Establish data standards and enforce them across all data entry points and sources.

7. Underinvesting in Data Security

   1. What is it: Neglecting to implement proper data security measures such as encryption, access control, and monitoring.

   2. Potential Impact: Data breaches, compliance violations, and loss of customer trust.

   3. Example: Equifax’s 2017 breach exposed 147 million records due to poor security practices.

   4. Best Practice Fix: Implement a multi-layered security approach with encryption, access controls, and real-time monitoring.

8. Not Defining Clear Data Ownership

   1. What is it: Failing to assign responsibility for specific data assets to individuals or departments.

   2. Potential Impact: Lack of accountability, leading to mismanagement or neglect of critical data.

   3. Best Practice Fix: Assign data owners who are responsible for the quality, usage, and lifecycle of their data.

9. Skipping Proper Data Documentation

   1. What is it: Failing to maintain documentation of data sources, transformations, and governance. This is one place that I go on about the dangers of "Agile Methodology". Not that it is bad on its own, but documentation is typically the first thing that suffers when you seek agility in an organization!

   2. Potential Impact: Difficulties in understanding and trusting the data, leading to errors in decision-making.

   3. Best Practice Fix: Maintain comprehensive metadata and data lineage documentation to track the flow of data.

10. Overlooking Data Privacy Regulations

    1. What is it: Ignoring or inadequately adhering to data privacy laws such as GDPR or CCPA.

    2. Potential Impact: Heavy fines, reputational damage, and loss of customer trust.

    3. Example: Google was fined $57 million for GDPR violations due to improper consent practices. An intersting one to Google about, of course:-).

    4. Best Practice Fix: Regularly audit data processes for compliance and update practices according to evolving regulations.

11. Using Outdated or Unsupported Data Tools

    1. What is it: Relying on legacy systems or tools that are no longer maintained or secure.

    2. Potential Impact: System failures, security vulnerabilities, and reduced performance.

    3. Best Practice Fix: Regularly evaluate and update data infrastructure to ensure tools are modern, secure, and supported.

12. Failing to Prioritize Data Integration

    1. What is it: Not connecting data from different systems, leading to fragmentation and incomplete insights.

    2. Potential Impact: Inaccurate reporting, poor customer service, and disjointed operations.

    3. Best Practice Fix: Use ETL (Extract, Transform, Load) processes, middleware, or other relevant methodologies/technologies to integrate data from disparate sources into a single repository.

13. Assuming All Data is Valuable

    1. What is it: Collecting and storing every piece of data without assessing its relevance or usefulness.

    2. Potential Impact: Increased storage costs, complexity in data management, and lower performance.

    3. Best Practice Fix: Focus on collecting high-quality, relevant data and regularly purge unnecessary or outdated data.

14. Not Providing Data Literacy Training

    1. What is it: Failing to train employees on how to interpret and use data effectively.

    2. Potential Impact: Misinterpretation of data, poor decision-making, and underutilization of data tools.

    3. Best Practice Fix: Implement regular data literacy training programs tailored to different levels of expertise within the organization.

15. Lack of Scalability in Data Infrastructure

    1. What is it: Building data systems that cannot grow with increasing data volumes or business needs.

    2. Potential Impact: Performance bottlenecks, downtime, and costly system overhauls.

    3. Best Practice Fix: Design data systems to scale dynamically by using cloud-native architectures and horizontal scaling techniques.

16. Not Establishing Clear Data Use Policies

    1. What is it: Lacking formal policies on how data should be accessed, used, and shared within the organization.

    2. Potential Impact: Data misuse, security breaches, or legal violations.

    3. Best Practice Fix: Create comprehensive data use policies that define access levels, usage rules, and protocols for handling sensitive data.

17. Rushing to Implement AI Without Clean Data

    1. What is it: Deploying AI models without ensuring the underlying data is accurate, consistent, and complete.

    2. Potential Impact: Poor AI model performance and inaccurate predictions.

    3. Example: IBM Watson for Oncology made incorrect treatment recommendations due to poor training data.

    4. Best Practice Fix: Focus on data cleansing and quality assurance before feeding data into AI models.

18. Failing to Align Data Strategy with Business Objectives

    1. What is it: Implementing data initiatives without considering their alignment with the organization’s overall goals.

    2. Potential Impact: Underutilized systems, wasted resources, and missed business opportunities.

    3. Best Practice Fix: Develop a data strategy that directly supports key business objectives and regularly review its effectiveness.

19. Not Archiving Old or Unused Data

    1. What is it: Retaining data indefinitely without determining its relevance or necessity.

    2. Potential Impact: Increased storage costs and legal risks from holding onto unnecessary or sensitive data.

    3. Best Practice Fix: Implement data lifecycle management policies that archive or delete old data based on usage patterns and regulatory requirements.

20. Assuming Cloud Migration Solves All Data Problems

    1. What is it: Believing that moving data to the cloud will automatically resolve governance, quality, or integration issues.

    2. Potential Impact: Continued data problems, only now in a cloud environment, along with unexpected costs.

    3. Best Practice Fix: Plan cloud migrations carefully, addressing data governance and quality beforehand, and ensure cloud solutions are cost-effective and scalable.

21. Lack of Real-Time Data Access

    1. What is it: Failing to provide access to real-time data, limiting the ability to respond quickly to changes.

    2. Potential Impact: Decision-making based on outdated information, leading to missed opportunities or poor outcomes.

    3. Best Practice Fix: Implement real-time data streaming or event-driven architectures to provide up-to-date insights.

22. Relying Too Much on One Vendor

    1. What is it: Becoming overly dependent on a single vendor for critical data infrastructure or tools.

    2. Potential Impact: Vendor lock-in, pricing power shifts, and increased risk during outages or failures.

    3. Best Practice Fix: Diversify vendors and maintain vendor-neutral solutions where possible, with contingency plans in place.

23. Over-Reliance on Shadow IT for Data Management

    1. What is it: Allowing departments to create and manage their own data solutions outside the oversight of the central IT team.

    2. Potential Impact: Uncontrolled data sprawl, security risks, and lack of governance.

    3. Best Practice Fix: Integrate shadow IT initiatives into the broader data strategy, providing IT governance while allowing flexibility.

24. Not Having a Disaster Recovery Plan for Data

    1. What is it: Failing to create a backup or disaster recovery strategy for critical data systems.

    2. Potential Impact: Data loss, operational disruptions, and severe financial consequences.

    3. Example: In 2011, a Japanese automotive company lost critical data due to a natural disaster, as they had no offsite backup.

    4. Best Practice Fix: Establish a disaster recovery plan with regular backups (offsite or cloud-based) and conduct recovery drills to ensure business continuity.

25. Failing to Automate Data Management

    1. What is it: Relying on manual processes for managing and processing data when automation solutions are available.

    2. Potential Impact: Higher operational costs, increased human error, and inefficiency in data handling.

    3. Best Practice Fix: Invest in automation tools such as ETL processes, data cataloging, and AI-based monitoring to streamline data workflows and reduce the risk of human error.

Like I said already, we can discuss more than the above 25 items and for sure, we can dissect each one in more details. But these are another activities for another days. See you soon in a different post...