The 7 Pillars of #TrueDataOps

Implementing a new data source can be extremely costly, so it is important to do this once and only once. This may mean loading data even though we don’t know what we will do with it. By focusing on ELT, we take the energy out of the process by loading everything in the format that is available.

ELT also supports the principles of data governance, data auditability, and data lineage because information ingested into the data lake or data warehouse is almost exactly what was extracted with zero, or near-zero changes. Understanding the provenance and reliability of data derived from this ingested data is therefore far easier.

In situations where there are “near zero changes” an ELT model must be an EtLT model; the small “t” reflects the business or regulatory need to reject certain secure or PII data items, or the need to obfuscate or hash data before it is transferred or loaded to any downstream system.

We already do this ALL the time. Consider this database table: Employee ID, Employee Name, Salary, Job Title. When we reload the table a week later, it looks exactly the same, which is standard. But we also may have lost information by overwriting valuable historical data. Even though there was no transformation, we’ve lost the history.

The "Spirit of ELT" takes ‌the concept of ELT further and advocates that we avoid ALL actions which remove data that could be useful or valuable to someone later, including changes in data. This has been a well-understood problem, to which there are well-understood solutions, but they're time-consuming and expensive. To use these solutions everywhere, we need to make the cost of configuration and implementation trivial.

To achieve these objectives, continuous integration relies on using a revision control system and branches to manage the project's source code. All artifacts required to build the project should be placed in this repository. In this practice and in the revision control community, convention dictates that the system should be buildable from a fresh checkout without requiring additional dependencies.

In the new world of data development, all logic and jobs performed on the data as part of an overall pipeline (or pipelines) must be treated as a single code base. The same principles of management and governance are also applied.

While #TrueDataOps also started as a set of best practices, it is evolving into an independent approach to data lifecycle management. DataOps applies to the entire lifecycle - from integrating with model generation, orchestration, and deployment, to health, diagnostics, governance, and business metrics.

Similar to DevOps code development, prototyping begins with a new branch—including data. Importantly, prototyping can take place within the same system with continuous improvements being made until it is ready to go live. The ability to branch will help you escape the traditional waterfall approach, which requires a definitive business goal before development can begin. Instead, you can take a general goal and create a new branch (which includes configuration, code, AND data, i.e., a fully self-contained and complete sandbox). You can then build a first version and can keep iterating and testing until you deliver against the stakeholder’s (sometimes vague!) requirements without compromising data integrity or quality.

DataOps also seeks to streamline data lifecycle management, or even to make it completely invisible. For example, choosing to build on the Snowflake Data Cloud makes data lifecycle management redundant. Not only is data automatically compressed, but it is built on AWS S3, Google Cloud storage, or Azure blob technologies, the cheapest storage options available today. All cloud data platforms will follow this design pattern over the next few years, making environment management seamless and data lifecycle management effectively redundant.

In engineering, maintainability is the ease with which a product can be maintained to minimize the downtime or impact to the end user. There are principles of component or code design that have been developed and refined over many years that enable a code base to be more maintainable and robust for the long term.

Take some of Eric Raymond's rules from The Art of Unix Programming. Though published nearly 20 years ago, it’s still the seminal work in this area, ‌which suggests that maintainability can be improved by:

• Building modular programs
• Writing readable programs
• Using composition
• Writing simple programs
• Writing small programs
• Writing transparent programs
• Writing robust programs
• Building on potential users' expected knowledge
• Avoiding unnecessary output
• Writing programs that fail in a way that's easy to diagnose
• Valuing developer time over machine time
• Writing abstract programs that generate code instead of writing code by hand
• Prototyping software before polishing it
• Writing flexible and open programs
• Making the program and protocols extensible

Whether its 'programs' as its was then, or "components" as it is now, these principles are so valuable.

The #TrueDataOps philosophy favors small, atomic pieces of code that can be reused quickly and easily to reduce overall prototyping and development time. In effect, components.

By preferring configuration over code wherever possible, choosing low-code everywhere else, and reducing code to small, reusable components, it becomes easier to refine, optimize, and replace sections without affecting the end user experience.

Great environment management for #TrueDataOps requires all environments to be built, changed, and (where relevant) destroyed automatically. It also requires something not relevant in the web world: a fast way of making a new environment look just like Production. This could involve the duplication of TB of data.

For example, the Snowflake data platform is one of the only cloud data platforms providing underlying capabilities (such as Zero Copy Cloning) that enable #TrueDataOps environment management facilities today. But many will follow. Cloud data platforms with these unique capabilities, could do for data analytics what Kubernetes has done for microservices.

Done correctly, properly automated Environment Management is a massive step forward for the development of an organization's data platform. It also reduces costs by automating manual steps for creating, destroying, and (worst of all) trying to cross-check environments.

You also need to define your source of truth—the baseline by which data quality and accuracy is measured. When dealing with software, the source of truth is the application’s supporting code repository rather than the application itself.

The same principle exists under #TrueDataOps philosophy. The code repository supporting your code and components is the definitive source of truth for your applications.

Governance is further strengthened with automatically created audit trails. Every change, test and approval is fully documented and stored forever, allowing you to prove how data has and is being used.

In a fast-paced agile development environment, it is impossible to balance optimism (the belief that a new deployment will work first time) with a lack of development process. Automated testing and monitoring provides a way to counter optimism and detect problems caused by optimism.

Organizations like Amazon, Netflix, and Etsy are regarded as world leaders when it comes to rapid testing and deployment. They can successfully deploy new updates incredibly fast (in some cases, every few seconds) with near-perfect availability. This is particularly impressive when you consider they manage millions of lines of code. Automated testing is the key capability that underpins this, and indeed underpins most of the efficiency improvements in software development over the last two decades.

The mathematics for this is simple; if you want to confidently release, you have to test every part of the system (“total” test coverage). As your system gets bigger, this is more work. If you want to release faster, you have to repeat this far more frequently. The only way to solve the equation where the test sets are getting larger and the frequency of execution is getting orders of magnitude faster is full automation.

Automated testing is about trying to stop "bad stuff" getting into production. Automated monitoring is about accepting that, while automated testing is critically important, it can also never be perfect, and eventually something will slip through, and this is where good monitoring is key. The most advanced DevOps organizations today don't see this as a failing, but as a reality that they can control.

#TrueDataOps monitoring involves more than confirming system availability. It also checks the production quality of all data and artifacts in your analytics processes, and tests new code during deployment. By detecting unexpected variations and generating operations statistics, your data team gathers new insights that can be used to further enhance performance of the pipeline and the underlying platform.

It is also important to realize that the definition of data availability has been expanded under #TrueDataOps. As well as being able to run queries, #TrueDataOps defines availability as the ability to return valid data for decision-making.

Even if your platform achieves 99.999% uptime, availability may not be as high as you think if it cannot deliver actionable insights. It is perfectly possible that broken pipelines are compromising your data. But without automated testing, these failures may be undetected and negatively affecting the accuracy of your analytics.

It's like a manufacturing line developing data products, and it's the customer that receives that product at the end of the line. Today all those teams operate independently; coordination is both challenging and fragile, and product quality is unpredictable at best.

#TrueDataOps is designed to coordinate and operate like a highly performing, predictable, quality-orientated, industrial manufacturing line. It expects a heterogeneous tooling environment and is focused on enabling all those teams to collaborate efficiently, and orchestrating the entire lifecycle to deliver the highest quality product to the end customer.

#2: Collaboration and self-service at a customer and stakeholder level.
Most organizations are trying to become more data-driven. They want data insights to inform decision-making. This requires pulling data from multiple different parts of the business into one place. The theory goes that once we have all the data in one place, everyone can use it, answer their own questions, and solve their own use cases by joining multiple datasets together.

Historically, this has been seen as a technological problem, solved with a data warehouse where we can assemble all these different data sets together. This is, of course, part of the solution. However, the reality for most enterprises has been somewhat different.

• It's very hard for typical business users to understand the data that's available and how to use it. The more successful an organization is at bringing data sets together, the worse this problem gets.
• Many datasets have some level of data sensitivity. This has meant that while datasets are loaded into the same data warehouse, their access has been restricted to just the department who originated it (and a few select data gurus). Technology silos have been replaced with privacy silos.
• Ingesting new data has been a long, painful, and expensive process and has therefore needed a strong business case. Where one or two users just wanted a simple spreadsheet included so that they could join/analyze some other data already in the data warehouse, this has rarely been possible or cost-effective.

The ideal state here has always been that, with enough access to data across the whole organization, people will be able to create and discover insights that had never occurred to them in advance.

The reality today is quite different: usually you need to justify how and why you need the data to get access to it, and this destroys the discovery element that is so core to a data-driven business.

To address these issues, and close the gaps to the original data-driven business goals, #TrueDataOps requires:

• A business-user-facing data catalog for users to be able to search and understand the data available and how to use it.
• A way of creating different versions of data with different levels of anonymization, so that datasets with some level of sensitivity can still be safely shared with the rest of the organization, allowing the discovery element.
• Very lightweight and fast ways of getting simpler data in from end users. This doesn't necessarily have to follow all the same controls and processes as the main organizational data—what we might call "Community Data."