Building Your Single Source of Truth - An Interview with a Data Pipeline Architect

Building Strong Data Analytics Infrastructure for A Single Source of Truth

Data is the key to success when it comes to running your business more efficiently and effectively. However, businesses often face several barriers that prevent them from becoming fully data-driven. Often, the root cause is a lack of solid data infrastructure, one of the first steps to implementing a successful data strategy. 

According to CFO's interviewed by Adaptive Insights, the three biggest mistakes are keeping data siloed (69%), lackluster reactions to sales and operating margins (60%), and inaccurate data for forecasting and planning (40%).

Let's take a look at the top 5 barriers to becoming data-driven to then understand why a lean data pipeline can support your organization's single source of truth for data analytics.

1. Data Silos: Disparate systems make data integration difficult which leads to inconsistent data, inefficiencies, and an incomplete view of the organization's overall data
2. Insufficient Technology: Many organizations rely on outdated technology that lacks the capabilities required for modern data analytics. Legacy systems can be rigid, expensive to maintain, and incompatible with new technologies.
3. Data Quality Issues: Poor data quality can lead to misguided decisions, loss of trust in data, and ultimately, financial losses.
4. Privacy and Security Concerns: Data breaches and privacy violations can result in legal repercussions, financial penalties, and damage to an organization’s reputation.
5. Rapidly Changing Technology Landscape: The technology landscape is constantly evolving, with new tools, platforms, and methodologies emerging at a rapid pace. Keeping up with these changes can be challenging, leading to the potential obsolescence of current systems and skills.

Every business is different and struggles with a unique grouping of these challenges, but at the root of most of these roadblocks is a lack of solid data infrastructure. This is where a lean data pipeline comes in to make your data strategy a reality.

First: Do you have a solid data analytics infrastructure set up to process your data for analysis? 

The Lean Data Pipeline: Your Single Source of Truth for Business Analytics

The Lean Data Pipeline is a flexible, technology-agnostic approach to standardize the business logic embedded in your analytical data ecosystem in three steps:

1. Extract & Collect
2. Conform Data
3. Visualize & Analyze

The Lean Data Pipeline: Thoughts From an Expert

Howard Lombard (ISO & Chief Architect at Arkatechture) is a published data architecture professional and software development manager with 20+ years of demonstrated data architecture experience working within on-premises and hybrid cloud operational and data warehousing environments. I picked his brain on how a lean data pipeline data model lays the foundation for data analytics transformation and serves as the single source of truth for in-depth analysis and predictive analytics.

Here's what he had to say:

We load all the data everyday. Where it makes sense, we'll put that into a time series where customers can choose to keep every single day that's been loaded to do day-over-day trends. But as the data gets older, having daily snapshots isn't as valuable as having month-end snapshots because it's important to look at month-over-month trends and looking at trends for this month this year versus the same period last year. And so the more history you have, the more you can see performance in the same month year-over-year for example, or you can just look at the last six months. We provide easy filtering. So you can make that time series as long or as short as you want."

How is the lean data pipeline different from a regular data warehouse? 

The difference is we put all the raw data in a time series. And so the benefit there is a data warehouse usually contains data that's been curated, standardized and enriched through subject areas. We actually do a time series on all of the raw data. So we load all the data every day. So in a month or in three months, when somebody says, 'hey, I wish I could have these additional 10 data elements in my data warehouse, so that I could see that on my dashboard when we make the change and expose those teams' 10 data elements ' Not only do they see them starting today and going forward, but because we've saved all the historical data in a time series and the raw data, they instantly have the history of those data elements. That's the bigger difference is we put the raw data in a time series so that when you use it later, you have the benefit of having captured the history and you don't see a new data element and only see that with data going forward, you've got the historical data to back populate it." 

Why is it called a "lean" data pipeline? 

We did have one customer say to us. ' I'm not sure I want a lean data pipeline. I have a lot of data.'  The lean doesn't mean it is small, but rather that it is cheap and scalable...So it's very easy to stand it up, it's affordable. It's scalable and very reusable across businesses, in various industries."

Daily Analysis at Your Fingertips

Once you have the lean data pipeline set up and automated, it's ready to connect to your business intelligence (BI) tool of choice like Tableau or Power BI for self-service visualization and analysis. Everyone is working from the same data, and on the same page across departments. Schedule and deliver actionable trend and exception reports with ease. Tackle any future hurdles along your data journey with confidence in your data. 

Discover what ACV Auctions achieved with their single source of truth