Most data ends up in Business Intelligence (BI) reports. That’s no surprise: BI transforms raw data into actionable insights, helping businesses to make informed decisions, spot trends, and drive strategy with confidence.
But when it comes to structuring data for BI, one big question arises: Should data transformations happen within the BI tool, or should you create a flat table in the database?
This debate exists for good reason. Both approaches have their strengths, and neither is universally better. Flat tables often win points for being simple and fast to build. On the flip side, they can be inflexible and difficult to maintain at scale. Meanwhile, semantic models are praised for their scalability and reusability but tend to be harder to set up and govern consistently across teams.
The division often mirrors team roles: data analysts and engineers prefer flat tables for control and simplicity, while BI developers lean towards semantic models for flexibility and user-friendliness in tools like Power BI or Looker.
In this article, I’ll weigh the pros and cons of both approaches and share my perspective on the best way forward. Let’s dive in.
The flat table approach consolidates all your necessary data into a single, denormalized table before it reaches the BI tool. That means one table with all the columns - facts and attributes - already joined and ready for use. The simplest visualization of this structure is a table inside one Excel sheet. This method has several business-related advantages:
But, there are some downsides to this approach:
One of the most common concerns with flat tables is performance and storage. Since they contain a lot of redundant data by design, they can become large and occasionally slower to query, especially when dealing with massive datasets or frequent refresh cycles. This is a real trade-off, but whether it actually becomes a problem depends on your system architecture and performance expectations. Flat tables may struggle in high-volume environments, when powering many dashboards, or when real-time performance is critical - particularly if you’re using databases that aren’t optimized for analytical workloads.
However, many of these limitations can be addressed with proper setup. Columnar databases like Redshift, BigQuery, or Snowflake only scan the columns needed, significantly reducing overhead. Loading only the required data into dashboards, applying smart distribution and sort keys, or using incremental refreshes can also improve performance. Most importantly, optimization should be driven by clear performance KPIs - not assumptions. If a dashboard loads within your defined threshold and users are satisfied, chasing “faster” just for the sake of it doesn’t bring real value.
Without normalization, data duplication is inevitable, increasing storage costs and complicating updates.
Maintaining a flat table becomes inefficient as datasets grow, leading to costly optimizations.
A semantic model structures data into fact and dimension tables, optimizing storage and improving efficiency. Instead of consolidating all data into one table, facts (metrics like sales, revenue, or orders) are kept separate from dimensions (descriptive attributes like customers, products, or time periods), resulting in:
But, of course, there are also some trade-offs:
As always, it depends on the specific use case.
Flat tables will always be better when you care about version control, easier testing, and less dependency on specific BI tools. Semantic models will win when you rely heavily on your BI tool’s built-in modeling features.
But if I had to choose, I’d lean towards the OBT approach due to its consistency and business advantages. One of the most common issues I see is inconsistent KPIs across reports because too many transformations are done within the BI tool.
That said, I’m a fan of a hybrid approach. Combining both methodologies could be the best solution:
This approach leverages the best of both worlds - ensuring data consistency while maintaining flexibility and efficiency. It also allows businesses to scale efficiently while minimizing technical debt.
According to KPMG’s 2024 Survey of Sustainability Reporting, 80% of the 5,800 surveyed companies have established carbon reduction targets. This number highlights the increasing focus on sustainability but also the critical need for accurate emissions tracking. While setting targets is the first step, high-quality, reliable data is essential for consistent and accurate reporting across all areas of a business, particularly in supply chains.
The Greenhouse Gas (GHG) Protocol categorizes emissions into three scopes, each representing different sources of emissions within a company’s operations:
While Scope 1 and 2 emissions are relatively easier to track, Scope 3 emissions are more complex and often represent the largest share of a company’s total carbon footprint. For many manufacturing companies, the biggest portion of Scope 3 emissions comes from supply chain operations, which will be the focus of this article. We’ll explore the challenges of Scope 3 reporting and how improved data can support compliance and better decision-making.
Companies track their carbon footprint for various reasons, including environmental responsibility, corporate social responsibility (CSR), and good brand perception. Additionally, regulatory requirements play a key role in driving sustainability reporting. Accurate emissions data is crucial to maintaining compliance and meeting corporate sustainability goals.
Regulations surrounding emissions reporting are becoming increasingly strict globally. In the European Union, the Corporate Sustainability Reporting Directive (CSRD) requires large companies to disclose detailed environmental, social, and governance (ESG) data, including greenhouse gas emissions. This mandate took effect in 2024 and requires reports published in 2025 to include this data. As a result, nearly 50,000 EU companies now need to report their Scope 3 emissions.
Similar regulations exist in the UK (SECR) and in over 40 other countries, including the US, Canada, and Japan (alongside EU countries), all of which have implemented or are planning to implement corporate emissions disclosure requirements for greenhouse gases (GHG).
Effective Scope 3 emissions reporting comes with a number of obstacles, including data complexity, the frequent need for updates, and ensuring accuracy across a diverse range of
sources. Some of the key challenges are:
One of the biggest challenges in Scope 3 emissions reporting is poor data quality, which can significantly impact the accuracy of sustainability metrics. Companies often deal with incomplete, inconsistent, or outdated data from multiple sources.
Examples of Poor Data Quality Issues include:
The poor data quality, complexity, and diversity of data required for Scope 3 emissions reporting make it challenging to get a full and accurate picture. To track emissions properly, companies need to collect a wide range of information, including:
Many sources of data make data foundations extremely difficult to establish. Not all the data comes from the organization’s internal sources but also requires collaboration with other parties (e.g., vendors).
Another major challenge arises from geographical data. Vendor and Plant locations change frequently, leading to potential errors in emissions calculations. For example, if a vendor relocates its plant, this change must be automatically reflected in route calculations to maintain accurate CO₂ estimates.
Solid data foundations aren’t just about collecting the right inputs - they also require clearly defined, well-structured metrics. The way data is organized should support easy and consistent metric implementation later on. Without this, even the best-quality data can become difficult to interpret or apply reliably across reports, especially when calculating Scope 3 emissions that rely on complex, multi-source inputs.
A common approach in logistics is estimating emissions using a simple emissions factor model based on CO₂ emitted per kilometer traveled for different transport modes. However, material-related emissions are far more complex. This requires gathering and applying carbon factors from multiple sources, such as:
These carbon factors need to be regularly updated, mapped to specific materials, and often collected manually, making the process cumbersome and prone to errors.
Only a few specialists can bridge the gap between sustainability regulations and data analytics. Applying the correct emission factors (particularly for specific materials) requires deep domain knowledge and industry-specific expertise. This skills gap makes it difficult for many organizations to ensure accurate emissions reporting. To accurately report emissions, professionals need a blend of sustainability knowledge and data expertise. The challenge is that few people have both skill sets - they either specialize in environmental science or in data analytics, but not both.
The challenges around Scope 3 emissions reporting show just how complicated it can get ‒ different sources, frequent updates, and a lot of moving parts. Without a reliable data foundation, companies risk inaccurate reporting, which can lead to poor decisions and compliance issues.
This is where a structured system like a sustainability mart can help.
When it comes to structuring such a mart, there are two common approaches:
Personally, I would opt for a flat table approach rather than creating a semantic model inside a BI tool. I’ve explained this choice further in another article, but with sustainability data, the choice between these two approaches is even clearer. Master data plays a crucial role in metric calculations, which makes it almost impossible to keep it outside the mart. A sustainability mart integrates critical data sources into one system, enabling accurate calculations and reporting of ESG metrics. Such a mart can include a variety of relevant components, for example:
With these data points in one integrated system, companies can accurately calculate material and logistics-related emissions. By leveraging a sustainability mart, companies can:
Accurate sustainability reporting starts with clean, structured data. For companies, mistakes in emissions reporting can result in both regulatory risks and growing stakeholder scrutiny. While domain knowledge remains crucial, planning and implementing a data-driven strategy is the key to long-term success. Ultimately, a well-designed sustainability data system can transform reporting from a compliance burden into a strategic asset, providing companies with valuable insights into their operations and sustainability performance.
Master data management (MDM) often sparks debate. It’s a frequently proposed budget item that rarely gets approved, yet it’s a fundamental reason business leaders struggle to achieve granular and unified metrics. This article aims to demystify MDM by showing that it doesn’t always have to involve big budgets or complex tools. Instead, we will define master data and explore two straightforward approaches to MDM that can be implemented without lengthy multi-year technology roadmaps. While the complexity of MDM varies by case, we believe most organizations can achieve significant improvements by following a few practical steps.
Master data management is the process of creating and maintaining a single source of truth for critical business data. It ensures consistency, accuracy, and accessibility across an organization’s operations. By harmonizing data, MDM eliminates discrepancies, supports better reporting, and improves decision-making processes.
The most commonly used dimensions in manufacturing company metrics include:
Many master data management projects fail to achieve their intended goals. In some cases, these projects never even get off the ground because they fail to clearly communicate their value to business leaders. Based on our observations, there are three common pitfalls when companies approach master data:
To avoid common pitfalls in data management, it is essential to break down the project into two distinct approaches: a collaborative approach and a consolidated approach. While this framework is not our own invention, it has been a reliable methodology we have employed successfully for many years.
By dividing master data management work into these two approaches, organizations can tackle tasks more effectively. This structured method allows for a clearer focus on priorities and better resource allocation. Moreover, starting with small, incremental steps—rather than attempting a large, comprehensive project—is often a more practical and achievable way to ensure success in master data management.
The collaborative approach to MDM is ideal for managing lookup data and creating custom groupings. This method relies on an application with a user-friendly interface that enables teams to collaborate effectively on master data management tasks. Key features include:
Advantages
Disadvantages
This type of setup has proven effective for defining dimensions critical to measuring business unit or sales performance. For instance, teams managing sales territories have achieved significant success by aligning their territory dimension reporting with their sales organization structures, ensuring consistency with sales incentive plan reporting.
In contrast, the consolidated approach is tailored for application data that is maintained directly at the source. This method involves aggregating data from multiple applications into a central repository without making any modifications. Its key characteristics include:
Advantages:
Disadvantages:
The consolidated approach is most effective when source data from applications is of high quality and does not require frequent adjustments by users. It is ideal for organizations seeking streamlined operations where data reliability and minimal intervention are priorities.
Considering that both collaborative and consolidated master data approaches have their own unique advantages and disadvantages, a hybrid strategy can often be the most effective. By using these approaches together:
If You Want Real Change in Master Data Quality, You Need Visibility
Once, I was in a meeting with the CEO of a Business Unit discussing the sales funnel. The conversation was not going well because the data was, quite simply, poor. However, the CEO said that he would be willing to continue reviewing the sales funnel reports even if the data was flawed, as long as it was clear which data was problematic and why.
The CEO made a smart move here. Instead of insisting that everything be flawless, he told his leadership team that he was willing to work with imperfect data, provided it was clear why the data was flawed and who was responsible for addressing the issues.
Data quality metrics might seem like a dull report that no business leader would bother opening. However, imagine a business report on key metrics like orders, sales, or inventory that includes a data quality score. Such a report would undoubtedly capture attention. This kind of visibility not only ensures the report gets reviewed but also drives accountability. It motivates people to address data issues, whether in the source system (consolidated master data) or within the application (collaborative master data).
Master data management plays a pivotal role in unifying metrics and insights for informed decision-making. By understanding the collaborative and consolidated approaches, organizations can select the strategy that best aligns with their needs and data quality standards. Whether fostering collaboration for lookup data or consolidating application data for streamlined efficiency, effective MDM can transform how businesses leverage their data assets.
Business leaders are always under constant pressure to improve operational efficiency, maintain product quality, and optimize financial performance. Good data and, more importantly, good metrics are important for effectively addressing these challenges.
A critical yet often underutilized tool for achieving reliable business metrics is the data mart. In this article, we will explore what a data mart is and how it functions. We will also define what is a “good metric” and examine how data marts enable business leaders with metrics that enhance visibility, control, and predictability—ultimately improving business performance.
A data mart is a specialized data repository typically stored in a SQL database designed to serve the specific needs of a particular department, business unit, or function. It contains data relevant to its intended audience, structured to provide streamlined access to insights.
Key characteristics of a data mart include:
By organizing data into easily digestible segments, data marts bridge the gap between raw data and actionable metrics, enabling business leaders to make informed decisions faster.
Not all metrics are created equal. A good metric stands out by its ability to provide meaningful, actionable insights that align with organizational goals. To evaluate the quality of your metrics, consider the following attributes:
To illustrate the impact of missing metric attributes on business performance, let’s consider a case study. Imagine a scenario where a CEO asks the CFO to temporarily reduce travel expenses to meet profitability targets. Now, suppose the CFO has to work with a Travel Expense metric lacking the five key attributes. Here’s what happens:
Metrics that embody these attributes not only enhance decision-making but also build trust among stakeholders by providing a clear, auditable trail from raw data to actionable insights.
Manufacturing companies operate in complex environments where good metrics are critical for visibility, control, and success. Data marts make good metrics possible through five essential activities:
These capabilities make data marts indispensable for driving performance improvements and informed decision-making in manufacturing operations.
Data has become popular in business, placing significant pressure on data marts and the traditional business intelligence profession. Many so-called data professionals promise quick ROI through complex data science, AI, and ML projects or by promoting revolutionary new technologies often portrayed as miraculous solutions.
To avoid costly missteps and wasted resources on data initiatives that fail to deliver, it is important to recognize how the temptation of quick fixes and over-promised results can mislead companies. Below are common pitfalls to steer clear of when seeking the insights and metrics your business needs:
Excessive Focus on Tools and Technology: While focusing on tools and technology is not inherently bad, it becomes problematic when attention shifts to unattainable promises tied to “magical” solutions. Good data and insights come from strong processes and robust stakeholder alignment—essential foundational work. Moreover, tools are increasingly standardized, so as long as the fundamentals are solid, selecting one tool over another usually won’t yield vastly different results (unless you’re Google).
The One ERP Strategy: This strategy, often proposed by IT leaders in manufacturing with extensive ERP experience, is no longer effective. A single ERP system does not guarantee centralized or unified metrics. Non-ERP data, such as CRM data, has become equally critical and must be included. Furthermore, mergers and acquisitions often result in multiple ERP systems, meaning “one ERP” rarely remains as singular as promised.
Putting Complex Data Modeling in the ERP: Often proposed by IT leaders, this strategy is not always a straightforward decision. In some cases, performing data modeling in the ERP makes sense, but in others, it does not. Customizing and updating data marts is now faster and more cost-effective than modifying ERPs. If your organization operates multiple ERPs, it might be more practical to move certain calculations, including key ones such as accounting reserve calculations, to the data marts layer.
Expecting ROI from All Data Initiatives: This expectation often stems from a misunderstanding of the diverse aspects of data work. Not all data projects are about driving revolutionary business changes; many are essential for day-to-day operations. Without solid data marts, business functions may create their own versions of metrics, often leading to bad practices and inefficiencies, and this is also a good way for numbers to never match properly.
As we have seen, avoiding these pitfalls is important for building a reliable data and metrics ecosystem that supports business leaders. Recognizing these challenges allows organizations to focus on sustainable strategies that drive long-term success.
Data marts provide powerful technology and best practices for delivering “good metrics” that are compliant, usable, granular, aligned, and comparable. These metrics not only improve visibility and control but also enhance predictability—key drivers of business performance.
Integrating data marts into your business performance analytics strategy accelerates access to insights, enabling faster decision-making, more efficient operations, and improved business performance.
The growing number of tools and solutions in the data ecosystem can be overwhelming for business leaders. Some data solutions are so advanced that they resemble full-scale “digital transformation” initiatives—a term often used by IT professionals.
However, one area of data analytics has remained largely unchanged for the past 30 years: business intelligence. Companies once relied on robust data warehouses with well-defined structures paired with straightforward reporting tools that efficiently extracted insights.
Over time, data warehouses began to fade from prominence. This decline was driven by:
But data warehousing played a critical role in organizations by supporting unified and standardized metrics. Today, this function is largely neglected, leaving businesses to grapple with its absence. As a result, leaders are increasingly witnessing the impact on performance, compliance, and control, compounded by challenges in effective monitoring.
Data warehouses became the definition of slow and rigid IT. Change requests often took months—or even years—to implement, as BI teams resisted any changes that might disrupt production systems. This inflexibility drove business functions to develop their own localized data warehouses, typically ignoring best practices, total cost of ownership (TCO), and accumulating significant technical debt.
As data warehouses were centralized within IT departments, they lost connection to business domains. IT teams, by nature, lacked deep business understanding, leading to a disconnect. IT expected well-defined, scoped requirements to execute in a waterfall approach, while business professionals anticipated proactive insights and solutions from the “data people.”
As IT departments took control of data modeling in data warehouses, business professionals began questioning the accuracy and reliability of the metrics they consumed. Compliance with policies and metric calculations became a significant concern: If I can’t understand how a metric is calculated, how can I trust it?
In the early days of data warehousing, the application landscape was relatively simple, with ERP systems dominating as the core business application. Data warehouses were often designed as extensions of these ERP systems.
Over time, two major shifts occurred. First, mergers and acquisitions introduced multiple ERPs into the ecosystem. Second, new applications—such as CRMs, HRMs, and MES systems—entered the mix, further complicating the landscape. IT departments attempted to address these challenges by pushing for a single ERP implementation across the organization. Meanwhile, business leaders grew frustrated, questioning how many years it would take to answer even basic questions like, “How many employees do we have?”
At first glance, a data mart might seem like a minor evolution from traditional Business Intelligence (BI) run by IT:
A data mart is a subset of a data warehouse, focusing on a specific business area or department. It contains a curated set of data tailored to the needs of that group, making it easier and faster to access insights without navigating the entire data warehouse. This closely mirrors the structure of a typical data warehouse OLAP cube, leading to the impression that introducing data marts is just a rebranding exercise.
However, this view overlooks the fundamental shift brought about by data marts, which are driven by a new set of principles. These principles can be traced back to modern data platform architectures, which themselves evolved from the Big Data movement (remember Hadoop?). The three key principles that distinguish data marts today are:
The challenge today is that these principles are not universally adopted across the industry. Depending on who you speak to, you’ll encounter different perspectives on how data marts are implemented and leveraged.
Technically, a data mart is simply a flat table within a SQL database. This table is usually accessed through reporting tools like Power BI, Tableau, or Cognos. Analysts can connect to the table and create typical BI reports, and as the data is updated, the connected reporting tools automatically reflect these changes.
In a modern data platform, the data mart is developed using SQL code within the modeling layer. This code is typically hosted in a code repository, such as GitHub or Bitbucket. By storing the code in a repository, the data mart becomes transparent, easy to contribute to, and aligns with the three principles outlined earlier.
Now, let’s explore how data marts, with their transparency, flexibility, and version control principles, help address the challenges often encountered in traditional data warehouse systems.
Data marts are faster to develop because they allow contributions from multiple teams. By using SQL models stored in a code repository, teams can collaborate and contribute to the code. This eliminates the bottleneck caused by a single centralized team handling all the work.
With data marts, teams outside of IT can actively contribute to model creation and data mart definition, bridging the gap caused by limited business knowledge. When paired with good governance, this approach brings data modeling closer to the business, enhancing domain understanding.
Since data mart code is stored in verified and recognized repositories, it becomes fully transparent, addressing the issue of “black boxes” that only a few individuals understand. Additionally, implementing data catalogs can further improve transparency by providing metadata and data lineage, making these insights accessible even to less technical users.
As part of modern data platforms, data marts can handle significantly larger datasets. This makes ingesting data from multiple applications cost-effective, allowing integration from various systems into a single, well-defined data mart through proper data modeling.
Data marts represent a modern solution to many of the challenges faced by traditional data warehousing systems. By embracing transparency, flexibility, and version control, data marts empower teams across the organization to collaborate and contribute to data modeling, breaking down silos and reducing bottlenecks.
With their ability to bring data modeling closer to the business, data marts enhance domain understanding and improve compliance through transparency and clear data lineage. Moreover, the cost-effectiveness of integrating multiple data sources in a single data mart aligns with the demands of today’s complex data ecosystems.
Incorporating data marts into your data strategy can help businesses move beyond the limitations of legacy systems, enabling faster decision-making, better insights, and more agile operations.