Blog

AI Access Management: Three Governance Layers

2026-06-25T15:41:00Z

As described in the previous articles in this data governance series, effective enterprise data governance requires combining the useful parts of hub-and-spoke with the useful parts of data mesh domains.

The hub provides shared governance, certified facts and dimensions, secured schemas, access control, best practices, and production standards.

Domains provide business context, domain-specific logic, and ownership close to the business.

AI introduces an additional governance question:

How should we govern the data access of AI agents?

This challenge is more complex than typical BI access management because AI serves as both a reasoning and a consumption layer.

With a dashboard, the access pattern is usually clearer. A user opens a Power BI report, the report connects to a semantic model or database, and the user sees whatever the report is allowed to show.

With AI agents, the flow is less obvious. The user asks a question. The agent may call tools, query databases, inspect metadata, generate SQL, run Python, summarize results, use memory, or send part of the context to an LLM.

This introduces a governance challenge that does not exist in traditional BI environments.

There is no point in building a careful data access model with secured tables, secured schemas, and identity groups if a GenBI agent can then answer questions about any data in the organization.

AI access management must be layered.

There are three different questions:

What the LLM can see;
What the user can see;
What the agent can see.

These are not the same thing.

These layers sit atop the governance model discussed in previous articles. Certified facts, certified metrics, secured schemas, identity groups, and domain ownership remain the foundation. AI introduces an additional execution layer, but it should not replace the existing governance model.

If we confuse them, AI governance becomes either too loose or too restrictive. Too loose, and sensitive data leaks through the agent. Too restrictive, and the AI system becomes useless because it cannot do meaningful work.

Effective AI governance depends on keeping these layers separate.

Agent vs LLM

Before going further, it is important to clarify the difference between the agent and the LLM.

For governance purposes, it is useful to treat them separately.

The LLM is the model endpoint. It receives prompts, reasons over context, and generates responses. In most enterprise setups, this endpoint is provided by an external vendor or by a model deployed in a controlled environment.

The agent is the broader system around the LLM. It may include deterministic code, tools, permissions, memory, database connections, SQL execution, Python execution, logging, observability, and workflow logic.

For example, an agent may:

receive a user question;
check the user’s identity groups;
inspect the catalog;
generate SQL through the LLM;
validate the SQL;
run the SQL inside the company infrastructure;
summarize the result;
and return only the approved output to the user.

In that case, the LLM is only one component of the agent.

This distinction matters because deterministic agent code can be audited and controlled more easily than the LLM itself.

If the agent runs within the company infrastructure, under a controlled harness, it can safely perform tasks that should not be delegated directly to the LLM.

For example, the agent can query secured schemas, apply permission checks, execute SQL, enforce row limits, filter columns, log access, and reject unsafe requests.

The LLM should not be treated as the place where all governance happens. It should be treated as a reasoning component inside a governed system.

The Harness and AI Supervisor

Harness engineering is becoming increasingly important.

A harness is the controlled environment in which AI agents run. It is the sandbox, container, or execution layer that surrounds the LLM, making the agent safe enough to use in enterprise settings.

Within the harness sits an AI Supervisor.

The supervisor acts as the orchestration layer. When a user submits a request, the supervisor evaluates the user’s intent and determines which AI agent is best suited to handle the task.

For example:

a BI agent may answer analytical questions;
a SQL agent may generate and validate queries;
a catalog agent may search metadata and lineage;
a forecasting agent may perform predictive analysis.

The supervisor routes the request to the appropriate agent rather than allowing every agent to respond to every question.

Once selected, the AI agent does not operate independently. Instead, it relies on services provided by the harness. The agent can call the harness for:

permission checks;
authentication and authorization;
memory retrieval;
tool execution;
logging and audit trails;
policy validation;
SQL execution;
result filtering;
and other governance controls.

This separation is important. The supervisor decides which agent should run. The agent performs the task. The harness provides the governed services that the agent depends on.

The harness should be primarily deterministic code.

This is important for two reasons.

First, deterministic code can be audited. You can inspect it, test it, version it, and explain how it behaves.

Second, deterministic code gives you a way to control the more unpredictable behavior of LLMs.

An LLM may misunderstand a request, generate incorrect SQL, or attempt to answer with incomplete context. The harness should prevent these mistakes from becoming data governance issues.

A good harness typically includes:

an AI supervisor;
permission systems;
authentication and authorization;
tool execution controls;
memory systems;
logging and audit trails;
SQL validation;
result filtering;
row and column limits;
policy checks;
and escalation rules.

For example, if a user asks:

Show me customer profitability by region.

The supervisor first determines which agent should handle the request. The agent should then verify:

who the user is;
which identity groups the user belongs to;
whether the user can access profitability data;
which secured schemas the user can query;
whether customer-level data is allowed;
whether margin or cost fields are restricted;
and whether the result should be aggregated.

Only then should the query be executed.

The harness is where AI governance becomes operational.

Layer 1: What the LLM Can See

The first layer is what the LLM can see.

As a rule, do not send company hub data directly to the LLM unless there is a strong reason and the environment is trusted for that level of data.

This is especially important for sensitive facts, customer data, employee data, margin data, cost data, legal data, supplier data, and anything that could create commercial or regulatory exposure.

There are two reasons for this.

The first reason is security and confidentiality. Even when vendors provide enterprise protections, the safest pattern is still to minimize what is sent outside the controlled environment. The LLM should receive only the context it needs to perform the task.

The second reason is cost and performance. Data analytics is data-heavy. Sending large datasets into the LLM context is usually expensive, slow, and unnecessary. It can also reduce answer quality because the context window gets filled with raw data rather than useful instructions, schema information, or query results.

LLMs are usually better used for:

generating SQL;
explaining SQL;
creating Python code;
interpreting metadata;
proposing joins;
summarizing small approved result sets;
explaining errors;
and helping fix failed queries.

For example, instead of sending one million invoice rows to the LLM, the agent can send:

the user question;
the approved schema names;
the relevant table metadata;
the available columns;
the metric definitions;
and the permission context.

The LLM can generate a SQL query.

The harness can validate and execute the query inside the company environment.

Then the LLM can receive only a small approved result set or summary.

This is a much safer pattern.

The practical rule is:

The LLM should reason over metadata, instructions, and approved summaries, not raw enterprise data by default.

Layer 2: What the User Can See

The second layer is what the user can see.

This should be kept as simple as possible.

The best pattern is to use the same identity groups that already govern access to secured schemas.

For example, a user may belong to:

north_america_sales
global_finance
powerbi_sales_analysts
executive_reporting_builders

The AI agent should use these groups to determine what data the user can access.

If the user belongs to north_america_sales, the agent may allow access to:

hub_facts_nam
hub_dimensions_customer_sales
hub_dimensions_shared

If the user is not in a finance group, the agent should not answer questions that require cost or margin fields.

For example, if a Sales user asks:

What is the margin by customer for North America?

The agent should not simply answer because the question is analytically valid.

The agent should check whether the user is allowed to see margin data. If not, it should refuse or redirect to an approved metric such as revenue or discount.

The important point is that the AI agent should not create a parallel access model.

It should reuse the enterprise access model.

This requires a permission system inside the AI harness.

Ideally, this permission system is global. All agents should call the same permission service rather than each agent implementing its own rules.

This makes governance simpler.

It also makes auditability easier because the organization can answer questions such as:

Which users can access which secured schemas through AI?
Which identity groups can query margin-related data?
Which agents can access customer-level data?
Which requests were denied?
Which users asked for restricted data?

The practical rule is:

The user should only be able to access data through the agent, as they can through approved enterprise access paths.

AI should not become a shortcut around the data access model.

Layer 3: What the Agent Can See

The third layer is what the agent can see.

This is different from what the user can see.

An agent may need access to more context than the user sees in the final answer. For example, the agent may need metadata, lineage, table descriptions, column definitions, sample values, metric definitions, query logs, or error messages to perform its work.

But this does not mean the agent should have unlimited access to all data.

An agent is a combination of LLM calls, tools, deterministic code, permissions, memory, and execution logic. In some frameworks, an agent may appear to be a skill. In others, such as Python-based systems, the agent may be a program running inside the company’s infrastructure.

If the agent runs inside the harness, it is safer than sending everything directly to the LLM. But it still needs access boundaries.

For example, a SQL-generating agent may need to inspect:

table names;
column names;
relationships;
metric definitions;
allowed schemas;
and previous query errors.

But it does not necessarily need unrestricted access to every row in every table.

The agent should receive the minimum access required to complete the task.

This is especially important because agents can accidentally surface internal information to users.

For example, suppose the agent has access to global financial data, enabling it to answer questions for Finance users. If the same agent is used by Sales users, the harness must ensure that the agent does not leak finance-only information into a Sales answer.

This can happen in subtle ways.

The agent might:

generate SQL against the wrong schema;
join to a restricted table;
use a restricted metric in an explanation;
summarize data from memory that the user cannot access;
or expose a column name that reveals sensitive logic.

So the agent’s access should be controlled at multiple levels:

tool access;
schema access;
metadata access;
memory access;
execution access;
and response filtering.

The practical rule is:

The agent may need more operational context than the user, but it should never be allowed to return data the user is not authorized to see.

Example: Sales User Asking About Margin

Consider a Sales user asking an AI agent:

Which customers have the lowest margin in North America?

This question touches all three layers.

At the user layer, the system checks whether the user belongs to a group that can see margin data. If the user belongs only to north_america_sales, they may be allowed to view revenue but not costs or margins.

At the agent layer, the agent may know that margin metrics exist and that Finance users can access them. But the agent must not use those metrics for this user.

At the LLM layer, the LLM should not receive raw margin data. It may receive a policy-aware instruction such as:

The user does not have access to margin metrics. Do not generate SQL using cost or margin columns. Offer an approved alternative such as revenue, discount, or sales volume.

A good response might be:

I cannot analyze customer margin with your current access. I can help you analyze revenue, discount, order volume, or sales performance by customer for North America.

This is the right behavior.

The agent is useful, but it respects the existing access model.

Example: Finance User Asking About Margin

Now consider a Finance user asking the same question.

The user belongs to global_finance.

The harness checks that this group has access to:

hub_facts_finance
hub_dimensions_finance
hub_dimensions_customer_finance
hub_dimensions_shared

The agent can then generate SQL against the approved finance schemas.

The LLM may generate the query, but the harness validates it before execution. The query runs inside the company infrastructure. The result is aggregated, filtered, and returned to the user.

The LLM may receive only the final approved result set for summarization.

This is the clean pattern:

User access is checked through identity groups;
Agent execution is controlled by the harness;
LLM exposure is minimized;
Results are returned only if the user is authorized.

The Clean AI Governance Model

The cleanest approach is to place AI inside the existing governance architecture, not beside it.

The hub still owns certified facts, dimensions, secured schemas, access patterns, and certified metrics.

Domains still own business meaning, domain-specific data products, and validation.

BI teams still own reports and semantic models.

AI agents should consume these governed assets through the same access paths as other consumers.

The only difference is that AI adds a new execution layer: the harness.

The harness should enforce:

who the user is;
what the user can access;
what tools the agent can call;
what data the LLM can see;
what can be stored in memory;
what queries can be executed;
and what results can be returned.

This keeps AI governance aligned with the rest of the data platform.

Conclusion

AI governance is not only about model risk or prompt safety. For enterprise data, it is also about access management.

The key is to recognize that AI access is layered.

The LLM can see one thing. The agent can see another thing. The user can see another thing.

These layers must be governed separately.

The LLM should receive as little internal data as possible. It should reason over metadata, instructions, code, and approved summaries rather than raw enterprise data by default.

The user should only see data they are authorized to access through identity groups and secured schemas.

The agent should run inside a controlled harness that enforces permissions, executes tools, validates queries, manages memory, logs actions, and prevents leakage.

This is how AI can fit into a governed data platform.

Not by bypassing the hub-and-domain model, but by extending it.

The hub provides the certified data foundation. Domains provide business meaning. The AI harness provides controlled execution. The LLM provides reasoning.

Only when these layers are separated can AI agents safely answer business questions without becoming a new uncontrolled access path.

Hub & Domains: A Practical Data Operating Model

2026-06-25T13:30:00Z

In data, the concept of domains has become useful over the last few years.

A domain combines a business area, a team, data products, and operational responsibilities. The underlying idea is straightforward: the people closest to the business should also be closest to the data. That also means that the team that owns it is responsible for its quality, meaning, support, and evolution.

Instead of the old, hyper-centralized data warehouse model, where a single team managed a single enterprise data warehouse for the whole organization, the move was towards more decentralized, business-oriented ownership.

In theory, this makes a lot of sense.

Sales understands sales data better than the central IT team does. Finance understands margin, revenue, and cost rules better than a generic data platform team. Supply Chain understands inventory, shipments, fulfillment, and plant-level operations better than a centralized reporting function.

So the argument for domains is strong: data should be owned closer to the business context.

But the problem with domains is governance.

From Hub & Spoke to Hub & Domains

Much of the work required to make data reliable at the enterprise level is specialized, technical, operational, and frankly, not very exciting. Managing access. Maintaining conforming facts and dimensions. Controlling production deployments. Monitoring SLAs. Running incident processes. Managing secured views. Auditing who can see the margin data. Keeping documentation up to date. Testing certified metrics. Reviewing breaking changes.

Small, analytics-focused domain teams are usually not designed for this kind of work.

They may be very good at business analysis, dashboarding, and domain-specific data modeling. But they often lack the capacity, tools, or operational discipline to manage enterprise-grade governance.

This is why I think we need to bring back some centralization into the domain model. Without this balance, organizations often end up with either excessive centralization that slows delivery or excessive decentralization that complicates governance.

Not full centralization. Not a return to the old model where one team owns everything and business teams wait months for every change.

But some centralization.

The model I recommend is the hub & domains model.

It takes the useful part of the domain concept (such as business ownership and proximity) and combines it with the useful part of the hub-and-spoke model ‒ centralized governance, platform control, standards, and distribution.

In a classic hub-and-spoke model, the hub handles what can be done efficiently in one place: the data platform, infrastructure, shared standards, governance rules, access patterns, monitoring, deployment processes, and shared data assets.

The spokes focus on business-specific work.

In a data organization, those spokes are domains.

So instead of thinking only in terms of data mesh or hub-and-spoke, I prefer the term Hub & Domains.

The hub owns the shared foundation. Domains own the business-specific extensions.

Why Hub & Domains Work Better Than Pure Decentralization

The fully decentralized version of data mesh sounds attractive, but it creates a practical problem.

If every domain independently manages data access, metric definitions, dashboards, pipelines, semantic models, and data contracts, the organization quickly ends up with too many inconsistent practices.

One domain may define revenue one way. Another may define customer hierarchy differently. Over time, these differences become metric reconciliation problems. One team may have proper access controls. Another may expose sensitive data through a dashboard. One domain may have strong documentation. Another may depend on one analyst’s notebook.

Hub & Domains is a more practical compromise.

The hub centralizes the things that should be standardized:

data platform and infrastructure;
conformed dimensions and certified metrics;
access management patterns and governance;
best practices;
security and auditability.

The domains own the things that require business proximity:

business validation;
domain-specific rules;
domain-specific facts and dimensions;
local metrics;
domain data products.

This is not about taking ownership away from the business. It is about separating what should be centralized from what should stay close to the business.

Responsibility Matrix

The challenge with Hub & Domains is that it can become messy if responsibilities and boundaries are not clearly defined.

The usual recommendation is to create a RACI matrix.

I still recommend this.

But honestly, maintaining a RACI is hard work. Responsibilities move over time. Teams change. People leave. New tools are introduced. Domains mature at different speeds. And there is responsibility creep everywhere.

This is why responsibilities should not only be documented. They should be designed into the operating model.

A good principle is:

Design responsibilities so that they require as little manual control as possible.

For example, if a domain does not have access to change a certified metric in the hub code repository, there is no need to monitor whether the domain changes the certified metric logic. They simply cannot do it.

If BI analysts do not have direct access to global hub tables, there is less risk of them accidentally exposing restricted columns in a dashboard.

Governance should be supported through architecture, permissions, code ownership, and deployment workflows. Documentation and RACI models remain useful, but they become significantly more effective when responsibilities are reinforced by the platform’s design.

A practical split could look like this:

What the Hub Should Own

A hub should not own everything. If it does, the company goes back to the old centralized model.

The hub should focus on areas where standardization improves governance, reduces risk, and simplifies support across the organization.

1. How data is distributed, and access is managed

This includes secured tables, secured schemas, identity groups, access reviews, and approved distribution patterns.

For example, the hub may maintain a global invoice fact:

hub_facts.fact_invoice_line

But consumers should not query it directly.

Instead, the hub may expose secured schemas such as:

hub_facts_nam
hub_facts_sales
hub_facts_finance
hub_dimensions_customer_sales
hub_dimensions_shared

Each schema is mapped to identity groups such as:

north_america_sales
global_finance
powerbi_sales_analysts
executive_reporting

This gives the hub a clean way to control data access without making every access request a custom table-by-table exception.

2. How production code is approved and deployed in the hub

The hub should also control how production-grade data assets are added or changed.

This does not mean the hub writes all the code.

Domains may contribute code. Analysts may propose changes. Business teams may define new logic. But if the asset becomes certified and shared, it should go through the hub’s production process.

For example, if Sales creates a useful conversion model that becomes important to multiple teams, it should not remain hidden in a domain notebook or a Power BI dataset. It should be reviewed, tested, versioned, documented, and moved into the hub or into a governed domain production repository.

The hub should define the path from local asset to certified asset.

To understand Hub & Domains, it helps to remember the five access layers:

the fact or dimension table;
the secured table or SQL view;
the secured schema;
the identity group;
the consumption layer.

The hub should control layers 1-4.

The fifth layer (the consumption layer) is different. This is where Power BI, Tableau, Excel, notebooks, AI agents, semantic models, and dashboards live.

The hub cannot be fully responsible for how BI analysts distribute data inside the consumption layer.

The hub can provide secure schemas. It can provide certified facts and dimensions. It can define approved metrics. It can enforce access at the database or lakehouse level.

But once a BI analyst builds a dashboard, creates a semantic model, exports data, adds row-level security in Power BI, or shares a report with users, accountability must move to the consumption owner.

This does not mean the hub has no interest in the consumption layer. The hub should provide standards and may audit critical reports.

But ownership must be clear.

The hub handoff point is the secured schema. After that, the dashboard, semantic model, and final distribution are owned by the BI team, domain, or reporting owner.

Hub to Domain

The most important sharing pattern is hub-to-domain.

The hub should share data with domains via secure schemas, not via direct access to every underlying table. The exact technical implementation depends on the database or lakehouse architecture.

If the company uses one database per domain, the hub can share secured views into the domain database. The domain team then finds approved hub data inside its own workspace.

For example, the Supply Chain domain may receive:

hub_facts_supply_chain
hub_dimensions_material
hub_dimensions_plant
hub_dimensions_shared

The Sales domain may receive:

hub_facts_sales
hub_dimensions_customer_sales
hub_dimensions_material_sales
hub_dimensions_shared

The Finance domain may receive:

hub_facts_finance
hub_dimensions_finance
hub_dimensions_legal_entity
hub_dimensions_profit_center

If the company uses a single database with multiple schemas, the same concept still applies. The hub creates secure schemas and grants access to the domain identity groups.

If the company uses a data-sharing mechanism, the hub shares the secure data products within the domain workspace.

The architecture can vary. The principle should not.

The hub:

creates the certified facts and dimensions;
creates secured tables or secured views;
groups them into secured schemas;
maps schemas to identity groups;
shares the schemas with domains.

The domain then uses those approved assets to build its own domain-specific products.

For example, Supply Chain may build an availability-to-promise model. Sales may build a pipeline conversion model. Finance may build a management reporting model.

But the domain does not need uncontrolled access to all hub tables to do this.

Domain to Hub

In the data mesh world, domains often share data directly with other domains. They publish a data product, define a data contract, and other domains consume it.

This can work in theory.

But in many enterprises, point-to-point domain sharing becomes dependency hell.

Sales depend on the Supply Chain. Finance depends on Sales. Operations depend on Finance. Then the definition changes in one domain, and nobody knows which downstream model broke. Access policies differ by domain. Quality checks are inconsistent. Documentation is incomplete. Nobody has a full view of the dependency graph.

Auditability becomes difficult. Data quality becomes difficult. Change management becomes difficult.

So in Hub & Domains, I recommend a different pattern:

Domains should uplift reusable data assets to the hub.

This means that if a domain creates a data asset that becomes useful beyond that domain, the next step should not be uncontrolled point-to-point sharing. The next step should be a conversation with the hub.

For example, suppose the Sales domain creates a sales conversion model.

At first, this may be a domain-level product. It supports Sales management and is maintained by the Sales analytics team.

But over time, Finance wants to use it for forecasting. Marketing wants to use it for campaign attribution. Leadership wants it in executive reporting.

At that point, the model is no longer only a Sales domain asset. It has become an enterprise dependency.

The Sales domain should then work with the hub to uplift the asset.

This may mean:

moving the code into the hub repository;
applying hub testing standards;
documenting the grain and business rules;
aligning dimensions;
defining ownership;
creating secured schemas;
defining access groups;
publishing it as a certified or domain-validated data product.

Once uplifted, the hub can distribute the asset using the same hub-to-domain or hub-to-BI patterns.

Hub to BI Analysts

Not every team belongs to a domain, and not every analyst needs a domain.

A domain is needed when a team produces and maintains reusable data assets: facts, dimensions, transformations, data products, or domain-specific models.

But if the goal is mostly reporting, like for example Power BI dashboards, Tableau dashboards, Excel analysis, or management packs, it is usually simpler to treat the team as a consumer group.

For BI analysts, the hub should provide conformed, secure facts and dimensions.

For example, a Power BI sales analyst may receive access to:

hub_facts_sales
hub_dimensions_customer_sales
hub_dimensions_material_sales
hub_dimensions_shared

The analyst can then build dashboards and semantic models on top of those assets.

But the analyst should not receive direct access to:

hub_facts.fact_invoice_line

unless there is a very strong reason.

This keeps the model cleaner.

The hub owns the certified facts and dimensions. The BI analyst owns the dashboard and consumption logic.

Conclusion

Domains are useful because they bring data ownership closer to the business.

But domains alone are not enough.

Enterprise data still needs shared standards, access management, certified facts, conformed dimensions, deployment processes, monitoring, and governance. This work is difficult to decentralize well.

That is why Hub & Domains is a better operating model for many companies.

The hub centralizes what should be consistent. The domains own what requires business proximity. BI teams consume certified assets and own the reporting layer.

The goal is not to return to a fully centralized data warehouse model. The goal is to avoid the opposite problem: every domain building its own isolated version of the truth.

A good Hub & Domains model allows organizations to maintain governance and consistency without moving decision-making too far away from the business. The hub makes data safe, certified, and reusable. The domains make data relevant to the business.

Together, they make data distribution easier to govern without slowing everything down.

Certified Metrics: From Fact to Dashboard

2026-06-25T11:00:00Z

Most companies do not struggle with metrics because no one has created a formula. They struggle because the formulas are not governed, the underlying facts are not certified, the dimensions are not conformed, and the same number is recreated in too many places.

As organizations invest more in reporting, analytics, and AI, the importance of metric governance continues to grow. When business functions rely on different definitions, calculations, or dimensions, trust in the numbers quickly erodes.

In a previous article, we discussed the governance and distribution of certified facts and dimensions. Metric certification builds on that foundation. If facts and dimensions are not governed consistently, different teams will inevitably produce different versions of the same metric, even when they believe they are answering the same business question.

A metric is not certified simply because it appears in a dashboard. A metric is certified when the organization understands:

which fact table it comes from;
which measures it uses;
which dimensions it depends on;
where the formula is calculated;
who owns the definition;
who approved it;
where it can be reused;
and how changes are controlled.

This distinction matters because metric governance is often discussed too late. Companies first build dashboards, then notice that numbers do not match, and then start debating definitions. At that point, the problem is harder to fix because the logic is already spread across reports, extracts, semantic models, spreadsheets, and local analyst code.

A better approach is to certify metrics from the bottom up.

The order matters.

You cannot reliably certify a dashboard if the aggregate metric is not certified. You cannot certify an aggregate metric if the underlying measures and dimensions are not certified. And you cannot certify measures if the fact table itself is not trusted.

The certification model should therefore follow the chain:

certified fact;
certified measures;
certified dimensions;
certified aggregate metric;
certified dashboard.

Only then can users trust the number.

Certification Model

Certification is more than writing down a formula.

A certified metric should answer a wider set of questions:

What is the business definition?
What is the calculation?
Is it an additive measure or an aggregate metric?
Which fact table does it use?
Which base measures does it depend on?
Which dimensions are approved for slicing it?
Who owns the metric?
Who approved it?
Is access restricted?
Where is the metric allowed to be used?
What happens when the definition changes?

Certification should make the metric operational. It should be clear how the number is calculated, where it is calculated, who owns it, and when it can be trusted.

Order Matters: From Fact to Dashboard

A common mistake is to certify the dashboard first.

A dashboard may look official because it is used by leadership, published in Power BI, or included in a recurring management meeting. But that does not mean the numbers inside it are certified.

A dashboard is only as trustworthy as the chain underneath it.

If the fact table is wrong, the dashboard is wrong.

If the measures are inconsistent, the dashboard is inconsistent.

If the dimensions are not conformed, users may see different numbers even when the metric formula is correct.

If the aggregate metric is calculated differently across reports, the dashboard becomes another version of the truth.

This is why certification should move from data foundation to consumption:

Certify the fact. Certify the measures. Certify the dimensions. Certify the aggregate metric. Then certify the dashboard.

This order prevents a common failure mode: companies certify reports without certifying the data structures that feed them.

Certified Fact

The first step is the certified fact.

A fact table represents a business event, transaction, or measurable state. Examples include invoice lines, sales order lines, purchase order lines, inventory snapshots, shipments, payments, production confirmations, or general ledger postings.

For example:

fact_invoice_line

A certified fact should define the business process and the grain.

For example, is the fact one row per invoice line? One row per order line? One row per shipment? One row per inventory snapshot by material, plant, and date?

This matters because all metrics built on top of the fact depend on the grain.

A certified fact should define:

the business event or process;
the grain of the table;
the source system;
the primary business keys;
the available measures;
the connected dimensions;
the refresh frequency;
the owner;
the quality checks;
and the known limitations.

For example, a certified invoice fact may be documented as:

Fact: fact_invoice_line
Grain: one row per invoice line
Source: ERP billing document
Owner: Finance / Data Hub
Measures: sales amount, discount amount, cost amount, quantity invoiced
Dimensions: customer, material, legal entity, sales organization, calendar

This is the foundation. If the fact is not certified, the metrics built on top of it are already unstable.

The practical rule is:

Do not certify important metrics on top of uncertified facts.

Certified Measures

The second step is certified measures.

Measures are the numeric fields available inside the fact table. In the Kimball language, these are usually called measures. Sometimes they are also called derived measures when they are calculated during transformation.

These are usually additive measures such as:

sales amount;
cost amount;
discount amount;
margin amount;
quantity sold;
quantity shipped;
inventory value;
invoice amount;
payment amount;
open order quantity;
production quantity.

These measures can usually be summed across dimensions.

For example:

sales_amount

can be summed by customer, product, region, month, legal entity, or sales channel.

This is why additive measures are the easiest metrics to certify from the hub. The hub can calculate them once, store them in the fact table, test them, document them, and expose them consistently to downstream users.

For example, a certified invoice fact may contain:

sales_amount
discount_amount
cost_amount
margin_amount
quantity_invoiced
net_revenue_amount

These measures are part of the fact table itself. They should not be recreated in every dashboard.

The practical rule is:

If a metric is additive and belongs to the grain of the fact table, calculate it in the data warehouse.

For example, if net_revenue_amount is always defined as gross revenue minus discounts, then that calculation should not be repeated in Power BI, Excel, SQL notebooks, and multiple semantic models. It should be calculated once in the certified fact.

This does not remove all business debate. The business still needs to agree on the definition. But once the definition is agreed, the hub is usually the right place to implement it.

Certified Dimensions

The third step is certified dimensions.

This is where many metric programs fail.

Sometimes numbers do not match because the metric definition is wrong. But very often, the metric is fine. The real problem is the dimension.

For example, two analysts may both use the same certified metric:

sales_amount

But one analyst groups sales using one version of the region dimension, while another analyst uses a different version.

One report uses:

dim_customer_sales_region

Another report uses:

dim_management_region

Both reports calculate sales correctly, but they group sales differently. The numbers do not match because “EMEA” does not mean the same thing in both reports.

This is not a metric problem. It is a dimension problem.

In my experience, this is one of the biggest causes of reporting disagreement. Companies spend a lot of time debating metric formulas, but the real issue is often that dimensions are not conforming.

Examples include:

different customer hierarchies;
different product hierarchies;
different region mappings;
different business unit definitions;
different cost center groupings;
different rules for active versus inactive customers;
different calendar definitions;
different legal entity mappings;
different currency conversion rules.

If dimensions are not conformed, certified measures will still produce inconsistent reports.

For example, a certified sales_amount measure does not solve the problem if Sales, Finance, and Operations each use a different customer hierarchy.

This is why metric certification and dimension governance must be handled together.

A certified metric should document not only the formula, but also the approved dimensions that can be used with it.

For example:

Certified measure: sales_amount
Certified fact: fact_invoice_line
Approved dimensions: dim_customer, dim_material, dim_calendar, dim_legal_entity, dim_sales_organization

If an analyst uses a local spreadsheet mapping instead of the certified customer dimension, the result may be useful for exploration, but it should not be treated as certified reporting.

The practical rule is:

Do not certify formulas without certifying the dimensions used to slice them.

Certified Aggregate Metric

The fourth step is the certified aggregate metric.

Not all metrics can be calculated row by row inside the fact table. Some only make sense after aggregation.

A common example is sales margin percentage:

(sales - cost) / sales

This looks simple, but it should not usually be calculated at the invoice-line level and then summed or averaged.

The correct calculation is usually:

SUM(sales_amount - cost_amount) / SUM(sales_amount)

or:

SUM(margin_amount) / SUM(sales_amount)

This means the metric must be calculated after the relevant rows have been aggregated.

These are aggregate metrics. They are the type of metrics you would normally calculate with a pivot table formula in Excel: first aggregate the base measures, then calculate the ratio.

Examples include:

margin percentage;
conversion rate;
on-time delivery percentage;
average selling price;
average order value;
return rate;
fill rate;
forecast accuracy;
inventory turnover;
days sales outstanding;
utilization percentage.

These metrics depend on the aggregation context.

For example, margin percentage can be calculated by customer, product, month, region, business unit, or sales representative. The formula may be the same, but the grouping changes the result.

This is why aggregate metrics usually belong in a semantic model, a BI model, a metrics layer, or a carefully controlled reporting layer.

The important point is that they still need certification.

A certified aggregate metric should define:

the formula;
the base measures used;
the required fact table;
the approved dimensions;
the aggregation behavior;
the handling of nulls and zero denominators;
the expected grain;
and the owner.

For example, a certified margin percentage metric should specify whether it is:

SUM(sales_amount - cost_amount) / SUM(sales_amount)

or:

SUM(margin_amount) / SUM(net_revenue_amount)

Those are not always the same thing.

Small formula differences create significant trust and reconciliation problems.

The practical rule is:

Store additive components in facts. Certify aggregate formulas separately.

Certified Dashboards

The fifth step is the certified dashboard.

A dashboard should be certified only when the full chain underneath it is certified.

That means:

the source fact is certified;
the measures are certified;
the dimensions are approved;
the aggregate metrics are defined and governed;
the semantic model or BI model is controlled;
access rules are understood;
and the business owner has approved the dashboard for use.

A dashboard should not be certified simply because it is popular, polished, or used by senior management. Certified dashboards should consume certified metrics, not recreate them. Otherwise, every dashboards becomes a potential source of new business logic and metric variation.

For example, a Power BI dashboard may show:

Sales Amount;
Margin Amount;
Margin %;
Average Selling Price;
Sales by Customer;
Sales by Region;
Sales by Product Group.

If the dashboard defines these measures locally, and another dashboard defines them differently, the company still has multiple versions of the truth.

A better model is for the dashboard to consume metrics from a governed semantic model or metric layer built on certified facts and dimensions.

Semantic models can be useful here. They may exist inside tools such as Power BI, Tableau, or Looker. They may also exist as a separate semantic layer or metrics service.

A semantic model can define:

measures;
aggregate metrics;
relationships;
hierarchies;
approved filters;
row-level security;
and reusable business definitions.

This is useful, but it should not hide weak foundations.

If the underlying facts are not certified, the semantic model becomes a thin layer over unstable data. If the dimensions are not conformed, the semantic model may produce consistent formulas but inconsistent business cuts. If every department creates its own semantic model, the company ends up with many versions of the truth again, just at a higher layer.

The clean model is:

Facts and additive measures belong in the data warehouse. Aggregate metrics may live in a semantic model. Dimensions must conform. Dashboards should consume certified metrics, not redefine them.

Certification Levels

Not all metrics need the same level of governance. A practical certification process should align the level of control with the metric’s importance and sensitivity.

Experimental

Used for analysis or exploration. No formal approval. No SLA. Logic may change. Not recommended for official reporting.

This is useful when analysts are still testing ideas.

For example, a Sales analyst may create a temporary pipeline quality score to explore whether certain opportunity attributes predict conversion. That metric may be useful, but it should not immediately become an enterprise KPI.

Domain Validated

Validated by a business domain. Suitable for domain-level reporting, but not necessarily approved across the whole enterprise.

For example, Sales may define a pipeline quality score that is useful for sales management but not used by Finance.

This level is useful when a metric is real and governed inside a domain, but not yet accepted as an enterprise standard.

Hub Certified

Approved for enterprise reporting. The definition is documented, tested, owned, and built on certified facts and dimensions.

Examples include:

net sales;
gross margin;
inventory value;
on-time delivery percentage;
sales order backlog;
working capital;
production quantity.

These metrics can be reused across domains, dashboards, semantic models, and management reporting.

Restricted Certified

Certified, but access is restricted because the metric depends on sensitive data.

Examples include:

margin percentage;
customer profitability;
employee productivity;
vendor payment risk;
credit exposure;
cost-to-serve.

The metric is trusted, but not universally available.

This distinction is important because not every metric should be enterprise-certified. Some metrics are local, temporary, sensitive, or domain-specific.

The goal is not to certify everything. The goal is to prevent important metrics from being recreated without control.

Metrics Catalog

It is not enough to define certified metrics in documentation, semantic models, SQL files, or BI reports. Users need a place to discover which metrics exist, what they mean, where they come from, and whether they are approved for use.

This is the role of the metrics catalog.

A metric catalog provides visibility into certified metrics, ownership, definitions, approved dimensions, and certification status. It becomes a reference point for analysts, business users, and reporting teams seeking to understand which metrics to use and which definitions are considered authoritative.

Conclusion

Certified metrics are not only formulas.

A metric becomes trustworthy when the organization controls the full chain from fact to dashboard.

The sequence matters:

certify the fact;
certify the measures;
certify the dimensions;
certify the aggregate metric;
certify the dashboard.

This is where many metric programs fail. They document formulas but ignore dimensions. They create semantic models but leave facts unstable. They certify dashboards but do not certify the underlying data.

A better certification model connects the full chain:

certified fact;
certified measures;
certified dimensions;
certified aggregate metric;
certified dashboard.

Only then can organizations build reporting, analytics, and AI initiatives on a trusted foundation.

The practical goal is not to certify every number in the company. The goal is to make the important numbers reliable, reusable, and explainable.

That is what certified metrics are really about.

Distributing Facts and Dimensions: Governance, Access & Ownership

2026-06-24T10:00:00Z

Modern data platforms often focus on ingestion, modeling, orchestration, and reporting. However, once facts and dimensions have been built, a different challenge emerges: how should those assets be distributed across the organization?

This is not only a security question. It is also a question of governance, ownership, and operating models.

A central data hub may own certified facts and dimensions, but those assets should not automatically be exposed to every analyst, dashboard, domain team, or application. A sales invoice fact may contain revenue, discounts, cost, margin, customer, business unit, and geography. A customer dimension may contain commercial assignments, payment terms, credit attributes, or sensitive identifiers.

The challenge is to keep authoritative data centralized while exposing only the data required by each audience. Access management should therefore be designed in layers.

The Five Access Layers

A practical distribution model can be organized through five layers:

certified facts and dimensions;
secured tables;
secured schemas;
identity groups;
consumption applications.

Each layer has a different responsibility.

Layer 1: Certified Facts and Dimensions

The first layer is the underlying certified data asset.

For example, the hub may maintain a global invoice fact:

hub_facts.fact_invoice_line

This table is the authoritative source. It contains the complete dataset and is owned by the data hub.

This does not mean every consumer should access it directly. In fact, direct access is rarely required.

The global table may contain data for all regions, legal entities, business units, and commercial attributes. It may also include sensitive fields such as cost, margin, discounts, credit exposure, or customer-level details.

The role of Layer 1 is to provide the trusted foundation. It is not necessarily the consumption layer.

A good rule is:

Certified facts and dimensions should be built once in the hub but exposed through controlled-access layers.

Layer 2: Secured Tables

The second layer is the secured table.

If the source table is too broad to expose directly, the hub can publish a narrower SQL view that filters rows, removes sensitive columns, or applies business-specific restrictions.

For example:

hub_facts_nam.invoice_lines

This view may expose only North America transactions and exclude restricted columns such as cost or margin.

The secured table controls what data a consumer can see. It becomes the first layer of distribution between the certified fact and the consuming audience.

For example, the global fact may contain:

hub_facts.fact_invoice_line

But the hub may expose different secured versions:

hub_facts_nam.invoice_lines
hub_facts_europe.invoice_lines
hub_facts_sales.invoice_lines
hub_facts_finance.invoice_lines

Each secured table can represent a different approved view of the same underlying fact.

For example:

Finance may see revenue, cost, and margin.
Sales may see revenue and discounts, but not cost or margin.
Regional teams may only see records for their geography.
Executive reporting may receive aggregated facts instead of transaction-level data.

This keeps the global fact centralized while still allowing different audiences to consume safe, approved versions of it.

Layer 3: Secured Schemas

The third layer is the secured schema.

A secured schema groups related secured tables together and provides a logical access boundary.

Examples include:

hub_facts_nam
hub_facts_europe
hub_facts_apac
hub_facts_sales
hub_facts_finance
hub_dimensions_customer_sales
hub_dimensions_finance
hub_dimensions_shared

Consumers should normally be granted access to schemas, not directly to individual source tables.

The secured schema controls which collection of data products a consumer can access.

For example, the hub_facts_nam schema may contain the approved North America facts. The hub_dimensions_customer_sales schema may contain the customer attributes approved for sales reporting. The hub_dimensions_shared schema may contain broadly reusable dimensions such as calendar, currency, and material.

This gives the hub a clean access boundary. Instead of granting permissions table by table across the entire platform, the hub can package approved data products into schemas and grant access to those schemas.

Layer 4: Identity Groups

The fourth layer is the identity group.

Identity groups represent the people or teams who are allowed to access a secured schema.

Examples include:

north_america_sales
europe_sales
global_finance
commercial_leadership
sales_operations
powerbi_sales_analysts

Access is granted by mapping identity groups to secured schemas.

For example:

Identity Group	Secured Schema
north_america_sales	hub_facts_nam
europe_sales	hub_facts_europe
global_finance	hub_facts_finance
commercial_leadership	hub_facts_sales
powerbi_sales_analysts	hub_facts_sales, hub_dimensions_customer_sales, hub_dimensions_shared

This creates a simple model:

secured tables control what data is exposed;
secured schemas control how data is packaged;
identity groups control who can access it.

Access should normally be granted to groups, not directly to individuals. Individual access becomes difficult to maintain and audit over time. People change roles, move teams, leave the company, or join temporary projects. If access is managed on a person-by-person basis, it quickly becomes difficult to control.

A group-based model is easier to govern, review, and explain.

Layer 5: Consumption Applications

The fifth layer is the consumption application.

This includes tools such as Power BI, Tableau, Looker, Excel, notebooks, APIs, embedded analytics, and AI agents.

These tools may apply additional security. For example, Power BI supports row-level security, which can filter rows based on the user viewing a report. A semantic model may also restrict measures, hide columns, or expose different report pages to different audiences.

This layer is useful, but it should not be the only access control layer.

Application-level security is often closely tied to the user experience, but it can be inconsistent when different tools access the same data in different ways. A Power BI report may enforce row-level security, while a notebook, SQL endpoint, API, or AI agent may access the same data through a different path. Unless the control exists closer to the data, consistent enforcement becomes difficult.

For this reason, consumption-layer security should be treated as an additional control, not the foundation of the access model.

The hub should first control access through secured tables, secured schemas, and identity groups. Consumption applications can then add more specific restrictions where needed.

Data Access Matrix

Once the five layers are defined, the access model should be documented in a simple matrix.

The purpose of the matrix is not to describe every permission in the platform. It is to make the main access paths visible and auditable: which identity groups have access to which secured schemas, and which secured tables are available inside those schemas.

The data access matrix has these columns:

Identity Group
Members of the Group
Secured Schemas
Secured Tables
Dashboards (optional)

The data access matrix gives the hub a simple control document. It also provides visibility into how certified data is distributed across the organization and helps ensure access remains aligned with ownership and governance policies. It can be used during access approvals, audits, onboarding, access reviews, and design discussions with domains or BI teams.

A simplified version of the matrix below:

Fact Layered Governance

Facts are often the most sensitive assets on a data platform because they capture measurable business activity: revenue, costs, margins, inventory values, payments, production quantities, operational performance metrics, and financial transactions.

For this reason, facts should normally be exposed through the secured table, secured schema, and identity group model.

A single global fact may exist internally:

hub_facts.fact_invoice_line

But different audiences may consume different secured versions of that fact.

For example:

Finance may access a secured table that includes cost and margin.
Sales may access a secured table that excludes cost and margin.
Regional teams may access secured tables filtered to their geography.
Executive reporting may access aggregated facts rather than transaction-level details.

These secured tables can then be grouped into schemas such as:

hub_facts_nam
hub_facts_sales
hub_facts_finance

Identity groups are then mapped to the appropriate schema.

This allows the hub to maintain a single authoritative fact while exposing audience-specific versions through a controlled-access model.

The important principle is:

Do not duplicate ownership of the fact. Duplicate only the controlled access paths.

The hub still owns the certified fact. The secured tables and schemas are distribution mechanisms.

Dimension Layered Governance

Dimensions are usually less sensitive than facts, but they still require access controls.

Some dimensions can often be shared broadly:

dim_calendar
dim_currency
dim_plant
dim_warehouse
dim_material
dim_profit_center
dim_cost_center

These dimensions provide a common context for reporting and analytics. In many companies, broad access is acceptable.

But this should not be assumed for every dimension.

Dimensions such as customers, suppliers, employees, vendors, products, or patients may contain sensitive, commercial, financial, or personal information.

For example, a customer dimension may contain:

customer name;
address;
tax ID;
payment terms;
credit attributes;
sales assignments;
segmentation;
pricing group;
legal identifiers.

Not every user who needs customer reporting needs all of these attributes.

The hub can therefore publish secured versions such as:

dim_customer_public
dim_customer_sales
dim_customer_finance
dim_customer_sensitive

Each secured table exposes only the attributes appropriate for a specific audience. These secured tables can then be grouped into schemas and assigned to identity groups.

Dimension access should also align with fact access.

For example, a user with access to North America sales facts will usually need compatible access to customer, material, calendar, and legal entity dimensions. The secured schemas assigned to that user should contain both the facts and dimensions required for analysis.

The practical rule is simple:

Use the secured table, secured schema, and identity group model whenever dimensions contain information that should not be universally accessible.

Practical Access Management Rule

For many organizations, a practical access model follows a simple principle:

Build global facts and dimensions once. Expose them through secured tables and secured schemas. Map those schemas to identity groups. Give domains and BI analysts only the schemas they need. Let consumption tools add extra security where appropriate.

This model is not perfect for every scenario, but it is simple, transparent, and easier to operate than highly personalized table-by-table permissions or row-level security everywhere.

Row-level security still has a place. It may be appropriate when access rules are highly dynamic, when users need personalized data scopes, or when the organization has mature identity and policy management.

But it should be a deliberate design choice, not the default.

Most companies should start with the simpler model:

certified facts and dimensions;
secured tables;
secured schemas;
identity groups;
controlled consumption applications;
regular access reviews.

That is usually enough to make facts and dimensions safe to distribute without turning access management into an unmaintainable system.

What the Claude Code Leak Reveals About Enterprise AI

2026-06-19T12:20:00Z

The recent leak of the Claude Code source code provided a rare view of high-level AI development. A significant finding: the application dedicates only 1.6% of its codebase to core decision logic. The remaining 98.4% consists of standard development code: a layer now frequently referred to as “harness engineering.”

While technical circles have focused on the code itself, there has been limited discussion regarding its business implications. Specifically, there is a need to address ROI, direct P&L impact, and how AI initiatives should be structured to provide measurable value on a balance sheet.

While many companies are rushing to build sophisticated agents, a significant portion of the value still comes from writing solid, deterministic code. The challenge is not pushing every task to an AI but breaking down complex business processes into smaller, manageable steps. In most cases, these steps can be executed as deterministic functions: the equivalent of an Excel formula. In this paradigm, the Large Language Model (LLM) acts as a specialized orchestrator that delegates tasks to deterministic functions, rather than performing every step itself.

Source: Dive into Claude Code: The Design Space of Today’s and Future AI Agent Systems; Liu, J., Zhao, X., Shang, X., & Shen, Z. (2026).

Governance and Cost Control

The Claude Code codebase reveals a significant investment in a complex permission system that governs access, ownership, and responsibilities. This mirrors enterprise governance. And interesting enough, this system is entirely deterministic. It demonstrates that the most successful AI tools handle high-stakes governance through traditional, rule-based code rather than relying on probabilistic models.

This architectural choice highlights an important design principle: use LLMs where reasoning is required, but avoid using them for operational logic that can be implemented through deterministic code. Over-reliance on probabilistic models for every task introduces two risks: cost volatility and vendor dependency.

The solution is a “deterministic-first” architecture. By handling simple business tasks through traditional code, you minimize the number of tokens required for orchestration. Deterministic code is predictable, cost-effective, and auditable. When structural heavy lifting (process execution and governance) is handled by your code, the LLM is freed to focus on reasoning and decision support. This reduces token consumption while making the overall solution easier to govern and operate.

A useful excerpt from the Claude Code leak paper:

The safety-by-default principle is implemented through seven independent layers. A request must pass through all applicable layers, and any single layer can block it:

Tool pre-filtering (tools.ts): Blanket-denied tools are removed from the model’s view before any call, preventing the model from attempting to invoke them.
Deny-first rule evaluation (permissions.ts): Deny rules always take precedence over allow rules, even when the allow rule is more specific.
Permission mode constraints (types/permissions.ts): The active mode determines the baseline handling for requests that match no explicit rule.
Auto-mode classifier: An ML-based classifier evaluates tool safety and may deny requests that the rule system would allow.
Shell sandboxing (shouldUseSandbox.ts): Approved shell commands may still execute inside a sandbox, restricting filesystem and network access.
Not restoring permissions on resume (conversationRecovery.ts): Session-scoped permissions are not restored on resume or fork.
Hook-based interception (types/hooks.ts): PreToolUse hooks can modify permission decisions; PermissionRequest hooks can resolve decisions asynchronously alongside the user dialog (or before it, in coordinator mode).

Process Before Tools

Value is built through productivity gains. Consider a simple lemonade stand: if you are squeezing lemons by hand, you manage one lemon per minute. You create significant value by investing in a citrus juicer, which increases output to five lemons per minute.

Productivity improvements stem from either better equipment or process optimization. If you pre-squeeze lemons before a rush, you can serve more customers with the same amount of manual labor by optimizing the workflow.

The underlying economics of productivity are the same regardless of the technology being implemented. Whether you are purchasing heavy machinery or implementing generative AI, the goal remains the same: a process must produce more output per unit of cost. AI is a tool that must fit into a logical process design, not a substitute for it.

From Leak to Strategy

To ensure AI implementations generate measurable business value, follow a structured operational approach:

Deconstruct: Map existing processes to understand exactly how work is performed.
Define Future State: Identify where bottlenecks can be removed and where process simplification is possible.
Isolate Deterministic Steps: Identify process parts that follow clear rules and do not require expensive AI inference.
Orchestrate: Apply AI only to the decision logic, using it to manage those deterministic functions effectively.

AI should be the final component of a multi-step improvement initiative, not the justification for the project itself. For process deconstruction, use standard operational frameworks such as Swimlane diagrams, Affinity diagrams, SIPOCs, or Fishbone analyses.

Process Discipline and AI Design

A common mistake is to apply AI to every step of the process simply because it is available. This often results in solutions that are more complex than necessary, more expensive to operate, and harder to govern. The Claude Code leak illustrates the importance of clearly defining what should be handled by deterministic code and what by AI.

If you treat AI like a junior associate who needs strict instructions, guardrails, and clear parameters, you gain predictability. If you treat it like an autonomous executive, you invite chaos. The goal is to build an environment where the AI provides the reasoning, but your architecture provides the reliability. When you design with this level of restraint, you stop experimenting and start engineering business outcomes.

Conclusions

The Claude Code leak is a practical case study in AI architecture. By limiting pure AI inference to 1.6% of the codebase, the developers built a scalable, reliable system.

The implication for business leaders is that successful AI initiatives are not determined by how much intelligence you can pack into an agent, but how little you actually need to achieve the result.

When you shift your focus from “AI-first” to “Logic-first,” you move beyond token cost optimization to building systems that are auditable, stable, and integrated into your bottom line. Stop asking what your AI can decide, and start asking what your deterministic code can handle. That is where real business value is created.

Flat Tables vs. Snowflake Semantic Models: The Ultimate BI Data Debate

2025-04-10T14:07:00Z

Most data ends up in Business Intelligence (BI) reports. That’s no surprise: BI transforms raw data into actionable insights, helping businesses to make informed decisions, spot trends, and drive strategy with confidence.

But when it comes to structuring data for BI, one big question arises: Should data transformations happen within the BI tool, or should you create a flat table in the database?

This debate exists for good reason. Both approaches have their strengths, and neither is universally better. Flat tables often win points for being simple and fast to build. On the flip side, they can be inflexible and difficult to maintain at scale. Meanwhile, semantic models are praised for their scalability and reusability but tend to be harder to set up and govern consistently across teams.

The division often mirrors team roles: data analysts and engineers prefer flat tables for control and simplicity, while BI developers lean towards semantic models for flexibility and user-friendliness in tools like Power BI or Looker.

In this article, I’ll weigh the pros and cons of both approaches and share my perspective on the best way forward. Let’s dive in.

Flat Tables (One Big Table - OBT)

The flat table approach consolidates all your necessary data into a single, denormalized table before it reaches the BI tool. That means one table with all the columns - facts and attributes - already joined and ready for use. The simplest visualization of this structure is a table inside one Excel sheet. This method has several business-related advantages:

Tool Agnostic – Since all transformations happen at the database level, the solution isn’t tied to a specific BI tool. Moving from Power BI to Qlik, Tableau, or another tool becomes much easier.
Consistent Business Logic – Keeping all business rules and transformations in SQL ensures a single source of truth. This minimizes discrepancies between reports and eliminates the risk of conflicting KPIs.
Version Control & Auditing – Storing logic in SQL allows for version control, making it easier to track changes, roll back updates, and maintain data integrity.
Simpler Data Access – Some users don’t need fancy visualizations—just clean, structured data they can extract and analyze. A flat table makes querying easier without needing to understand complex relationships between tables.
Faster Development & Maintenance – Once the OBT is set up, adding new KPIs and making modifications is often quicker and easier than managing complex BI tool transformations.

But, there are some downsides to this approach:

Storage & Performance Concerns

One of the most common concerns with flat tables is performance and storage. Since they contain a lot of redundant data by design, they can become large and occasionally slower to query, especially when dealing with massive datasets or frequent refresh cycles. This is a real trade-off, but whether it actually becomes a problem depends on your system architecture and performance expectations. Flat tables may struggle in high-volume environments, when powering many dashboards, or when real-time performance is critical - particularly if you’re using databases that aren’t optimized for analytical workloads.

However, many of these limitations can be addressed with proper setup. Columnar databases like Redshift, BigQuery, or Snowflake only scan the columns needed, significantly reducing overhead. Loading only the required data into dashboards, applying smart distribution and sort keys, or using incremental refreshes can also improve performance. Most importantly, optimization should be driven by clear performance KPIs - not assumptions. If a dashboard loads within your defined threshold and users are satisfied, chasing “faster” just for the sake of it doesn’t bring real value.

Redundant Data

Without normalization, data duplication is inevitable, increasing storage costs and complicating updates.

Harder to Scale for Large Enterprise Systems

Maintaining a flat table becomes inefficient as datasets grow, leading to costly optimizations.

Semantic Models

A semantic model structures data into fact and dimension tables, optimizing storage and improving efficiency. Instead of consolidating all data into one table, facts (metrics like sales, revenue, or orders) are kept separate from dimensions (descriptive attributes like customers, products, or time periods), resulting in:

Optimized Storage & Performance – Since dimensions are stored separately and referenced via keys, this approach reduces redundancy and makes queries more efficient.
Better Scalability – When dealing with large datasets, semantic models allow for efficient partitioning, incremental loading, and aggregation, making them ideal for complex analytical queries.
More Flexibility in BI Tools – Many BI tools (like Power BI, Looker, and Tableau) have powerful modeling capabilities that work best when data is structured into facts and dimensions.
Supports Advanced Analytics & AI – Having a well-defined semantic layer allows for better integration with AI/ML models and predictive analytics, as structured relationships enhance data analysis capabilities.

But, of course, there are also some trade-offs:

Tool Dependency – Since modeling is done within the BI tool, switching tools may require rebuilding logic, which can be time-consuming and prone to errors.
More Complex for End Users – Users who prefer simple, direct SQL queries may find it harder to navigate multiple tables and relationships.
Risk of Inconsistent KPIs – When too many transformations occur within the BI tool, different reports might show different results for the same KPI, leading to confusion and mistrust in the data.
Longer Development Time – Setting up a semantic model requires detailed planning, coordination, and maintenance, which can slow down report delivery compared to a flat table approach.
Lack of Versioning & Testing – Last but not least, most BI tools don’t offer robust support for version control, testing, or CI/CD pipelines. This makes governance and change management significantly more difficult and slower compared to managing logic in the database. Rolling out changes - especially at the organizational level - can quickly turn into a logistical headache.

Which option wins?

As always, it depends on the specific use case.

Flat tables will always be better when you care about version control, easier testing, and less dependency on specific BI tools. Semantic models will win when you rely heavily on your BI tool’s built-in modeling features.

But if I had to choose, I’d lean towards the OBT approach due to its consistency and business advantages. One of the most common issues I see is inconsistent KPIs across reports because too many transformations are done within the BI tool.

That said, I’m a fan of a hybrid approach. Combining both methodologies could be the best solution:

Facts and essential KPIs should be calculated in SQL at the database level, ensuring a single source of truth.
Master data (dimensions) can remain separate and be joined in the BI tool, optimizing storage and performance.
Hybrid storage strategy for large datasets, storing historical or less frequently used data in a flat table while keeping active data in a semantic model for real-time reporting needs.
Metadata & governance layer ensuring business logic consistency, whether KPIs are defined in SQL or the BI tool.

This approach leverages the best of both worlds - ensuring data consistency while maintaining flexibility and efficiency. It also allows businesses to scale efficiently while minimizing technical debt.

Driving Sustainability with Data: Improving CO₂ Emissions Reporting Across Supply Chains

2025-04-10T13:20:00Z

According to KPMG’s 2024 Survey of Sustainability Reporting, 80% of the 5,800 surveyed companies have established carbon reduction targets. This number highlights the increasing focus on sustainability but also the critical need for accurate emissions tracking. While setting targets is the first step, high-quality, reliable data is essential for consistent and accurate reporting across all areas of a business, particularly in supply chains.

Understanding the Greenhouse Gas Protocol

The Greenhouse Gas (GHG) Protocol categorizes emissions into three scopes, each representing different sources of emissions within a company’s operations:

Scope 1: Direct emissions from company-owned or controlled sources, such as fuel combustion in company vehicles or on-site manufacturing processes.
Scope 2: Indirect emissions from purchased electricity, steam, heating, and cooling used by the company.
Scope 3: Indirect emissions from the company’s broader value chain, including logistics, procurement, and material sourcing.

While Scope 1 and 2 emissions are relatively easier to track, Scope 3 emissions are more complex and often represent the largest share of a company’s total carbon footprint. For many manufacturing companies, the biggest portion of Scope 3 emissions comes from supply chain operations, which will be the focus of this article. We’ll explore the challenges of Scope 3 reporting and how improved data can support compliance and better decision-making.

Why Sustainability Compliance Matters

Companies track their carbon footprint for various reasons, including environmental responsibility, corporate social responsibility (CSR), and good brand perception. Additionally, regulatory requirements play a key role in driving sustainability reporting. Accurate emissions data is crucial to maintaining compliance and meeting corporate sustainability goals.

Regulatory Requirements for Emissions Reporting

Regulations surrounding emissions reporting are becoming increasingly strict globally. In the European Union, the Corporate Sustainability Reporting Directive (CSRD) requires large companies to disclose detailed environmental, social, and governance (ESG) data, including greenhouse gas emissions. This mandate took effect in 2024 and requires reports published in 2025 to include this data. As a result, nearly 50,000 EU companies now need to report their Scope 3 emissions.

Similar regulations exist in the UK (SECR) and in over 40 other countries, including the US, Canada, and Japan (alongside EU countries), all of which have implemented or are planning to implement corporate emissions disclosure requirements for greenhouse gases (GHG).

Challenges of Supply Chain Sustainability Reporting

Effective Scope 3 emissions reporting comes with a number of obstacles, including data complexity, the frequent need for updates, and ensuring accuracy across a diverse range of
sources. Some of the key challenges are:

Poor data quality

One of the biggest challenges in Scope 3 emissions reporting is poor data quality, which can significantly impact the accuracy of sustainability metrics. Companies often deal with incomplete, inconsistent, or outdated data from multiple sources.

Examples of Poor Data Quality Issues include:

Inconsistent material weights or classifications: If different systems use different units (kg vs. tons), this can lead to miscalculations in material emissions.
Duplicate records in master data: Multiple entries for the same supplier or product cause redundancy, leading to errors in reporting.
Data silos and manual entries: Sustainability data is often scattered across ERP systems, spreadsheets, and supplier databases, leading to high risks of human error and inefficiency.

Lack of a Solid Data Foundation

The poor data quality, complexity, and diversity of data required for Scope 3 emissions reporting make it challenging to get a full and accurate picture. To track emissions properly, companies need to collect a wide range of information, including:

Vendor master data (such as LFA1 table in SAP)
Material master data (such as MARA table in SAP)
Plant master data to be able to track shipping routes
Purchasing data on the Purchase Order document level

Many sources of data make data foundations extremely difficult to establish. Not all the data comes from the organization’s internal sources but also requires collaboration with other parties (e.g., vendors).

Another major challenge arises from geographical data. Vendor and Plant locations change frequently, leading to potential errors in emissions calculations. For example, if a vendor relocates its plant, this change must be automatically reflected in route calculations to maintain accurate CO₂ estimates.

Solid data foundations aren’t just about collecting the right inputs - they also require clearly defined, well-structured metrics. The way data is organized should support easy and consistent metric implementation later on. Without this, even the best-quality data can become difficult to interpret or apply reliably across reports, especially when calculating Scope 3 emissions that rely on complex, multi-source inputs.

Collecting Industry Carbon Factors

A common approach in logistics is estimating emissions using a simple emissions factor model based on CO₂ emitted per kilometer traveled for different transport modes. However, material-related emissions are far more complex. This requires gathering and applying carbon factors from multiple sources, such as:

EPD (Environmental Product Declarations): product-specific, 3rd party-verified environmental data per product category
CDP (Carbon Disclosure Project): vendor-specific CO₂ footprints
EPA (Environmental Protection Agency): standardized US-based emission factors for industries (e.g., transport, energy, waste)
Industry average data: generic emission benchmarks based on sector-wide studies for certain materials that lack specific CO₂ data

These carbon factors need to be regularly updated, mapped to specific materials, and often collected manually, making the process cumbersome and prone to errors.

Shortage of Experts Who Understand Both Data and Sustainability

Only a few specialists can bridge the gap between sustainability regulations and data analytics. Applying the correct emission factors (particularly for specific materials) requires deep domain knowledge and industry-specific expertise. This skills gap makes it difficult for many organizations to ensure accurate emissions reporting. To accurately report emissions, professionals need a blend of sustainability knowledge and data expertise. The challenge is that few people have both skill sets - they either specialize in environmental science or in data analytics, but not both.

How to Improve CO₂ Reporting with Structured Data

The challenges around Scope 3 emissions reporting show just how complicated it can get ‒ different sources, frequent updates, and a lot of moving parts. Without a reliable data foundation, companies risk inaccurate reporting, which can lead to poor decisions and compliance issues.

This is where a structured system like a sustainability mart can help.

When it comes to structuring such a mart, there are two common approaches:

Flat Table Approach: This method consolidates all necessary data into a single, denormalized table. It ensures consistency, simplifies development, and is tool-agnostic - making it a favorite among data teams. However, it can become harder to manage at scale, with potential performance and storage challenges depending on dataset size and architecture.
Snowflake Schema Approach in BI Tools (e.g., Power BI): In this approach, data is organized into fact and dimension tables, which are related through keys. While it’s more efficient for large datasets and complex queries, it introduces tool dependency and leads to inconsistent KPIs if governance isn’t tightly controlled.

Personally, I would opt for a flat table approach rather than creating a semantic model inside a BI tool. I’ve explained this choice further in another article, but with sustainability data, the choice between these two approaches is even clearer. Master data plays a crucial role in metric calculations, which makes it almost impossible to keep it outside the mart. A sustainability mart integrates critical data sources into one system, enabling accurate calculations and reporting of ESG metrics. Such a mart can include a variety of relevant components, for example:

With these data points in one integrated system, companies can accurately calculate material and logistics-related emissions. By leveraging a sustainability mart, companies can:

Generate standardized reports for Sustainability Management to ensure consistent and comparable reporting across departments and reporting periods, simplifying compliance with regulatory requirements
Monitor trends through historical snapshots to help businesses track emission reduction progress, identify inefficiencies, and make data-driven sustainability decisions
Automatically track and apply changes in master data (e.g., updates to vendor or material information) to prevent outdated or incorrect sustainability data

Conclusion: The Path to Better Sustainability Reporting

Accurate sustainability reporting starts with clean, structured data. For companies, mistakes in emissions reporting can result in both regulatory risks and growing stakeholder scrutiny. While domain knowledge remains crucial, planning and implementing a data-driven strategy is the key to long-term success. Ultimately, a well-designed sustainability data system can transform reporting from a compliance burden into a strategic asset, providing companies with valuable insights into their operations and sustainability performance.

A Simple Approach to Master Data Management to Unify Metrics and Insights

2025-01-14T00:00:00Z

Master data management (MDM) often sparks debate. It’s a frequently proposed budget item that rarely gets approved, yet it’s a fundamental reason business leaders struggle to achieve granular and unified metrics. This article aims to demystify MDM by showing that it doesn’t always have to involve big budgets or complex tools. Instead, we will define master data and explore two straightforward approaches to MDM that can be implemented without lengthy multi-year technology roadmaps. While the complexity of MDM varies by case, we believe most organizations can achieve significant improvements by following a few practical steps.

What is Master Data Management?

Master data management is the process of creating and maintaining a single source of truth for critical business data. It ensures consistency, accuracy, and accessibility across an organization’s operations. By harmonizing data, MDM eliminates discrepancies, supports better reporting, and improves decision-making processes.

The most commonly used dimensions in manufacturing company metrics include:

Product Hierarchies
Territory Hierarchies
Key Account Management Groupings
Industry Codes
Legal Entities
Regions and Countries
Fiscal Time Dimensions

Typical pitfalls of master data management projects

Many master data management projects fail to achieve their intended goals. In some cases, these projects never even get off the ground because they fail to clearly communicate their value to business leaders. Based on our observations, there are three common pitfalls when companies approach master data:

Boiling the Ocean: Attempting to solve all data problems at once instead of prioritizing critical areas often leads to overwhelming complexity and project delays.
Overemphasis on Tools: Focusing too much on selecting and implementing expensive tools can distract from addressing foundational issues like data quality and governance.
Neglecting Data Ownership: Failing to establish clear accountability for different aspects of master data leads to fragmented efforts and inconsistent results. For example, defining which teams are responsible for maintaining specific data dimensions is often overlooked but important for success.

Avoiding pitfalls in data management projects

To avoid common pitfalls in data management, it is essential to break down the project into two distinct approaches: a collaborative approach and a consolidated approach. While this framework is not our own invention, it has been a reliable methodology we have employed successfully for many years.

By dividing master data management work into these two approaches, organizations can tackle tasks more effectively. This structured method allows for a clearer focus on priorities and better resource allocation. Moreover, starting with small, incremental steps—rather than attempting a large, comprehensive project—is often a more practical and achievable way to ensure success in master data management.

The Collaborative Master Data Approach

The collaborative approach to MDM is ideal for managing lookup data and creating custom groupings. This method relies on an application with a user-friendly interface that enables teams to collaborate effectively on master data management tasks. Key features include:

Approval Workflows: Changes to lookups and master data are managed through structured approval processes, ensuring accuracy and accountability.
Real-Time Collaboration: Teams can work together to refine and update master data in real-time, promoting consistency across departments and improving alignment.

Advantages

Enhanced Flexibility: This approach is particularly beneficial for organizations that need adaptable and active user involvement in managing dimensions.
Simplicity: Many teams already utilize this approach informally by maintaining lookup tables in spreadsheets. At its core, this method can be as simple as sharing a spreadsheet via tools like SharePoint.

Disadvantages

Increased Manual Effort: This approach requires manual intervention, including manual approval processes, which means at least two users must be involved in managing the master data.
Reduced Auditability: If changes are not properly tracked—such as when data is stored in untracked spreadsheets—it becomes challenging to trace historical changes and reconstruct how master data appeared at earlier points in time.

This type of setup has proven effective for defining dimensions critical to measuring business unit or sales performance. For instance, teams managing sales territories have achieved significant success by aligning their territory dimension reporting with their sales organization structures, ensuring consistency with sales incentive plan reporting.

The Consolidated Master Data Approach

In contrast, the consolidated approach is tailored for application data that is maintained directly at the source. This method involves aggregating data from multiple applications into a central repository without making any modifications. Its key characteristics include:

Advantages:

Data Integrity Preserved: Data remains unchanged in its original source, ensuring that users can only modify it within the application of origin. This preserves the accuracy and reliability of the master data.
Reduced Manual Intervention: By consolidating data from trusted sources, this approach minimizes the need for manual adjustments, thereby reducing errors and enhancing efficiency.

Disadvantages:

Complex Decentralization: Because master data is distributed across multiple applications, coordinating updates and maintaining consistency can be challenging. This issue is particularly acute in organizations with multiple ERP systems managed by diverse local IT teams.
Restricted Modifications: Business leaders often require minor adjustments or recoding of dimensions. In this approach, such changes must be negotiated with the production system of record, which can slow down responsiveness and adaptability.

The consolidated approach is most effective when source data from applications is of high quality and does not require frequent adjustments by users. It is ideal for organizations seeking streamlined operations where data reliability and minimal intervention are priorities.

The Hybrid Approach: Combining both the Collaborative and Consolidated Approach

Considering that both collaborative and consolidated master data approaches have their own unique advantages and disadvantages, a hybrid strategy can often be the most effective. By using these approaches together:

Maintain Complex Master Data Within Applications: Organizations can retain more complex master data within applications, potentially avoiding the need for a costly and intricate MDM system altogether.
Enable Business Flexibility: Business leaders gain the flexibility to add their own local dimensions, improving the visibility of custom dimensions in reporting and enhancing auditability and change management processes.
Clarify Ownership Roles: A hybrid approach facilitates clear ownership of master data. For example, Customer data quality in the ERP can be managed by the Customer Service team, while the Sales organization takes responsibility for Sales Territory structures, ensuring accountability and streamlined operations.

If You Want Real Change in Master Data Quality, You Need Visibility

Once, I was in a meeting with the CEO of a Business Unit discussing the sales funnel. The conversation was not going well because the data was, quite simply, poor. However, the CEO said that he would be willing to continue reviewing the sales funnel reports even if the data was flawed, as long as it was clear which data was problematic and why.

The CEO made a smart move here. Instead of insisting that everything be flawless, he told his leadership team that he was willing to work with imperfect data, provided it was clear why the data was flawed and who was responsible for addressing the issues.

Data quality metrics might seem like a dull report that no business leader would bother opening. However, imagine a business report on key metrics like orders, sales, or inventory that includes a data quality score. Such a report would undoubtedly capture attention. This kind of visibility not only ensures the report gets reviewed but also drives accountability. It motivates people to address data issues, whether in the source system (consolidated master data) or within the application (collaborative master data).

Conclusion

Master data management plays a pivotal role in unifying metrics and insights for informed decision-making. By understanding the collaborative and consolidated approaches, organizations can select the strategy that best aligns with their needs and data quality standards. Whether fostering collaboration for lookup data or consolidating application data for streamlined efficiency, effective MDM can transform how businesses leverage their data assets.

What As a Data Mart? And What Challenges Does It Solve?

2025-01-02T00:00:00Z

Business leaders are always under constant pressure to improve operational efficiency, maintain product quality, and optimize financial performance. Good data and, more importantly, good metrics are important for effectively addressing these challenges.

A critical yet often underutilized tool for achieving reliable business metrics is the data mart. In this article, we will explore what a data mart is and how it functions. We will also define what is a “good metric” and examine how data marts enable business leaders with metrics that enhance visibility, control, and predictability—ultimately improving business performance.

What Is a Data Mart?

A data mart is a specialized data repository typically stored in a SQL database designed to serve the specific needs of a particular department, business unit, or function. It contains data relevant to its intended audience, structured to provide streamlined access to insights.

Key characteristics of a data mart include:

Focus: Designed to support specific business areas, such as top-line (orders, sales, backlog, standard cost & margins), purchasing (supplier performance, material productivity), inventory (aging, excess & obsolescence), and others.
Efficiency: Simplifies access to key data, reducing the time needed to generate insights. This efficiency stems from having all the logic centralized in one place rather than spread across various reports and dashboards.
Usability: Usability is high because standard connections allow users to utilize familiar reporting tools like PowerBI, Tableau, or Excel. This ensures accessibility, making it easy to query data and present it in user-friendly formats.

By organizing data into easily digestible segments, data marts bridge the gap between raw data and actionable metrics, enabling business leaders to make informed decisions faster.

This makes it challenging to benchmark typical business-as-usual travel costs and determine how much can be cut without negatively impacting operations.

Not all metrics are created equal. A good metric stands out by its ability to provide meaningful, actionable insights that align with organizational goals. To evaluate the quality of your metrics, consider the following attributes:

Compliant: The metric adheres to industry standards and internal governance policies, ensuring accuracy and reliability. This is especially important for calculations affecting key financial figures such as profitability, working capital, and reserves, where adherence to internal policies ensures consistency and trustworthiness.
Usable: It is accessible, easy to interpret, and designed for practical application by its intended audience.
Granular: The metric provides detailed information, enabling deep drill-down analysis to uncover root causes and trends. Achieving granularity is challenging but essential, as it enhances the quality of insights and directly influences decision-making through accurate explanations and narratives.
Aligned: It supports strategic objectives and connects to critical financial or operational outcomes like the P&L, balance sheet, or product quality. All stakeholders agree on the metric’s calculation and meaning, ensuring organizational clarity and alignment.
Comparable: A good metric is consistent across contexts and systems, enabling meaningful comparisons between departments, time periods, or benchmarks. This involves tracking events like organizational changes, sales territory changes, standard cost updates, and currency fluctuations. When changes occur, metrics should be easily reinstated, and different versions of the same metric should remain accessible for comparison across changes.

To illustrate the impact of missing metric attributes on business performance, let’s consider a case study. Imagine a scenario where a CEO asks the CFO to temporarily reduce travel expenses to meet profitability targets. Now, suppose the CFO has to work with a Travel Expense metric lacking the five key attributes. Here’s what happens:

Non-Compliant: Travel expenses are incorrectly booked in cost categories, making it difficult to calculate the cost savings that could be generated.
Unusable: Details like hotel, flight, and car rental bookings are locked in the travel application, requiring weeks for analysts to compile raw data for a detailed analysis.
Lacking Granularity: Without granular insights, such as travel by department or business function, it is impossible to quickly identify cost cuts without risking business operations.
Misaligned: With travel expenses not booked in the correct cost categories, business leaders across functions will disagree on the actual figures, wasting time aligning on a baseline.
Non-Comparable: Because travel expenses are not easily available, year-over-year analysis becomes difficult. This makes it challenging to benchmark typical business-as-usual travel costs and determine how much can be cut without negatively impacting operations.

Metrics that embody these attributes not only enhance decision-making but also build trust among stakeholders by providing a clear, auditable trail from raw data to actionable insights.

How Data Marts Enable Business Leaders with Good Metrics

Manufacturing companies operate in complex environments where good metrics are critical for visibility, control, and success. Data marts make good metrics possible through five essential activities:

Centralized flat tables stored in SQL databases: Data marts are ready to use and provide a simplified format for querying data. These tables streamline access to key metrics by organizing raw data into structured and understandable formats, ensuring faster and more reliable insights for decision-makers.
Centralized Metric Calculation Logic: Data marts centralize the logic behind metric calculations, ensuring consistency and reducing errors.
Automated Processing: Data pipelines powering data marts fully automate the end-to-end transformation process, turning raw data into trusted and actionable metrics.
Integration of Data Sources: Data marts consolidate data from multiple sources, both legacy and modern, into single, auditable metrics.
Monitored Ingestions and Transformations: Data pipelines powering data marts are fully monitored to ensure that the metrics business leaders rely on are always available. When issues arise, effective communication is promptly initiated, and resolutions are implemented quickly to minimize disruptions.

These capabilities make data marts indispensable for driving performance improvements and informed decision-making in manufacturing operations.

Common Pitfalls in Achieving Reliable Metrics

Data has become popular in business, placing significant pressure on data marts and the traditional business intelligence profession. Many so-called data professionals promise quick ROI through complex data science, AI, and ML projects or by promoting revolutionary new technologies often portrayed as miraculous solutions.

To avoid costly missteps and wasted resources on data initiatives that fail to deliver, it is important to recognize how the temptation of quick fixes and over-promised results can mislead companies. Below are common pitfalls to steer clear of when seeking the insights and metrics your business needs:

Excessive Focus on Tools and Technology: While focusing on tools and technology is not inherently bad, it becomes problematic when attention shifts to unattainable promises tied to “magical” solutions. Good data and insights come from strong processes and robust stakeholder alignment—essential foundational work. Moreover, tools are increasingly standardized, so as long as the fundamentals are solid, selecting one tool over another usually won’t yield vastly different results (unless you’re Google).

The One ERP Strategy: This strategy, often proposed by IT leaders in manufacturing with extensive ERP experience, is no longer effective. A single ERP system does not guarantee centralized or unified metrics. Non-ERP data, such as CRM data, has become equally critical and must be included. Furthermore, mergers and acquisitions often result in multiple ERP systems, meaning “one ERP” rarely remains as singular as promised.

Putting Complex Data Modeling in the ERP: Often proposed by IT leaders, this strategy is not always a straightforward decision. In some cases, performing data modeling in the ERP makes sense, but in others, it does not. Customizing and updating data marts is now faster and more cost-effective than modifying ERPs. If your organization operates multiple ERPs, it might be more practical to move certain calculations, including key ones such as accounting reserve calculations, to the data marts layer.

Expecting ROI from All Data Initiatives: This expectation often stems from a misunderstanding of the diverse aspects of data work. Not all data projects are about driving revolutionary business changes; many are essential for day-to-day operations. Without solid data marts, business functions may create their own versions of metrics, often leading to bad practices and inefficiencies, and this is also a good way for numbers to never match properly.

As we have seen, avoiding these pitfalls is important for building a reliable data and metrics ecosystem that supports business leaders. Recognizing these challenges allows organizations to focus on sustainable strategies that drive long-term success.

Conclusion

Data marts provide powerful technology and best practices for delivering “good metrics” that are compliant, usable, granular, aligned, and comparable. These metrics not only improve visibility and control but also enhance predictability—key drivers of business performance.

Integrating data marts into your business performance analytics strategy accelerates access to insights, enabling faster decision-making, more efficient operations, and improved business performance.

Faster Metrics with Data Marts (overcoming Data Warehouses challenges)

2024-12-23T00:00:00Z

The growing number of tools and solutions in the data ecosystem can be overwhelming for business leaders. Some data solutions are so advanced that they resemble full-scale “digital transformation” initiatives—a term often used by IT professionals.

However, one area of data analytics has remained largely unchanged for the past 30 years: business intelligence. Companies once relied on robust data warehouses with well-defined structures paired with straightforward reporting tools that efficiently extracted insights.

Over time, data warehouses began to fade from prominence. This decline was driven by:

advancements in data platforms
shifting focus among analytics professionals toward various data ROI initiatives
unresolved internal challenges that persisted throughout their 30-year history.

But data warehousing played a critical role in organizations by supporting unified and standardized metrics. Today, this function is largely neglected, leaving businesses to grapple with its absence. As a result, leaders are increasingly witnessing the impact on performance, compliance, and control, compounded by challenges in effective monitoring.

Data Warehousing Challenges: A Double-Edged Sword

Rigid and Slow

Data warehouses became the definition of slow and rigid IT. Change requests often took months—or even years—to implement, as BI teams resisted any changes that might disrupt production systems. This inflexibility drove business functions to develop their own localized data warehouses, typically ignoring best practices, total cost of ownership (TCO), and accumulating significant technical debt.

Poor Domain Understanding

As data warehouses were centralized within IT departments, they lost connection to business domains. IT teams, by nature, lacked deep business understanding, leading to a disconnect. IT expected well-defined, scoped requirements to execute in a waterfall approach, while business professionals anticipated proactive insights and solutions from the “data people.”

Black Boxes: The Struggle with Compliance

As IT departments took control of data modeling in data warehouses, business professionals began questioning the accuracy and reliability of the metrics they consumed. Compliance with policies and metric calculations became a significant concern: If I can’t understand how a metric is calculated, how can I trust it?

Multiple Sources and the “One ERP” Myth

In the early days of data warehousing, the application landscape was relatively simple, with ERP systems dominating as the core business application. Data warehouses were often designed as extensions of these ERP systems.

Over time, two major shifts occurred. First, mergers and acquisitions introduced multiple ERPs into the ecosystem. Second, new applications—such as CRMs, HRMs, and MES systems—entered the mix, further complicating the landscape. IT departments attempted to address these challenges by pushing for a single ERP implementation across the organization. Meanwhile, business leaders grew frustrated, questioning how many years it would take to answer even basic questions like, “How many employees do we have?”

Are Data Marts The Way Forward?

At first glance, a data mart might seem like a minor evolution from traditional Business Intelligence (BI) run by IT:

A data mart is a subset of a data warehouse, focusing on a specific business area or department. It contains a curated set of data tailored to the needs of that group, making it easier and faster to access insights without navigating the entire data warehouse. This closely mirrors the structure of a typical data warehouse OLAP cube, leading to the impression that introducing data marts is just a rebranding exercise.

However, this view overlooks the fundamental shift brought about by data marts, which are driven by a new set of principles. These principles can be traced back to modern data platform architectures, which themselves evolved from the Big Data movement (remember Hadoop?). The three key principles that distinguish data marts today are:

Transparency: Data models should be open and accessible to analysts outside of IT BI teams.
Flexibility: Data models should be easy to enhance, modify, and update.
Version Control: Data models should be version-controlled in code, similar to software development practices.

The challenge today is that these principles are not universally adopted across the industry. Depending on who you speak to, you’ll encounter different perspectives on how data marts are implemented and leveraged.

OK, but what is a data mart?

Technically, a data mart is simply a flat table within a SQL database. This table is usually accessed through reporting tools like Power BI, Tableau, or Cognos. Analysts can connect to the table and create typical BI reports, and as the data is updated, the connected reporting tools automatically reflect these changes.

In a modern data platform, the data mart is developed using SQL code within the modeling layer. This code is typically hosted in a code repository, such as GitHub or Bitbucket. By storing the code in a repository, the data mart becomes transparent, easy to contribute to, and aligns with the three principles outlined earlier.

Data Marts: How They Resolve Data Warehouse Challenges

Now, let’s explore how data marts, with their transparency, flexibility, and version control principles, help address the challenges often encountered in traditional data warehouse systems.

Rigid and Slow

Data marts are faster to develop because they allow contributions from multiple teams. By using SQL models stored in a code repository, teams can collaborate and contribute to the code. This eliminates the bottleneck caused by a single centralized team handling all the work.

Poor Domain Understanding

With data marts, teams outside of IT can actively contribute to model creation and data mart definition, bridging the gap caused by limited business knowledge. When paired with good governance, this approach brings data modeling closer to the business, enhancing domain understanding.

Black Boxes: The Struggle with Compliance

Since data mart code is stored in verified and recognized repositories, it becomes fully transparent, addressing the issue of “black boxes” that only a few individuals understand. Additionally, implementing data catalogs can further improve transparency by providing metadata and data lineage, making these insights accessible even to less technical users.

Multiple Sources and the “One ERP” Myth

As part of modern data platforms, data marts can handle significantly larger datasets. This makes ingesting data from multiple applications cost-effective, allowing integration from various systems into a single, well-defined data mart through proper data modeling.

Conclusion

Data marts represent a modern solution to many of the challenges faced by traditional data warehousing systems. By embracing transparency, flexibility, and version control, data marts empower teams across the organization to collaborate and contribute to data modeling, breaking down silos and reducing bottlenecks.

With their ability to bring data modeling closer to the business, data marts enhance domain understanding and improve compliance through transparency and clear data lineage. Moreover, the cost-effectiveness of integrating multiple data sources in a single data mart aligns with the demands of today’s complex data ecosystems.

Incorporating data marts into your data strategy can help businesses move beyond the limitations of legacy systems, enabling faster decision-making, better insights, and more agile operations.