Unit III – Data Provisioning & Data Visualization

1. Explain the concepts of ETL Architecture, Extraction, Transformation, and Loading

(ETL) in the context of Business Intelligence. [Both Papers]

Introduction to ETL in Business Intelligence:

ETL stands for Extract, Transform, and Load. It is a critical process in Business Intelligence (BI) used to
integrate data from multiple sources into a data warehouse for reporting, analysis, and decision-making.

ETL Architecture Components:

1. Data Sources:
o These are the original locations of data such as relational databases (MySQL, Oracle), flat
files (CSV, Excel), cloud storage (AWS S3), or APIs.
2. ETL Layer:
o This is the heart of the architecture. It includes three main operations:
▪ Extraction: Fetching raw data from different sources.
▪ Transformation: Cleaning, structuring, and converting data into a suitable format.
▪ Loading: Pushing the transformed data into the target system (data warehouse).
3. Staging Area:
o A temporary storage where data is held after extraction and before transformation. It helps in
data validation, sorting, and error handling.
4. Data Warehouse:
o The final destination where cleaned and transformed data is stored. It supports
multidimensional analysis and OLAP operations.
5. BI Layer:
o Tools like Power BI, Tableau, or SAP BO connect to the data warehouse to create
dashboards, reports, and visualizations.

ETL Process in Detail:

1. Extraction:
o This step involves identifying and collecting data from multiple sources.
o Data can be structured (SQL databases) or unstructured (log files, social media).
o Extraction must ensure minimal impact on source systems and support batch or real-time
collection.
2. Transformation:
o This step involves cleaning and converting data into a standardized format.
o Operations include:
▪ Data cleansing (removing duplicates, handling nulls)
▪ Data integration (merging data from various sources)
▪ Data aggregation (summarizing data for analysis)
▪ Business rules application (e.g., converting currency)
3. Loading:
 o The final step where transformed data is loaded into the data warehouse.
o It can be done in:
▪ Batch Mode (scheduled, periodic)
▪ Real-Time Mode (continuous updates)

Benefits in BI:

• Centralized data for analysis.

• Improved data quality and consistency.
• Supports historical analysis and trends.
• Faster decision-making through integrated dashboards.

Diagram of ETL Architecture (Textual View):

[ Data Sources ]
↓
[ Extraction ]
↓
[ Staging Area ]
↓
[ Transformation ]
↓
[ Loading ]
↓
[ Data Warehouse ]
↓
[ BI Reporting Tools ]

Conclusion:

ETL is the backbone of BI systems. It ensures that raw data from diverse sources is converted into
meaningful insights, enabling organizations to make data-driven decisions effectively.

2. Differentiate between Initial and Incremental Loading in ETL Processes. [Nov 2024]

Introduction:
In ETL (Extract, Transform, Load) processes, data loading is the phase where data is inserted into a data
warehouse or data mart. There are two primary types of loading strategies:
• Initial Load
• Incremental Load
Each serves a different purpose based on the data scenario.

Comparison Table: Initial vs. Incremental Loading

 Initial Load Incremental Load
Feature / Aspect

The process of loading the entire dataset The process of loading only new or changed
Definition
for the first time. data since the last load.

Used during the first-time data Used for periodic updates (daily, hourly, real-
Usage Scenario
warehouse setup or a full reload. time).

High volume — loads complete Low volume — loads only deltas

Data Volume
historical data. (new/changed records).

Performance
Time-consuming and resource-intensive. Fast and efficient with less system load.
Impact

Risk of Yes — if change tracking is not implemented

No — entire data is replaced freshly.
Duplication correctly.

Full data load via bulk insert or batch Change Data Capture (CDC), Timestamps, or
Techniques Used
process. Audit Columns.

Detailed Explanation:
1. Initial Load:
• It is performed once when a data warehouse or a table is created.
• All existing data is extracted from the source, transformed, and loaded completely.
• Used when:
o A new system is being developed.
o There is a need for full reprocessing due to corruption or design changes.
Example: Loading 5 years of historical sales data into a newly built warehouse.

2. Incremental Load:
• This method loads only the data that has changed since the last ETL run.
• It improves performance, reduces network traffic, and keeps data up to date.
• Implemented using:
o Timestamps (e.g., last_modified_date)
o Log tables
o Change Data Capture (CDC) tools like Oracle GoldenGate, SQL Server CDC, Debezium.
Example: Loading only today's new sales records into the sales fact table each night.
Conclusion:
• Initial Loading is foundational for setting up systems.
• Incremental Loading ensures timely and efficient updates.
• A well-designed ETL system typically uses both: initial load first, followed by incremental updates.

3. Analyse the Role of Lookups in the Transformation Process of ETL. [May 2024]

Introduction:
In ETL (Extract, Transform, Load), the Transformation phase often involves Lookups to enrich or validate
data. Lookups are crucial when data from the source needs to be matched, referenced, or translated using
values from another table or dataset.

What is a Lookup in ETL?

A Lookup is a transformation technique used to retrieve related data from a reference dataset based on a
common key.
Example: Matching a customer’s region code to a table of region names to get the full region name.

Key Roles of Lookups in Transformation:

1. Data Enrichment:
o Enhances raw data with additional information from related tables.
o E.g., Enriching an order record with customer details using customer ID.
2. Data Validation:
o Ensures that the incoming data is valid and consistent.
o E.g., Verifying that product codes exist in the master product table.
3. Foreign Key Mapping:
o Ensures referential integrity when loading into dimension and fact tables.
o E.g., Mapping state names to state IDs from the dimension table.
4. Code Translation:
o Converts codes into human-readable formats.
o E.g., Translating "M" and "F" into "Male" and "Female".
5. Slowly Changing Dimensions (SCDs):
o Lookups are used to check if a dimension record has changed and whether a new version
should be created (SCD Type 2).
 6. Joining with Reference Data:
o Similar to SQL joins (LEFT JOIN or INNER JOIN), used during transformation to combine
rows from different sources.

Types of Lookup Implementations:

Type Description

Static Lookup The reference data does not change during ETL runtime.

Dynamic Lookup The reference data can change during ETL; updated data is considered.

Cached Lookup Reference data is pre-loaded into memory for faster access.

Uncached Lookup Accesses the reference table row-by-row; slower but always current.

Diagram (Text View):

[Source Data]
|
[ Transformation Layer ]
|
[Lookup Table (e.g., Customer Master)]
↓
[Enriched Transformed Data]
↓
[Data Warehouse]

Example Scenario:
• Source Data: Order table with Customer_ID
• Lookup Table: Customer Master with Customer_ID, Customer_Name, Region
• Purpose: Add Customer_Name and Region to the order record before loading into warehouse

Conclusion:
Lookups are a powerful mechanism in ETL transformation. They enable data enhancement, maintain
consistency, and support complex data relationships — all of which are essential for accurate and
meaningful business intelligence reporting.
4. Develop a Plan for Data Quality Assessment and Profiling in the Context of a Specific
Business Dataset. [Nov 2024]

Scenario:

Let’s assume a business dataset from an e-commerce company’s customer data, which includes:

• Customer_ID
• Name
• Email
• Phone_Number
• Gender
• Date_of_Registration
• Region
• Total_Purchases

The objective is to assess and improve data quality to ensure reliable BI reports and personalized customer
insights.

Step-by-Step Plan for Data Quality Assessment & Profiling:

1. Define Data Quality Dimensions

Key dimensions to evaluate:

Dimension Description
Accuracy Is the data correct and valid? (e.g., correct phone numbers)
Completeness Are all required fields filled in?
Uniqueness Are there duplicate records?
Consistency Is data uniform across records and systems?
Timeliness Is the data up-to-date?
Validity Does the data match predefined formats and rules?

2. Perform Data Profiling

Use profiling tools (e.g., Talend, Informatica, Pandas Profiling in Python) to perform:

Profiling Type Description

Column Profiling Analyze individual fields (nulls, min/max, patterns)
Profiling Type Description
Cross-Column Profiling Check inter-column dependencies and rules
Data Type Analysis Ensure correct data types (e.g., date, integer)
Pattern Matching Validate email formats, phone patterns, etc.

Example Checks:

• Are all emails in proper format?

• Are Customer_IDs unique?
• Are there missing values in Phone_Number or Region?

3. Identify Data Quality Issues

From profiling, extract problems like:

• 7% missing Phone_Number
• 3% duplicate Customer_IDs
• Inconsistent Gender values like “M”, “Male”, “male”, “F”

4. Define Cleansing Rules

Establish rules to fix the problems:

• Standardize gender values to “Male” and “Female”

• Drop or merge duplicates using business logic
• Fill missing regions with "Unknown" or use other reference tables

5. Implement Data Quality Checks in ETL Pipeline

• Add validation rules in the Transformation step

• Use data quality dashboards to track completeness and errors
• Schedule periodic profiling to monitor quality over time

6. Use Tools and Frameworks

Tool Use
Talend DQ GUI-based quality profiling and cleansing
Python (Pandas) Custom profiling, regex validations, and cleaning
SQL Queries Rule-based validation and duplicate checks
Great Expectations Create unit tests for data in pipelines
7. Document and Report Findings

• Prepare a Data Quality Report showing:

o Percentage of completeness
o Number of duplicates removed
o Fields corrected or standardized
o Issues flagged for manual review

Conclusion:

A structured Data Quality Assessment Plan ensures clean, consistent, and reliable data, which is crucial
for accurate BI insights, compliance, and operational decisions.

5. Describe the Key Components of Data Provisioning, Including Data Quality, Data
Profiling, and Data Enrichment. [May 2024]

Introduction:

Data Provisioning is the process of making relevant data available to systems, applications, or users in a
structured, secure, and timely manner. It plays a foundational role in Business Intelligence (BI) and
analytics, ensuring that clean, complete, and context-rich data reaches the end user.

Key Components of Data Provisioning:

1. Data Quality

Definition:
Ensures that the data provided is accurate, complete, consistent, and valid.

Key Activities:

• Validation rules (e.g., format checks, range checks)

• De-duplication
• Standardization (e.g., date formats, naming conventions)
• Error detection and correction

Example:
Fixing email format errors or ensuring all phone numbers have 10 digits before provisioning data to a CRM
system.

2. Data Profiling
Definition:
The process of examining source data to understand its structure, content, relationships, and quality
issues before it is moved or used.

Key Activities:

• Analyzing null values, distinct values, patterns, and frequency

• Finding data anomalies, such as outliers or invalid entries
• Ensuring consistency between interrelated columns

Example:
Discovering that 25% of “Region” values are missing or that “Customer_ID” has duplicate values before
provisioning the data.

3. Data Enrichment

Definition:
The process of enhancing the dataset by adding new, useful information from internal or external sources.

Key Activities:

• Adding geolocation info based on postal codes

• Mapping gender from names
• Pulling demographics or social data from third-party APIs

Example:
Adding the customer’s income range and age group to a marketing dataset to improve personalization and
segmentation.

4. Data Extraction, Transformation, and Loading (ETL)

Definition:
The technical core of provisioning – data is:

• Extracted from source systems

• Transformed into the required structure/format
• Loaded into a data warehouse or system

Example:
Pulling sales data from SAP, cleaning and aggregating it, then loading it into Power BI for analysis.

5. Metadata Management

Definition:
Managing information about the data (data about data), including source, lineage, format, and
transformations applied.
Importance:

• Improves data governance

• Helps trace data back to its source for audit or debugging

6. Data Security & Access Control

Definition:
Ensuring that data is provisioned securely and only accessible to authorized users/systems.

Key Methods:

• Role-based access control

• Data masking
• Encryption during transmission

Conclusion:

To provision data effectively for any BI system, it is essential to:

• Assess and improve data quality

• Perform thorough data profiling
• Add value through data enrichment

These three components—when supported by strong ETL, metadata, and security practices—ensure that the
right data reaches the right people in the right form.

6. Evaluate the Impact of Data Duplication on the Overall Performance and Efficiency of a
Data Warehouse. [May 2024]

Introduction:

Data duplication refers to the presence of multiple identical or near-identical records in a data warehouse. It
is often a result of inconsistent data entry, lack of validation, or flaws in the ETL process. While it may seem
harmless at a glance, duplication negatively impacts performance, accuracy, and cost in data
warehousing and business intelligence.

Negative Impacts of Data Duplication:

1. Increased Storage Costs

 • Duplicate records occupy unnecessary disk space.
• For large-scale data warehouses, even a small percentage of duplicates can significantly inflate
storage requirements.
• Leads to higher infrastructure costs and longer backup/recovery times.

Example:
A sales table with 2 million records where 10% are duplicates leads to 200,000 redundant entries.

2. Slower Query Performance

• Larger datasets due to duplicates cause slower data retrieval.

• Indexes become less efficient, leading to longer scan times and CPU load during analytical queries.

Example:
A BI dashboard filtering customer orders may take seconds longer to load because of bloated duplicate
entries.

3. Inaccurate Business Insights

• Aggregated metrics like totals, averages, counts become misleading.

• Business decisions based on such data are flawed, leading to financial or strategic errors.

Example:
Duplicate sales records may inflate revenue reports, leading to incorrect forecasting or budget planning.

4. Data Integrity and Trust Issues

• Stakeholders may lose trust in the system if reports contain inconsistencies or incorrect totals.
• Data governance suffers as analysts constantly question data quality.

Example:
Two identical customer records receiving different marketing messages creates confusion and damages
brand trust.

5. Increased ETL Processing Time

• ETL jobs take longer due to increased volume and complex cleansing logic to detect/remove
duplicates.
• Transformation and validation scripts become more complex, raising maintenance burden.

6. Complications in Master Data Management (MDM)

 • Duplicates make it harder to maintain a single source of truth for key entities (e.g., customer,
product).
• MDM systems may fail to match or link records correctly, resulting in data silos.

Mitigation Strategies:

Method Description
Data Deduplication in ETL Use transformation logic to identify and remove duplicates
Primary Key Constraints Enforce uniqueness at the database level
Data Validation Rules Validate incoming records for existing keys or hashes
Master Data Management (MDM) Use identity resolution algorithms to unify records
Data Profiling Tools Detect duplicates before loading them into the warehouse

Conclusion:

Data duplication significantly degrades the performance, accuracy, and cost-efficiency of a data
warehouse. Proactively detecting and removing duplicates ensures reliable analytics, faster performance,
and greater business trust in the data.

7. Explain the Concept of Change Data Capture and Its Significance in Data Provisioning.
[May 2024]

What is Change Data Capture (CDC)?

Change Data Capture (CDC) is a technique used to identify and capture only the data that has changed
(inserted, updated, or deleted) in a source system and apply those changes to a target system such as a data
warehouse.

Instead of moving entire datasets repeatedly, CDC helps in transferring just the incremental changes,
ensuring efficiency, real-time updates, and data consistency.

Key Concepts of CDC:

1. Types of Changes Captured:

• Insert – New records added to the source.

• Update – Existing records modified.
• Delete – Records removed from the source.
2. Common CDC Techniques:

Technique Description
Timestamp Columns Use last_modified timestamp to filter recent changes.
Database Triggers Triggers log every data change in audit or history tables.
Log-Based CDC Monitors database transaction logs to identify changes. Most efficient.
Snapshot Comparison Compare current and previous dataset snapshots to find changes.

Significance of CDC in Data Provisioning:

1. Improves Performance and Efficiency

• By only loading the changes, CDC reduces the volume of data to be transferred and processed.
• Helps in real-time or near-real-time updates to data warehouses or BI tools.

2. Reduces ETL Load Times

• Full data loads are time-consuming and resource-heavy.

• CDC allows faster, smaller, and smarter data loads during ETL.

3. Maintains Data Consistency

• Ensures that the target system is always synchronized with the source.
• Reduces risk of outdated or stale data in reports and dashboards.

4. Enables Real-Time Analytics

• Vital for real-time dashboards, alerts, and data-driven decision-making.

• Commonly used in e-commerce, banking, and IoT environments.

5. Supports Auditing and Compliance

• CDC can maintain history of changes, useful for audit trails, GDPR compliance, etc.

6. Minimizes System Downtime

• Since CDC processes only incremental data, production systems are not heavily burdened.
• Ideal for high-availability environments where downtime is costly.
Example Scenario:

A retail company updates product prices daily. Instead of reloading the entire products table with
thousands of rows, CDC captures only the 50 updated products and updates them in the data warehouse.
This saves time, reduces system load, and keeps BI dashboards current.

Conclusion:

Change Data Capture is a crucial element in modern data provisioning, enabling efficient, real-time, and
low-latency data movement. It enhances performance, supports real-time analytics, and ensures data
accuracy across systems—making it a cornerstone for scalable Business Intelligence solutions.

8. Create a Simplified Data Mart and Explain Its Purpose in a BI System. [Both Papers]

What is a Data Mart?

A data mart is a subset of a data warehouse that is focused on a specific business area, department, or
subject (like sales, finance, HR, etc.). It contains summarized and relevant data that allows business users
to access and analyze information quickly without needing to scan through the entire data warehouse.

Simplified Example of a Data Mart: "Sales Data Mart"

Purpose: To provide insights into sales performance, customer orders, and revenue for decision-
making by the Sales Department.

Structure of the Sales Data Mart:

1. Fact Table: Fact_Sales

Field Description
sale_id Unique sale identifier
product_id Links to Product dimension
customer_id Links to Customer dimension
store_id Links to Store dimension
date_id Links to Date dimension
quantity_sold Number of items sold
 Field Description
total_amount Sale amount

2. Dimension Tables:

• Dim_Product
o product_id, product_name, category, price
• Dim_Customer
o customer_id, customer_name, location, loyalty_status
• Dim_Store
o store_id, store_location, store_manager
• Dim_Date
o date_id, date, month, quarter, year

Purpose of the Data Mart in BI:

1. Faster Query Performance

• Because it contains a focused subset of data, queries run faster than on a full-scale data warehouse.

2. Department-Specific Analytics

• Tailored to a specific business function (e.g., Sales), enabling custom dashboards and reports.

3. Easier Access for Business Users

• Business teams can explore and analyze data without technical help or deep SQL knowledge.

4. Supports Decision Making

• Dashboards created using the Sales Data Mart may show:

o Best-selling products
o Sales trends over time
o Region-wise performance
o Top customers by revenue

5. Reduces Load on Central Data Warehouse

• Analysts work on smaller datasets which lowers query load on the main data warehouse.
Conclusion:

A simplified data mart like a "Sales Data Mart" serves as a powerful, domain-specific data source for
business intelligence applications. It enables faster, more relevant, and more accessible data analysis,
empowering departments to make timely and informed decisions.

9. Compare and Contrast the Data Provisioning Methods of Different BI Tools such as
Tableau, Power BI, and Dundas BI. [Nov 2024]

Introduction:

Data provisioning in BI tools refers to the methods and processes used to connect, extract, transform,
and load data into the BI environment for analysis and visualization. Different BI tools offer varied
capabilities and approaches based on integration, performance, flexibility, and user interface.

Comparison Table:

Feature / Tool Tableau Power BI Dundas BI

Data Wide range: SQL, Excel, Strong MS ecosystem: Excel, Flexible connectors:
Connectivity cloud, APIs SQL, Azure, etc. SQL, Excel, REST, etc.
Tableau Prep (separate Integrated ETL &
Data Preparation Built-in Power Query Editor
tool) scripting engine
Live vs Import Both supported: live & in- Both supported: Import (default), Both supported with
Mode memory extracts DirectQuery advanced caching
Real-Time Data Yes (via DirectQuery/streaming Yes (streaming and
Yes (via live connections)
Support datasets) scheduled refresh)
Limited – focuses on Strong modeling (relationships, Moderate – supports
Data Modeling
visualization DAX, etc.) calculated elements
Basic in Prep; complex Built-in ETL with C#
ETL Capability Robust ETL in Power Query
needs external tools scripting if needed
Role-based access via Row-level security and Azure Fine-grained security
Data Governance
Tableau Server integration built into platform
Visual, intuitive for Drag-drop, Excel-like for More developer-centric,
Ease of Use
analysts business users technical users
Tableau Cloud, AWS, Tight integration with Azure Self-hosted, also supports
Cloud Integration
GCP, Azure services Azure & AWS

Key Takeaways:
Tableau:

• Best for data visualization and quick insights.

• Suitable for users needing live dashboards with external databases.
• Lacks strong built-in ETL; often paired with Tableau Prep or external tools like Alteryx.

Power BI:

• Excellent ETL and modeling capabilities via Power Query and DAX.
• Ideal for organizations already within the Microsoft ecosystem.
• Better for self-service BI where business users handle their own data.

Dundas BI:

• Geared more towards developers and IT teams for customized BI solutions.

• Strong in embedded BI, custom dashboards, and enterprise-grade control.
• Has advanced scripting and automation features for data provisioning and transformation.

Conclusion:

Each BI tool offers a distinct approach to data provisioning:

• Tableau is best for visual-first analysis with external data sources.

• Power BI excels in end-to-end BI workflows, especially within Microsoft environments.
• Dundas BI is optimal for custom, enterprise BI solutions requiring flexibility and integration.

10. Visual Representation of the Data Provision Process

Here’s a detailed flowchart description of the Data Provision process, highlighting key stages and
components in ETL and BI systems:
Explanation of Key Stages:

1. Source Systems: These are operational databases, external files, cloud apps, or any systems holding
business data.
2. Data Extraction: Extract data in either full loads or incremental loads (using CDC or timestamps).
3. Data Staging: Acts as a buffer or holding area where raw data resides temporarily.
4. Data Transformation: Core ETL work occurs here: cleaning inconsistent data, applying business
rules, joining datasets, enriching data, and performing lookups.
5. Data Loading: The cleaned, transformed data is loaded into data warehouses or smaller data marts
optimized for querying.
6. Data Provisioning: Data is provisioned through data marts, cubes, or direct connections, made ready
for consumption by BI tools.
7. BI & Analytics: Final stage where users interact with data via dashboards, reports, and analytics to
support decision-making.