Chapter 11:

Data Warehousing

Modern Database Management

Jeffrey A. Hoffer, Mary B. Prescott, Fred

R. McFadden

1
 Objectives
• Definition of terms
• Reasons for information gap between
information needs and availability
• Reasons for need of data warehousing
• Describe three levels of data warehouse
architectures
• List four steps of data reconciliation
• Describe two components of star schema
• Estimate fact table size
• Design a data mart

2
 Definition
• Data Warehouse:
Warehouse
– A subject-oriented, integrated, time-variant, non-updatable
collection of data used in support of management decision-
making processes
– Subject-oriented: e.g. customers, patients, students,
products
– Integrated: Consistent naming conventions, formats,
encoding structures; from multiple data sources
– Time-variant: Can study trends and changes
– Nonupdatable: Read-only, periodically refreshed
• Data Mart:
Mart
– A data warehouse that is limited in scope

3
 Need for Data Warehousing
• Integrated, company-wide view of high-quality
information (from disparate databases)
• Separation of operational and informational systems
and data (for improved performance)

4
5
 Data Warehouse Architectures
• Generic Two-Level Architecture
• Independent Data Mart
• Dependent Data Mart and Operational
Data Store
• Logical Data Mart and @ctive Warehouse
• Three-Layer architecture

All involve some form of extraction, transformation and loading (ETL)

ETL

6
 Figure 11-2: Generic two-level data warehousing architecture

L
One,
company-
wide
T warehouse

Periodic extraction  data is not completely current in warehouse

7
Figure 11-3 Independent data mart Data marts:
data warehousing architecture Mini-warehouses, limited in scope

T
E

Separate ETL for each Data access complexity

independent data mart due to multiple data marts
8
Figure 11-4 Dependent data mart with ODS provides option for
operational data store: a three-level architecture obtaining current data

T
E Simpler data access
Single ETL for
enterprise data warehouse Dependent data marts
(EDW) loaded from EDW
9
Figure 11-5 Logical data mart and real ODS and data warehouse
are one and the same
time warehouse architecture

T
E
Near real-time ETL for Data marts are NOT separate databases,
Data Warehouse but logical views of the data warehouse
 Easier to create new data marts 10
 Figure 11-7 Data Characteristics
Example of DBMS
log entry Status vs. Event Data
Status

Event = a database action

(create/update/delete) that
results from a transaction

Status

12
 Figure 11-8
Transient
Data Characteristics
operational data Transient vs. Periodic Data
With
transient
data,
changes to
existing
records are
written over
previous
records, thus
destroying
the previous
data content

13
 Figure 11-9:
Periodic
Data Characteristics
warehouse data Transient vs. Periodic Data

Periodic
data are
never
physically
altered or
deleted
once they
have
been
added to
the store

14
 Other Data Warehouse
Changes
• New descriptive attributes
• New business activity attributes
• New classes of descriptive attributes
• Descriptive attributes become more
refined
• Descriptive data are related to one another
• New source of data

15
 The Reconciled Data Layer
• Typical operational data is:
– Transient–not historical
– Not normalized (perhaps due to denormalization for performance)
– Restricted in scope–not comprehensive
– Sometimes poor quality–inconsistencies and errors
• After ETL, data should be:
– Detailed–not summarized yet
– Historical–periodic
– Normalized–3rd normal form or higher
– Comprehensive–enterprise-wide perspective
– Timely–data should be current enough to assist decision-making
– Quality controlled–accurate with full integrity

16
 The ETL Process

• Capture/Extract
• Scrub or data cleansing
• Transform
• Load and Index

ETL = Extract, transform, and load

17
 Capture/Extract…obtaining a snapshot of a chosen subset
of the source data for loading into the data warehouse
Figure 11-10:
Steps in data
reconciliation

Static extract = capturing Incremental extract =

a snapshot of the source capturing changes that
data at a point in time have occurred since the last
static extract 18
 Scrub/Cleanse…uses pattern recognition and AI
techniques to upgrade data quality
Figure 11-10:
Steps in data
reconciliation
(cont.)

Fixing errors: misspellings, Also: decoding, reformatting,

erroneous dates, incorrect field time stamping, conversion, key
usage, mismatched addresses, generation, merging, error
missing data, duplicate data, detection/logging, locating
inconsistencies missing data
19
 Transform = convert data from format of operational
system to format of data warehouse
Figure 11-10:
Steps in data
reconciliation
(cont.)

Record-level: Field-level:
Selection–data partitioning single-field–from one field to one field
Joining–data combining multi-field–from many fields to one, or
Aggregation–data summarization one field to many
20
 Load/Index= place transformed data
into the warehouse and create indexes
Figure 11-10:
Steps in data
reconciliation
(cont.)

Refresh mode: bulk rewriting Update mode: only changes

of target data at periodic intervals in source data are written to data
warehouse
21
Figure 11-11: Single-field transformation

In general–some transformation
function translates data from old
form to new form

Algorithmic transformation uses

a formula or logical expression

Table lookup–another
approach, uses a separate
table keyed by source
record code

22
Figure 11-12: Multifield transformation

M:1–from many source

fields to one target field

1:M–from one
source field to
many target fields

23
 Derived Data
• Objectives
– Ease of use for decision support applications
– Fast response to predefined user queries
– Customized data for particular target audiences
– Ad-hoc query support
– Data mining capabilities
 Characteristics
– Detailed (mostly periodic) data
– Aggregate (for summary)
– Distributed (to departmental servers)

Most common data model = star schema

(also called “dimensional model”)
24
 Figure 11-13 Components of a star schema
Fact tables contain factual
or quantitative data

1:N relationship between Dimension tables are denormalized to

dimension tables and fact tables maximize performance

Dimension tables contain descriptions

about the subjects of the business

Excellent for ad-hoc queries, but bad for online transaction processing
25
Figure 11-14: Star schema example

Fact table provides statistics for sales

broken down by product, period and store
dimensions

26
Figure 11-15 Star schema with sample data

27
 Issues Regarding Star Schema
• Dimension table keys must be surrogate (non-intelligent and
non-business related), because:
– Keys may change over time
– Length/format consistency
• Granularity of Fact Table–what level of detail do you want?
– Transactional grain–finest level
– Aggregated grain–more summarized
– Finer grains  better market basket analysis capability
– Finer grain  more dimension tables, more rows in fact table
• Duration of the database–how much history should be kept?
– Natural duration–13 months or 5 quarters
– Financial institutions may need longer duration
– Older data is more difficult to source and cleanse

28
 Figure 11-16: Modeling dates

Fact tables contain time-period data

 Date dimensions are important
29
On-Line Analytical Processing (OLAP)
• The use of a set of graphical tools that provides users
with multidimensional views of their data and allows
them to analyze the data using simple windowing
techniques
• Relational OLAP (ROLAP)
– Traditional relational representation
• Multidimensional OLAP (MOLAP)
– Cube structure
• OLAP Operations
– Cube slicing – come up with 2-D view of data
– Drill-down – going from summary to more detailed views

31
Figure 11-22: Slicing a data cube

32
 Summary report
Figure 11-24
Example of drill-down

Starting with summary

data, users can obtain Drill-down with
details for particular color added
cells

33
 Data Mining and Visualization
• Knowledge discovery using a blend of statistical, AI, and
computer graphics techniques
• Goals:
– Explain observed events or conditions
– Confirm hypotheses
– Explore data for new or unexpected relationships
• Techniques
– Case-based reasoning
– Rule discovery
– Signal processing
– Neural nets
– Fractals
• Data visualization – representing data in graphical/multimedia
formats for analysis

34