Data Warehouse Concepts and Architectures

Prof. Rose Andrea Dimaano, CPA, MSAC

Lesson Objectives
● Discuss historical factors
● Explain data warehouse characteristics
● Compare/contrast characteristics for operational
databases versus a data warehouses
Welcome to lesson three of module one on data warehouse concepts and architectures.
I’m going to start, with an important historical question
that I want you to think about throughout this lesson

What historical reason for data warehouse development does not primarily involve
technology?
This question is highly relevant today, because the success of data warehouse deployment
depends on this organizational capability.

This lesson provides informational background about data warehouses,

is a context for detailed skills that you will develop in other modules.

You have three learning objects in this lesson today. You should be able to discuss
historical factors
that drove the development of data warehouse deployment and technology.
In your own words, you should be able to explain characteristics of data warehouses.
To clarify your understanding of operational databases and data warehouses, you should be
able to explain several differences between the two types of databases.
The databases support decision making in organizations. The traditional decision
making hierarchy depicts management levels and volumes of decisions at each level.
Two Types of Databases

● The databases support decision making in

organizations.

● The traditional decision making hierarchy

depicts management levels and volumes of
decisions at each level.
What historical reason for
data warehouse development does not primarily
involve technology?
Decision Making Hierarchy

Decision making hierarchy Typical decisions

Identify new markets,

choose store locations
Choose suppliers,
forecast sales

Resolve order delays,

schedule employees
The databases support decision making in organizations.

The traditional decision making hierarchy depicts

management levels and volumes of decisions at each level.

Organizations believed that operational databases would

support decision making at all three levels.

Operational databases were designed for efficient transaction

processing, and support for lower level decisions such as resolving
order delays. However, organizations encountered great
difficulty with usage of operational databases for higher levels of
decision making.
Lower-level management deals with short-term problems related to individual
transactions. Periodic summaries of operational databases and exception reports assist
operational management. Middle management relies on summarized data that are
integrated across operational databases. Middle management may want to integrate
data across different departments, manufacturing plants, and retail stores. Top
management relies on the results of middle management analysis and external data
sources. Top management needs to integrate data so that customers, products,
suppliers, and other important entities can be tracked across the entire organization. In
addition, external data must be summarized and then integrated with internal data.

Operational databases serve lower level decision making. In the early days of relational
databases, it was assumed that operational databases would also support higher levels
of decision making.
Technology and Deployment Limitations

Lack of
integration

Missing
Performance
DBMS
limitations
features

Data
warehouse
technology
and
deployments
This difficulty spurred the development of data warehouse technology
and deployment of data warehouses starting in the mid 1990s.
The failure of operational databases to support high level decision making is a
combination of inadequacy of database technology, and limitations in
deployment of databases.

The limitations were a combination of inadequacy of database technology and

deployment limitations.

Product vendors discovered that lack key features to support summery data
and analytical calculations vital for business intelligence processing.

The SQL group by clause was inadequate to specify queries involving summery
data.

The select statement in SQL did not have any features for analytical
calculations such as moving averages.

Storage optimization methods weren't adequate for queries involving summary

data.
Missing features for summary data
- Storage and optimization techniques for summary queries
- Data modeling approaches
- Support for precomputed query results
- Support for different business analyst query tools

Organizations in DBMS vendors discover that a single database

could not be configured to provide adequate performance for
both transaction processing and business intelligence processing.

Performance limitation
- Performance problems with a separate database for both transaction processing and
business intelligence decision making
- Never solved. Use a separate database
The lack of integration was not a design failure.

Its operational databases primarily support transaction processing, not business

intelligence processing.

Organizations discovered that retrofitting integration for operational databases

was difficult.

Lack of integration
-Most important issue; highlight with pen
-Management issue
-Lack of integration with transaction databases and external data
sources
-Add value: integrate, standardize, clean, and summarize both internal
and external data sources

Initially separate companies developed technology independent of relational

databases
Relational database vendors did not focus on relational database
technology extensions until late 1990s

Gradually, solutions for these problem emerge in organizations

and product vendors.

Performance limitations demanded that data warehouses be

separate from operational databases.
The lack of integration requires an intense focus on adding value to
data sources through transformations in integration.

Lack of features spur development of a host of new features for

representation, manipulation storage, and processing of summary
data and analytical calculations.
 Data Warehouse Characteristics
● Essential part of infrastructure for business intelligence
● Logically centralized repository for decision making
○ Populated from operational databases and external data
sources
○ Integrated and transformed data
○ Optimized for reporting and periodic integration
Data warehouses have become an essential part of an organization's
infrastructure to support business intelligence.

Data warehouse, a term coined by William Inmon in 1990, refers to a

logically centralized data repository, where data from operational
databases and external sources are integrated, cleaned, and standardized
to support business intelligence.

The transformational activities, cleaning, merging, and

standardizing, are essential for achieving value for business
intelligence.

Data warehouses are optimized for reporting, often involving

summarization of large amounts of data as well as periodic processing to
integrate, and transform source data.
Definition:
- Central repository: populated from operational databases and external
data sources
- Many transformations to clean, standardize, and integrate
- Summarized: no need to identify individual transactions although
transaction details
may be useful for flexible data analysis; summary data for optimizing
reporting

Characteristics
- Subject-oriented: Organized around business entities (e.g., customers,
products, and employees) rather than business processes
- Integrated: many transformations to unify source data from independent
data sources (units of measure, data formats, naming conventions)
- Time-variant: historical data (time stamped); snapshots of business
processes captured at different points in time
- Nonvolatile: new data are appended periodically; existing data is not
changed; warehouse data may be archived after its usefulness declines
Comparison of Processing Environments
Transaction
processing
• Primary data from
transactions
• Daily operations and
short term decisions

Business intelligence
processing
• Transformed secondary
data
• Medium and long-term
decisions
Transaction processing uses primary data from large
volumes of transactions to support daily operations and
short-term decision making of an organization.

In contrast, business intelligence processing uses

transformed secondary data to support medium and long-term
decision making.

A data warehouse provides value for longer term decision

making through transformations and integration to
operational databases and external data sources.
Transaction processing:
- Operational databases: for daily business
- Decisions: involve details of products, customers, shipments,
manufacturing such as fulfill orders, resolve complaints, provide staffing

Business intelligence:
- Integrated and standardized data: difficult to directly use operational
data
- Substantial processing for transformations and integration
- Value for decision making results from standardizing and integrating data
across organizational units and external sources
- Decisions: broad view of customers, products, production, marketing for
capacity planning, store locations, new lines of business, …

Due to the support of different types of processing,

operational databases differ substantially from data warehouses.
 Question: A database that supports daily
activities and short term decisions is called:

A. Data warehouse
B. Business intelligence
C. Operational database
D. Integrated database
Data Comparison
Characteristic Operational Data Warehouse
Database
Currency Current Historical
Details level Individual Individual and summary
Orientation Process Subject
Records per Few Thousands
request
Normalization Mostly normalized Normalization relaxed
level
Update level Highly volatile Mostly refreshed (non volatile)
Data model Relational Relational (star schemas) and
multidimensional (data cubes)
Operational databases largely contain current data at the individual level,

while data warehouses have historical data at both the individual and summarized
levels.

Individual level data provides flexibility for responding to a wide range of

business intelligence needs while summarized data provides fast response to
repetitive queries.

For example, an operational database to support order processing requires data

about individual customers, orders, and inventory items.

While a business intelligence application may use monthly sales over a period of
several years.

Operational databases, therefore, have a process orientation, for example, all

data relevant to a particular business process, compared to a subject orientation for
a data warehouse. For example, all customer data or order data.
A transaction typically updates only a few records in an operational database,
or is a business intelligence application may query thousands to millions of
records from a data warehouse.

Normalization is less important for data warehouses, because of the focus on

reporting rather transaction processing.

Operational databases are highly volatile, processing large volumes of

transactions, while data warehouses are non-volatile, with periodic
refreshment to integrate new source data.

Operational databases primarily use the relational data model, while data
warehouses use star schema patterns of tables as well as a multidimensional
data model.

Schema patterns typically differ between operational databases and data

warehouses.
Operational database vs. Data warehouse
- Currency: age of data
- Historical: completed transactions; secondary data
- Detail level: DW do not need to identify individual transactions
although details can be stored; typically both detailed and
summary data are stored in a data warehouse
- Number of records processed (per request): operational
database query typically returns few records; DW query may
summarize thousands of records
- Normalization: normalization not important for DW because of
lack of update
- Data model: new data model for DWs although relational
implementation is generally used to implement the new data
model
- Star schemas with materialized views for summary data are
used in relational DBMSs
Schema Comparison
Operational database Data warehouse
In an operational database to support order processing,
many to many relationships or equivalently associative tables are
typically used to represent an order heading and details.

In contrast, a data warehouse will typically just show one level detail
and no many to many relationships.

In addition, specialized relationships such as self-referencing

and identification dependency are less commonly used in data warehouse
designs.

You will learn details about schema patterns for data warehouses in Lesson
three.

This lesson has introduced important concepts about data warehouses to

provide a context for detailed skills that you will develop in later modules.
Review:
one-to-one (1:1) In a traditional
American marriage, a man can be
married to only one woman; a woman
can be married to only one man.
one-to-many (1:N) A child has
exactly one biological father; a father
can have many biological children.
many-to-many (M:N) A student can
enroll in many classes; a class can
have many enrolled students.
Crow’s foot ERD notation
Small diagram of order entry database and sales star schema
- Small ERD of order entry database with customer, order, employee,
product, and M-N relationship between product and order; designed
for order processing
- More complex schema patterns for operational databases
- Order contains header (order) and detail lines (Contains M-N
relationship or a associative table)
- Star Schema with dimension tables (item, store, customer, and
timedim) and fact table (sales); designed for business intelligence
reporting
- Sales flattened just for order details (sales of individual items)
- Data integration: data moved from operational database to data
warehouse after completion of transactions along with substantial
amount of transformations (standardization, integration, consistency
checking, completeness checking, …)
In answer to the opening question, lack of integration is primarily an
organizational issue, not a technology issue.
The level of integration is a data quality concern. Low levels of
integration do not typically impact operational databases that support
transaction processing and short-term decision making.
The lack of integration is a major concern for data warehouses

supporting longer term decision making.

Achieving higher levels of integration may involved coordination

and compromise across organizational units, not technology.
Coordination and compromise is not easy to achieve. Many data
warehouse projects have failed due to the difficulty of coordination
and compromise.
Technology to implement higher levels of data integration is important
after coordination and compromise is achieved.
Summary
● Historical reasons for data warehouse development
● Key characteristics of data warehouses
● Differences between operational databases and data
warehouses
 Data
Warehouse
Concepts

Databse Data Historical

Warehouse