1

1
 Defined in many different ways, but not rigorously.
 A decision support database that is maintained separately from the
organization’s operational database
 Support information processing by providing a solid platform of
consolidated, historical data for analysis.
 “A data warehouse is a subject-oriented, integrated, time-variant, and
nonvolatile collection of data in support of management’s decision-making
process.”—W. H. Inmon
 Data warehousing:
 The process of constructing and using data warehouses

2
 Organized around major subjects, such as customer, product,
sales
 Focusing on the modeling and analysis of data for decision
makers, not on daily operations or transaction processing
 Provide a simple and concise view around particular subject
issues by excluding data that are not useful in the decision
support process

3
 Constructed by integrating multiple, heterogeneous data
sources
 relational databases, flat files, on-line transaction records

 Data cleaning and data integration techniques are applied.

 Ensure consistency in naming conventions, encoding

structures, attribute measures, etc. among different data

sources
 E.g., Hotel price: currency, tax, breakfast covered, etc.
 When data is moved to the warehouse, it is converted.

4
 The time horizon for the data warehouse is significantly longer
than that of operational systems
 Operational database: current value data
 Data warehouse data: provide information from a historical
perspective (e.g., past 5-10 years)
 Every key structure in the data warehouse
 Contains an element of time, explicitly or implicitly
 But the key of operational data may or may not contain
“time element”

5
 A physically separate store of data transformed from the
operational environment
 Operational update of data does not occur in the data
warehouse environment
 Does not require transaction processing, recovery, and
concurrency control mechanisms
 Requires only two operations in data accessing:
 initial loading of data and access of data

6
 OLTP OLAP
users clerk, IT professional knowledge worker
function day to day operations decision support
DB design application-oriented subject-oriented
data current, up-to-date historical,
detailed, flat relational summarized, multidimensional
isolated integrated, consolidated
usage repetitive ad-hoc
access read/write lots of scans
index/hash on prim. key
unit of work short, simple transaction complex query
# records accessed tens millions
#users thousands hundreds
DB size 100MB-GB 100GB-TB
metric transaction throughput query throughput, response

7
 High performance for both systems
 DBMS— tuned for OLTP: access methods, indexing, concurrency

control, recovery
 Warehouse—tuned for OLAP: complex OLAP queries,

multidimensional view, consolidation

 Different functions and different data:
 missing data: Decision support requires historical data which operational

DBs do not typically maintain

 data consolidation: DS requires consolidation (aggregation,

summarization) of data from heterogeneous sources

 data quality: different sources typically use inconsistent data

representations, codes and formats which have to be reconciled

 Note: There are more and more systems which perform OLAP analysis
directly on relational databases

8
Data Warehouse: A Multi-Tiered Architecture

Monitor
Metadata & OLAP Server
Other
sources Integrator

Analysis
Operational Extract Query
DBs Transform Data Serve Reports
Load
Refresh
Warehouse Data mining

Data Marts

Data Sources Data Storage OLAP Engine Front-End Tools

9
 Enterprise warehouse
 collects all of the information about subjects spanning the

entire organization
 Data Mart
 a subset of corporate-wide data that is of value to a specific

groups of users. Its scope is confined to specific, selected

groups, such as marketing data mart
 Independent vs. dependent (directly from warehouse) data mart
 Virtual warehouse
 A set of views over operational databases

 Only some of the possible summary views may be

materialized
10
 Data extraction
 get data from multiple, heterogeneous, and external sources
 Data cleaning
 detect errors in the data and rectify them when possible
 Data transformation
 convert data from legacy or host format to warehouse format
 Load
 sort, summarize, consolidate, compute views, check integrity,
and build indicies and partitions
 Refresh
 propagate the updates from the data sources to the warehouse

11
 Meta data is the data defining warehouse objects. It stores:
 Description of the structure of the data warehouse
 schema, view, dimensions, hierarchies, derived data defn, data mart

locations and contents

 Operational meta-data
 data lineage (history of migrated data and transformation path), currency

of data (active, archived, or purged), monitoring information (warehouse

usage statistics, error reports, audit trails)
 The algorithms used for summarization
 The mapping from operational environment to the data warehouse
 Data related to system performance
 warehouse schema, view and derived data definitions

 Business data
 business terms and definitions, ownership of data, charging policies

12
all all

region Europe ... North_America

country Germany ... Spain Canada ... Mexico

city Frankfurt ... Vancouver ... Toronto

office L. Chan ... M. Wind

13
 Sales volume as a function of product, month,
and region Dimensions: Product, Location, Time
Hierarchical summarization paths
o n
gi

Industry Region Year

Re

Category Country Quarter

Product

Product City Month Week

Office Day

Month
14
 Roll up (drill-up): summarize data
 by climbing up hierarchy or by dimension reduction
 Drill down (roll down): reverse of roll-up
 from higher level summary to lower level summary or detailed
data, or introducing new dimensions
 Slice and dice: project and select
 Pivot (rotate):
 reorient the cube, visualization, 3D to series of 2D planes
 Other operations
 drill across: involving (across) more than one fact table
 drill through: through the bottom level of the cube to its back-
end relational tables (using SQL)
15
 Top-down, bottom-up approaches or a combination of both
 Top-down: Starts with overall design and planning (mature)
 Bottom-up: Starts with experiments and prototypes (rapid)
 From software engineering point of view
 Waterfall: structured and systematic analysis at each step before
proceeding to the next
 Spiral: rapid generation of increasingly functional systems, short turn
around time, quick turn around
 Typical data warehouse design process
 Choose a business process to model, e.g., orders, invoices, etc.
 Choose the grain (atomic level of data) of the business process
 Choose the dimensions that will apply to each fact table record
 Choose the measure that will populate each fact table record
16
 Three kinds of data warehouse applications
 Information processing
 supports querying, basic statistical analysis, and reporting using
crosstabs, tables, charts and graphs
 Analytical processing
 multidimensional analysis of data warehouse data
 supports basic OLAP operations, slice-dice, drilling, pivoting
 Data mining
 knowledge discovery from hidden patterns
 supports associations, constructing analytical models, performing
classification and prediction, and presenting the mining results using
visualization tools
17
 Generalized relation:
 Relations where some or all attributes are generalized, with counts or other
aggregation values accumulated.
 Cross tabulation:
 Mapping results into cross tabulation form (similar to contingency tables).
 Visualization techniques:
 Pie charts, bar charts, curves, cubes, and other visual forms.
 Quantitative characteristic rules:
 Mapping generalized result into characteristic rules with quantitative
information associated with it, e.g.,

grad ( x)  male( x) 
birth _ region( x) "Canada"[t :53%]  birth _ region( x) " foreign"[t : 47%].
18
 Knowledge = Facts + Beliefs + Heuristics
 Success = Finding a good-enough answer with
the resources available
 Search efficiency directly affects success
 Several difficult problems do not have
tractable algorithmic solutions
 Human experts achieve high level of
performance through the application of
quality knowledge
 Knowledge in itself is a resource.
Extracting it from humans and putting it in
computable forms reduces the cost of
knowledge reproduction and exploitation
 Exponential growth in information storage
 Tremendous increase in information retrieval
 Information is a factor of production
 Knowledge is lost due to information overload
 Knowledge discovery in databases
 “non-trivial extraction of implicit, previously
unknown and potentially useful knowledge from
data”
 Data mining
 Discovery stage of KDD
 Problem definition
 Data selection
 Cleaning
 Enrichment
 Coding and organization
 DATA MINING
 Reporting
 Examples
 What factors affect treatment compliance?

 Are there demographic differences in drug

effectiveness?

 Does patient retention differ among doctors and

diagnoses?
 Which patients?
 Which doctors?
 Which diagnoses?
 Which treatments?
 Which visits?
 Which outcomes?
 Removal of duplicate records
 Removal of records with gaps
 Enforcement of check constraints
 Removal of null values
 Removal of implausible frequent values
 Supplementing operational data with outside data
sources
 Pharmacological research results
 Demographic norms
 Epidemiological findings
 Cost factors
 Medium range predictions