Unit-3

Managing Data Resources

Syllabus

Managing Data Resources- The need for data management, Challenges of data
management, Data independence, Data redundancy, Data consistency, Data
administration. Database Management System – Concepts and types of DBMS, Fields,
Records, Table, View, Reports and Queries. Data warehouse and Data mining –
Characteristics and uses of Data warehouse, Techniques of Data Mining, Business
Intelligence

Managing Data Resources

Managing data resources effectively is crucial for organizations to leverage data for decision-
making, operational efficiency, and strategic advantage. Here’s a detailed look at the need for
data management and the challenges involved:

The Need for Data Management

1. Improved Decision Making

o Accurate Data: Decision-makers rely on data to inform strategies and
operations. Accurate data eliminates guesswork, providing a reliable foundation
for decisions. For instance, sales data helps in forecasting future trends and
planning inventory.
o Real-Time Insights: Access to real-time data allows organizations to respond
swiftly to changes in the market or internal processes. For example, real-time
data on production levels can help adjust manufacturing schedules to meet
demand.
2. Operational Efficiency
o Streamlined Processes: Data management systems can automate and streamline
business processes, reducing manual intervention and errors. For example,
automated inventory systems update stock levels in real-time, ensuring efficient
inventory management.

1
 o Resource Optimization: Effective data management helps in tracking and
optimizing the use of resources, such as materials, labour, and capital. This leads
to cost savings and better resource allocation.
3. Data Quality and Integrity
o Consistency: Ensuring data consistency across different departments and
systems prevents discrepancies. For instance, consistent customer data across
sales, marketing, and service departments enhances customer relationship
management.
o Accuracy: Accurate data is critical for various functions, including financial
reporting, compliance, and strategic planning. Inaccurate data can lead to
incorrect conclusions and poor business outcomes.
4. Compliance and Security
o Regulatory Compliance: Organizations must comply with laws and regulations
governing data privacy and protection, such as GDPR and CCPA. Effective data
management ensures adherence to these regulations, avoiding legal penalties.
o Data Security: Protecting data from breaches, theft, and unauthorized access is
paramount. Implementing robust security measures safeguards sensitive
information, such as customer data and intellectual property.
5. Customer Satisfaction
o Personalized Services: Data management allows businesses to analyze
customer data and provide personalized experiences. For example, e-commerce
platforms use purchase history to recommend products.
o Timely Responses: Efficient data management systems enable quick access to
customer information, allowing timely and accurate responses to customer
inquiries and issues.
6. Strategic Planning
o Trend Analysis: Analysing historical data helps identify trends and patterns,
informing strategic decisions. For example, analysing sales trends can guide
product development and marketing strategies.
o Competitive Advantage: Organizations that effectively manage their data can
gain insights that lead to a competitive advantage. Data-driven strategies can
identify new market opportunities and optimize business operations.

2
Challenges of Data Management

1. Data Quality Issues

o Inaccurate Data: Inaccurate data can result from human error, system glitches,
or poor data entry practices. This can lead to misguided decisions and operational
inefficiencies.
o Data Inconsistency: When data is inconsistent across different systems or
departments, it can cause confusion and errors. For example, if customer
addresses are not consistently updated across systems, it can lead to incorrect
deliveries.
2. Data Integration
o Heterogeneous Sources: Integrating data from various sources (e.g., databases,
spreadsheets, cloud services) with different formats and standards is challenging.
Organizations need to standardize data formats and ensure compatibility.
o Legacy Systems: Many organizations still rely on legacy systems that may not
be compatible with modern data management tools. Integrating these systems
can be complex and costly.
3. Data Security and Privacy
o Cyber Threats: Protecting data from cyber threats such as hacking, phishing,
and malware attacks is a continuous challenge. Organizations must implement
strong cybersecurity measures to safeguard their data.
o Compliance: Meeting regulatory requirements for data protection and privacy
is crucial. Non-compliance can result in severe penalties and damage to
reputation.
4. Data Volume and Variety
o Big Data: Managing and analysing large volumes of data, known as big data,
requires significant storage capacity and advanced analytics tools. This can be
resource-intensive.
o Diverse Data Types: Handling different types of data (structured, unstructured,
and semi-structured) from various sources adds complexity. Organizations need
to adopt technologies that can process and analyze diverse data types.
5. Data Governance

3
 o Policies and Procedures: Establishing and enforcing data governance policies
and procedures ensures data quality, consistency, and compliance. This includes
data stewardship, data quality management, and data lifecycle management.
o Ownership and Responsibility: Defining clear ownership and responsibility for
data within the organization is crucial. This involves assigning data stewards and
ensuring accountability for data management.
6. Data Accessibility
o Timely Access: Ensuring that users have timely access to the data they need for
their tasks is critical. This requires efficient data retrieval systems and policies
that balance access with security.
o Usability: Data should be easily understandable and usable, even for non-
technical users. This involves user-friendly interfaces and tools for data analysis
and visualization.
7. Scalability
o Growing Data Needs: As organizations grow, their data management systems
must scale to accommodate increasing data volumes and user demands. This
requires scalable infrastructure and flexible data management solutions.
o Infrastructure Costs: Scaling data management infrastructure can be costly.
Organizations need to manage these costs while ensuring efficient data storage
and processing capabilities.
8. Data Lifecycle Management
o Retention and Disposal: Effective data management involves defining policies
for data retention, archiving, and disposal. This ensures that data is kept only as
long as needed and disposed of securely when no longer required.
o Version Control: Maintaining control over different versions of data ensures
that updates are managed properly and that users have access to the most current
and accurate information.

Data Independence

Data independence refers to the capacity to change the schema at one level of a database
system without altering the schema at the next higher level. It is essential for maintaining a
flexible and efficient database management system (DBMS). There are two types of data
independence:

4
 1. Logical Data Independence:
o Definition: The ability to change the conceptual schema without altering
external schemas or application programs.
o Importance: Logical data independence ensures that applications are insulated
from changes in the logical structure of the database. For example, adding a new
field to a table or merging two tables should not require changes to the programs
accessing the database.
o Challenges: Achieving logical data independence can be challenging due to the
complexity of mapping changes in the logical schema to the physical storage.
2. Physical Data Independence:
o Definition: The ability to change the internal schema without altering the
conceptual schema or external schemas.
o Importance: Physical data independence allows changes in the storage devices,
file organization, indexing methods, or access paths without impacting the
database's logical structure. For instance, switching from one storage format to
another should not affect the applications using the database.
o Challenges: While easier to achieve than logical data independence, maintaining
physical data independence requires a robust DBMS that can abstract physical
storage details.

Data Redundancy

Data redundancy refers to the unnecessary duplication of data within a database. While some
redundancy can be intentional for backup or performance reasons, excessive redundancy can
lead to various issues:

1. Causes:
o Unnormalized Database Design: A poorly designed database schema that does
not adhere to normalization principles can lead to redundant data.
o Data Integration: Combining data from multiple sources without proper
integration can create redundancy.
o System Migrations: Legacy systems or transitions between systems might
introduce redundancy.
2. Problems:

5
 o Inconsistency: Redundant data can lead to inconsistency, where the same data
appears differently in different places.
o Increased Storage Costs: Storing redundant data consumes more storage space,
increasing costs.
o Maintenance Overhead: Keeping redundant data synchronized requires
additional maintenance effort and can be error-prone.
3. Solutions:
o Normalization: Applying normalization techniques to organize the database
schema reduces redundancy.
o Database Design: Designing databases with integrity constraints and foreign
keys helps minimize redundancy.
o Data Integration Tools: Using tools and techniques for proper data integration
can help eliminate redundancy.

Data Consistency

Data consistency ensures that data remains accurate and reliable across the database.
Consistency means that any given set of data will yield the same results, ensuring the integrity
of the database.

1. Importance:
o Accurate Information: Consistent data is crucial for accurate reporting,
analysis, and decision-making.
o Trust and Reliability: Consistent data fosters trust in the system and its outputs.
2. Challenges:
o Concurrency: Managing simultaneous operations on the database can lead to
consistency issues, especially in multi-user environments.
o Data Redundancy: Redundant data can lead to inconsistencies if not properly
managed.
3. Ensuring Consistency:
o Transaction Management: Using transactions that adhere to the ACID
(Atomicity, Consistency, Isolation, Durability) properties ensures data
consistency.

6
 o Validation Rules: Implementing validation rules and constraints within the
database enforces data integrity.
o Data Cleaning: Regular data cleaning processes can help maintain consistency
by correcting or removing incorrect, duplicate, or outdated data.

Data Administration

Data administration involves the management and oversight of data resources to ensure their
availability, reliability, and security. Data administrators (DAs) or Database Administrators
(DBAs) are responsible for this function.

1. Roles and Responsibilities:

o Database Design and Maintenance: Designing the database schema, creating
and maintaining database structures, and optimizing database performance.
o Security Management: Implementing and managing security measures to
protect data from unauthorized access, breaches, and other threats.
o Backup and Recovery: Establishing and managing backup and recovery
procedures to protect data against loss and ensure business continuity.
o Performance Tuning: Monitoring and tuning database performance to ensure
efficient data retrieval and processing.
o User Management: Managing user access and permissions to ensure that users
have appropriate levels of access to the data.
o Data Quality Management: Ensuring data accuracy, consistency, and
reliability through data quality initiatives and processes.
2. Challenges:
o Scalability: Managing the growing volume of data and ensuring the database
can scale efficiently.
o Compliance: Ensuring compliance with various legal and regulatory
requirements related to data privacy and protection.
o Complexity: Handling the complexity of modern database systems, which may
involve multiple types of databases and integration with other systems.
o Security Threats: Protecting data from increasingly sophisticated cyber threats
and attacks.
3. Tools and Techniques:

7
 o Database Management Systems (DBMS): Utilizing robust DBMS tools for
efficient database administration.
o Monitoring Tools: Implementing monitoring tools to track database
performance, usage, and security.
o Automation: Leveraging automation for routine tasks such as backups, updates,
and performance tuning to reduce manual workload and minimize errors.

Database Management System (DBMS)

A Database Management System (DBMS) is a software system designed to manage

databases. It provides an interface for users and applications to interact with the data, ensuring
data consistency, integrity, and security. DBMSs are essential for organizing, storing, retrieving,
and managing data efficiently.

Concepts and Types of DBMS

1. Concepts of DBMS:
o Data Models: Data models define the logical structure of a database. The most
common data models are the relational model, hierarchical model, network
model, and object-oriented model.
o Schemas: A schema is the blueprint or architecture of a database. It defines how
data is organized and how the relationships among data are associated.
o Instances: An instance is a snapshot of the data in the database at a specific
moment. While the schema remains the same, instances change frequently as
data is added, deleted, or modified.
o Data Independence: Data independence refers to the ability to modify the
schema at one level without affecting the schema at another level.
o Database Languages: DBMS uses various languages to manage and interact
with the database, including Data Definition Language (DDL), Data
Manipulation Language (DML), Data Control Language (DCL), and
Transaction Control Language (TCL).
2. Types of DBMS:

8
 o Relational Database Management System (RDBMS): Uses a relational
model, where data is stored in tables (relations). Examples include MySQL,
PostgreSQL, Oracle, and SQL Server.
o Hierarchical Database Management System (HDBMS): Uses a tree-like
structure to represent relationships among data. An example is IBM's
Information Management System (IMS).
o Network Database Management System (NDBMS): Uses a graph structure to
represent complex relationships among data. An example is the Integrated Data
Store (IDS).
o Object-Oriented Database Management System (OODBMS): Stores data in
objects, similar to object-oriented programming. Examples include db4o and
Object DB.
o NoSQL Database Management System: Designed for distributed data stores
with large data storage needs. Examples include MongoDB, Cassandra, and
Redis.
o In-Memory Database Management System (IMDBMS): Stores data in the
main memory to provide faster data retrieval. Examples include SAP HANA and
Oracle Times Ten.

Fields, Records, Table, View, Reports, and Queries

1. Fields:
o Definition: A field is the smallest unit of data in a database table. It represents a
single attribute or column in a table.
o Examples: In a customer table, fields might include customer_id, first_name,
last_name, email, and phone_number.
2. Records:
o Definition: A record is a complete set of fields. It represents a single row in a
table.
o Examples: A record in the customer table might include the specific details for
one customer, such as 1, "John", "Doe", "john.doe@example.com", "123-456-
7890".
3. Table:

9
 o Definition: A table is a collection of related records, organized in rows and
columns. Each column represents a field, and each row represents a record.
o Examples: A table named "customers" might store information about customers,
with fields such as customer_id, first_name, last_name, email, and
phone_number.
4. View:
o Definition: A view is a virtual table created by a query. It does not store data
physically but presents data from one or more tables in a specific way.
o Examples: A view named "active_customers" might show only those customers
who are currently active, based on a query filtering out inactive customers.
5. Reports:
o Definition: A report is a formatted and organized presentation of data, usually
generated from queries. Reports are often used for analysis and decision-making.
o Examples: A sales report might summarize the total sales for each month,
displaying the data in a tabular or graphical format.
6. Queries:
o Definition: A query is a request for data or information from a database. Queries
are written in a query language, such as SQL, to retrieve, insert, update, or delete
data.
o Examples: A query to retrieve all active customers might look like: SELECT *
FROM customers WHERE status = 'active';.

Data Warehouse

A data warehouse is a centralized repository designed to store large volumes of structured data
from multiple sources. It supports data analysis and decision-making processes by consolidating
and organizing data in a way that is optimized for query and reporting operations.

Characteristics of Data Warehouse

1. Subject-Oriented:
o Focus on Business Domains: Data warehouses are designed around major
business areas such as sales, finance, or customer information rather than the

10
 organization's day-to-day operations. This focus makes it easier to analyze data
within the context of specific business subjects.
o Example: A sales data warehouse will specifically include data related to sales
transactions, customer purchases, and sales performance metrics, allowing for
targeted analysis of sales trends and customer behaviors.
2. Integrated:
o Unified Data: Data warehouses integrate data from various disparate sources,
ensuring consistency and standardization. Data from different databases and files
are cleaned and transformed into a common format.
o Example: Data from the sales department, marketing, and finance are merged
into a single cohesive structure, enabling comprehensive analysis across
departments.
3. Non-olatile:
o Stable Data Storage: Once data is entered into the data warehouse, it is not
altered or deleted. This stability ensures that historical data remains unchanged,
providing a consistent basis for analysis over time.
o Example: Historical sales data is stored without modification, allowing analysts
to track trends and patterns over the years.
4. Time-Variant:
o Historical Data: Data warehouses store historical data and time-variant data
which includes time-stamped records. This feature is crucial for trend analysis
and reporting.
o Example: Sales data is stored along with timestamps, allowing the organization
to analyze how sales performance has changed over different periods (e.g.,
monthly, quarterly, yearly).

Uses of Data Warehouse

1. Business Intelligence (BI):

o Enhanced Decision-Making: Data warehouses support BI tools that provide
insights and reports to improve strategic and operational decisions.
o Example: Executives use BI dashboards to monitor key performance indicators
(KPIs) and make informed decisions.
2. Data Mining:

11
 o Discovering Patterns: Data warehouses provide a foundation for data mining
techniques that uncover hidden patterns and correlations in large datasets.
o Example: Retailers use data mining to analyze customer purchase behaviors and
develop targeted marketing campaigns.
3. Data Integration:
o Consolidated View: Data warehouses integrate data from multiple sources,
providing a comprehensive view of the organization's data.
o Example: Combining sales data from different regions to get a global sales
performance view.
4. Historical Analysis:
o Trend Analysis: Data warehouses store historical data, enabling trend analysis
and forecasting.
o Example: Analyzing sales data from the past decade to predict future sales
trends.
5. Reporting and Querying:
o Efficient Reporting: Data warehouses enable fast and efficient generation of
complex reports and queries.
o Example: Generating monthly financial reports by querying the financial data
warehouse.

Data Mining

Data mining is the process of discovering patterns, correlations, and anomalies in large datasets
to predict outcomes. It involves using algorithms and statistical methods to extract useful
information from data.

Techniques of Data Mining

1. Classification:
o Definition: Assigning items to predefined categories or classes based on their
attributes.
o Example: Classifying customers into different segments (e.g., high-value,
medium-value, low-value) based on purchase history.
2. Clustering:

12
 o Definition: Grouping similar items together based on their attributes without
predefined categories.
o Example: Identifying customer segments with similar purchasing behaviors
using clustering algorithms.
3. Association Rule Learning:
o Definition: Discovering interesting relationships between variables in large
datasets.
o Example: Identifying products frequently bought together (e.g., bread and
butter) using market basket analysis.
4. Regression:
o Definition: Modeling the relationship between a dependent variable and one or
more independent variables to predict outcomes.
o Example: Predicting future sales based on factors like advertising spend and
seasonal trends.
5. Anomaly Detection:
o Definition: Identifying unusual patterns or outliers in data that do not conform
to expected behavior.
o Example: Detecting fraudulent credit card transactions by identifying deviations
from typical spending patterns.
6. Sequential Pattern Mining:
o Definition: Discovering regular sequences or patterns over time within a dataset.
o Example: Analyzing customer purchase sequences to identify common buying
patterns over time.

Business Intelligence (BI)

Business Intelligence (BI) encompasses technologies, processes, and applications used to

analyze an organization's raw data to improve decision-making and business performance.

Components of BI

1. Data Warehousing:
o Central Repository: Storing and managing large volumes of data from various
sources to support BI activities.

13
 o Example: A financial data warehouse consolidates data from accounting,
budgeting, and payroll systems.
2. Data Mining:
o Pattern Discovery: Using advanced algorithms to discover patterns and
relationships in data to generate insights.
o Example: Using data mining to identify factors that contribute to high employee
turnover.
3. Reporting and Querying:
o Information Delivery: Generating reports and responding to queries to provide
actionable insights.
o Example: Monthly sales reports that summarize performance by region and
product line.
4. OLAP (Online Analytical Processing):
o Multidimensional Analysis: Enabling users to analyze data from multiple
perspectives and dimensions.
o Example: Analysing sales data by region, product, and time period using OLAP
cubes.
5. Dashboards and Visualization:
o Visual Insights: Providing visual representations of data to help users
understand complex information quickly.
o Example: A dashboard displaying key performance indicators (KPIs) such as
revenue, profit margins, and customer satisfaction.
6. Performance Management:
o Monitoring and Management: Tracking performance against business goals
and objectives.
o Example: A balanced scorecard that measures performance across financial,
customer, internal process, and learning and growth perspectives.

Benefits of BI

1. Enhanced Decision-Making:
o Data-Driven Insights: BI provides accurate and timely information, enabling
better decision-making.

14
 o Example: Managers use BI reports to identify underperforming products and
make decisions to improve sales.
2. Operational Efficiency:
o Process Optimization: BI helps identify inefficiencies and areas for
improvement in business processes.
o Example: Analysing supply chain data to optimize inventory levels and reduce
costs.
3. Competitive Advantage:
o Market Insights: BI provides insights into market trends and customer
behaviours, helping organizations stay ahead of the competition.
o Example: Identifying emerging market trends and adjusting marketing
strategies accordingly.
4. Improved Customer Satisfaction:
o Personalized Services: BI helps understand customer preferences and
behaviours, allowing for personalized services and better customer experiences.
o Example: Using customer data to create targeted marketing campaigns and
personalized product recommendations.
5. Strategic Planning:
o Long-Term Insights: BI supports strategic planning by providing insights into
long-term trends and performance.
o Example: Analysing historical sales data to inform future business expansion
plans.

****

15