UNIT IV IMPLEMENTATION TECHNIQUES

RAID – File Organization – Organization of Records in Files – Data dictionary Storage – Column
Oriented Storage– Indexing and Hashing –Ordered Indices – B+ tree Index Files – B tree Index Files –
Static Hashing – Dynamic Hashing – Query Processing Overview – Algorithms for Selection, Sorting
and join operations – Query optimization using Heuristics - Cost Estimation.

1. RAID
(Redundant array of independent disks) originally redundant array of inexpensive disks) is a
way of storing the same data in different places on multiple hard disks to protect data in thecase of a
drive failure.
RAID: Redundant Arrays of Independent Disks
Disk organization techniques that manage a large numbers of disks, providing a view of a
single disk of high capacity and high speed by using multiple disks in parallel, and high reliability by
storing data redundantly, so that data can be recovered even if a disk fails
• These levels contain the following characteristics:
• It contains a set of physical disk drives.
• In this technology, the operating system views these separate disks as a single logical disk.
• In this technology, data is distributed across the physical drives of the array.
• Redundancy disk capacity is used to store parity information.
• In case of disk failure, the parity information can be helped to recover the data.

Motivation for RAID

• Just as additional memory in form of cache, can improve the system performance, in thesame
way additional disks can also improve system performance.
• In RAID we can use an array of disks which operates independently since there are many
disks, multiple I/O requests can be handled in parallel if the data required is on separatedisks
• A single I/O operation can be handled in parallel if the data required is distributed across
multiple disks.
Benefits of RAID
• Data loss can be very dangerous for an organization
• RAID technology prevents data loss due to disk failure
• RAID technology can be implemented in hardware or software
• Servers make use of RAID Technology
 RAID LEVEL 0
• RAID level 0 divides data into block units and writes them across a number of disks.
• As data is placed across multiple disks. it is also called ―data Striping‖.
• The advantage of distributing data over disks is that if different I/O requests are pending for
two different blocks of data, then there is a possibility that the requested blocks are on
different disks.
• There is no parity checking of data. So if data in one drive gets corrupted then all the data
would be lost.
• Thus RAID 0 does not support data recovery.
• Spanning is another term that is used with RAID level 0 because the logical disk will span all
the physical drives.
• RAID 0 implementation requires minimum 2 disks.

Advantages
• I/O performance is greatly improved by spreading the I/O load across many channels &
drives.
• Best performance is achieved when data is striped across multiple controllers with only one
driver per controller

Disadvantages
• It is not fault-tolerant, failure of one drive will result in all data in an array being lost

RAID LEVEL 1: Mirroring (or shadowing)

• Also known as disk mirroring, this configuration consists of at least two drives that
duplicate the storage of data. There is no striping.
 • ead performance is improved since either disk can be read at the same time. Write
performance is the same as for single disk storage.
• Every write is carried out on both disks. If one disk in a pair fails, data still available in the
other.
• Data loss would occur only if a disk fails, and its mirror disk also fails before the system is
repaired Probability of combined event is very small.

RAID LEVEL 2:
This configuration uses striping across disks, with some disks storing error checking and correcting
(ECC) information. It has no advantage over RAID 3 and is no longer

• Each bit of data word is written to a data disk drive (4 in this example: 0 to 3).

• Each data word has its Hamming Code ECC word recorded on the ECC disks.
• On Read, the ECC code verifies correct data or corrects single disk errors.
Advantages-
• On the fly' data error correction
• Extremely high data transfer rates possible
• The higher the data transfer rate required, the better the ratio of data disks to ECC disks.

Disadvantages-
• Very high ratio of ECC disks to data disks with smaller word sizes
• Entry level cost very high
• Requires very high transfer rate requirement to justify. No
commercial implementations exist

RAID LEVEL 3: Bit-Interleaved Parity

o A single parity bit is enough for error correction, not just detection, since we know which
disk has failed
o When writing data, corresponding parity bits must also be computed and written to a
parity bit disk
o To recover data in a damaged disk, compute XOR of bits from other disks (including parity
bit disk)
o I/O operation addresses all the drives at the same time, RAID 3 cannot overlap I/O. For
this reason, RAID 3 is best for single-user systems with long record applications.
o The data block is subdivided ('striped') and written on the data disks. Stripe parity is
generated on Writes, recorded on the parity disk and checked on Reads.
o RAID Level 3 requires a minimum of 3 drives to implement.
 Advantages
- Very high Read data transfer rate
- Very high Write data transfer rate

- Disk failure has an insignificant impact on throughput

- Low ratio of ECC (Parity) disks to data disks means high efficiency

Disadvantages
• Transaction rate equal to that of a single disk drive at best (if spindles are synchronized).
• Controller design is fairly complex.

• Very difficult and resource intensive to do as a 'software' RAID

RAID LEVEL 4: Block-Interleaved Parity

• When writing data block, corresponding block of parity bits must also be computed and
written to parity disk
• To find value of a damaged block, compute XOR of bits from corresponding blocks
(including parity block) from other disks.
Each entire block is written onto a data disk. Parity for same rank blocks is generated on Writes,
recorded on the parity disk and checked on Reads. RAID Level 4 requires a minimum of 3 drives to
implement.
Advantages
• Very high Read data transaction rate
• Low ratio of ECC (Parity) disks to data disks means high efficiency
• High aggregate Read transfer rate
• Low ratio of ECC (Parity) disks to data disks means high efficiency

Disadvantages
• Quite complex controller design
• Worst Write transaction rate and Write aggregate transfer rate
• Difficult and inefficient data rebuild in the event of disk failure
• Block Read transfer rate equal to that of a single disk

RAID LEVEL 5:
• RAID 5 uses striping as well as parity for redundancy. It is well suited for heavy read and
low write operations.
• Block-Interleaved Distributed Parity; partitions data and parity among all N + 1 disks,
rather than storing data in N disks and parity in 1 disk
 .

RAID LEVEL 6:
• This technique is similar to RAID 5, but includes a second parity scheme that is distributed
across the drives in the array. The use of additional parity allows the array to continue to
function even if two disks fail simultaneously. However, this extra protection comes at a cost.
• P+Q Redundancy scheme; similar to Level 5, but stores extra redundant information to guard
against multiple disk failures.
• Better reliability than Level 5 at a higher cost; not used as widely.
FILE ORGANIZATION
The database is stored as a collection of files.
o Each file is a sequence of records.
o A record is a sequence of fields.
Classifications of records
o Fixed length record
o Variable length record
(i) Fixed length record approach:
Assume record size is fixed each file has records of one particular type only different files
are used for different relations
o Simple approach
o Record access is simple Example pseudo code type
account = record
account_number char(10);
branch_name char(22);
balance numeric(8);
end
Total bytes 40 for a record

Two problems
- Difficult to delete record from this structure.
- Some record will cross block boundaries, that is part of the record will be stored in one
block and part in another. It would require two block accesses to read or write Reuse the free space
alternatives:

– Move records i + 1, . . ., n to n i, . . . , n – 1
 – do not move records, but link all free records on a free list
– Move the final record to deleted record place.

Free Lists
Store the address of the first deleted record in the file header.
Use this first record to store the address of the second deleted record, and so on
Variable-Length Records

Variable-Length Records arise in database systems in several ways:

• Storage of multiple record types in file.
• Record types that allow variable lengths for one or more fields.
• Record types that allow repeating fields

Byte string representation

Attach an end-of-record control character to the end of each record.

Difficulty with deletion
Disadvantage
It is not easy to reuse space occupied formerly by deleted record.
There is no space in general for records grows longer

Slotted Page Structure

Slotted page header contains:

 – number of record entries
– end of free space in the block
– location and size of each record
Pointer Method

• A variable-length record is represented by a list of fixed-length records, chained together

via pointers.
• Can be used even if the maximum record length is not known.
Disadvantage to pointer structure; space is wasted in all records except the first in a a chain.
Solution is to allow two kinds of block in file:
o Anchor block – contains the first records of chain
o Overflow block – contains records other than those that are the first records of
chains.

Organization of Records in Files

• Sequential – store records in sequential order, based on the value of the search key of
each record
• Heap – a record can be placed anywhere in the file where there is space
 • Hashing – a hash function computed on some attribute of each record; the result
specifies in which block of the file the record should be placed
1. SEQUENTIAL FILE ORGANIZATION
• Suitable for applications that require sequential processing of the entire file
• The records in the file are ordered by a search-key.
Deletion – use pointer chains
Insertion – locate the position where the record is to be inserted – if there is free space insert there
– if no free space, insert the record in an overflow block – In either case, pointer chain must be
updated
Sequential is the easiest method for file organization. In this method, files are stored
sequentially. This method can be implemented in two ways:
1. Pile File Method:
o It is a quite simple method. In this method, we store the record in a sequence, i.e., one after
another. Here, the record will be inserted in the order in which they are inserted into tables.
o In case of updating or deleting of any record, the record will be searched in the memory
blocks. When it is found, then it will be marked for deleting, and the new record is inserted.

Insertion of the new record:

Suppose we have four records R1, R3 and so on upto R9 and R8 in a sequence. Hence,
records are nothing but a row in the table. Suppose we want to insert a new record R2 in the
sequence, then it will be placed at the end of the file. Here, records are nothing but a row in any
table.
2. Sorted File Method:
In this method, the new record is always inserted at the file's end, and then it will sort the
sequence in ascending or descending order. Sorting of records is based on any primary
key or any other key.
In the case of modification of any record, it will update the record and then sort the file,
and lastly, the updated record is placed in the right place.

Insertion of the new record:

Suppose there is a preexisting sorted sequence of four records R1, R3 and so on upto R6 and R7.
Suppose a new record R2 has to be inserted in the sequence, then it will be inserted at the end of
the file, and then it will sort the sequence.

2. HEAP FILE ORGANIZATION

o It is the simplest and most basic type of organization.
o It works with data blocks.
o In heap file organization, the records are inserted at the file's end. When the records are
 inserted, it doesn't require the sorting and ordering of records.
o When the data block is full, the new record is stored in some other block. This new data
block need not to be the very next data block, but it can select any data block in the memory
to store new records. The heap file is also known as an unordered file.
o In the file, every record has a unique id, and every page in a file is of the same size. It is
the DBMS responsibility to store and manage the new records.
 Insertion of a new record
Suppose we have five records R1, R3, R6, R4 and R5 in a heap and suppose we want to
insert a new record R2 in a heap. If the data block 3 is full then it will be inserted in any of the
database selected by the DBMS, let's say data block 1.

If we want to search, update or delete the data in heap file organization, then we need to
traverse the data from staring of the file till we get the requested record.
If the database is very large then searching, updating or deleting of record will be time-
consuming because there is no sorting or ordering of records. In the heap file organization, we
need to check all the data until we get the requested record.

Indexing and Hashing

Defn: Indexing provides the most powerful and flexible tools for organizing files.Indexing
mechanisms used to speed up access to desired data.
➢ E.g., author catalog in library
Search Key - attribute or set of attributes used to look up records in a file.
Indexing has following disadvantages
1. Indexes take up space and time to keep them organized.
2. The amount of I/O increases with the size of index.
An index file consists of records (called index entries) of the form

search-key pointer
Index files are typically much smaller than the original file.

There are two basic kinds of indices:

• Ordered indices: Based on a sorted ordering of the values.
• Hash indices: Based on a uniform distribution of values across a range of buckets. The bucket
to which a value is assigned is determined by a function, called a hash function.

Each technique must be evaluated on the basis of these factors:

• Access types: The types of access that are supported efficiently. Access types can include finding
records with a specified attribute value and finding records whose attribute values fall in a specified
range.

• Access time: The time it takes to find a particular data item, or set of items, using the technique in
question.

• Insertion time: The time it takes to insert a new data item. This value includes the time it takes to
find the correct place to insert the new data item, as well as the time it takes to update the index structure.

• Deletion time: The time it takes to delete a data item. This value includes the time it takes to find the
item to be deleted, as well as the time it takes to update the index structure.

• Space overhead: The additional space occupied by an index structure. Provided that the amount of
additional space is moderate, it is usually worthwhile to sacrifice the space to achieve improved
performance.

Ordered Indices:
Each index structure is associated with a particular search key. An ordered index stores the
values of the search keys in sorted order, and associates with each search key the records that contain
it. A file may have several indices, on different search keys. If the file containing the records is
sequentially ordered, a clustering index is an index, whose search key also defines the sequential order
of the file. Clustering indices are also called primary indices. The search key of a clustering index is
often the primary key. Indices whose search key specifies an order different from the sequential order
of the file are called nonclustering indices, or secondary indices.
Index-sequential file: ordered sequential file with a primary index.

Dense and Sparse Indices:

Dense index: In a dense index, an index entry appears for every search-key value in the file. In a dense
clustering index, the index record contains the search-key value and a pointer to the first data record
with that search-key value. The rest of the records with the same search-key value would be stored
sequentially after the first record, since, because the index is a clustering one, records are sorted on the
same search key.
Figure Dense index.

Sparse index: In a sparse index, an index entry appears for only some of the search-key values. Sparse
indices can be used only if the relation is stored in sorted order of the search key, that is, if the index is
a clustering index. To locate a record, we find the index entry with the largest search-key value that is
less than or equal to the search-key value for which we are looking.
Figure :Sparse index.

• Less space and less maintenance overhead for insertions and deletions.
• Generally slower than dense index for locating records.
• Good tradeoff: sparse index with an index entry for every block in file, corresponding to
least search-key value in the block.
Multilevel Indices
Indices with two or more levels are called multilevel indices. Searching for records with a
multilevel index requires significantly fewer I/O operations than does searching for records by binary
search.
• If primary index does not fit in memory, access becomes expensive.
• To reduce number of disk accesses to index records, treat primary index kept on disk as a
sequential file and construct a sparse index on it.
o outer index – a sparse index of primary index
o inner index – the primary index file
• If even outer index is too large to fit in main memory, yet another level of index can be
created, and so on.
• Indices at all levels must be updated on insertion or deletion from the file.

Figure Two-level sparse index.

Index Update:

Insertion.
Dense indices:
1. If the search-key value does not appear in the index, the system inserts an index entry with the
search-key value in the index at the appropriate position.
2. Otherwise the following actions are taken:
a. If the index entry stores pointers to all records with the same search key value, the system
adds a pointer to the new record in the index entry.
b. Otherwise, the index entry stores a pointer to only the first record with the search- key
value. The system then places the record being inserted after the other records with the same
search-key values.
Sparse indices: We assume that the index stores an entry for each block.If the system creates a new
block, it inserts the first search-key value appearing in the new block into the index. On the other hand,
if the new record has the least search-key value in its block, the system updates the index entry pointing
to the block; if not, the system makes no change to the index.
Deletion.
Dense indices:
1. If the deleted record was the only record with its particular search-key value, then the system
deletes the corresponding index entry from the index.
2. Otherwise the following actions are taken:
a. If the index entry stores pointers to all records with the same search key value, the system deletes
the pointer to the deleted record from the index entry.
b. Otherwise, the index entry stores a pointer to only the first record with the search-key value. In this
case, if the deleted record was the first record with the search-key value, the system updates the index
entry to point to the next record.
Sparse indices:
1. If the index does not contain an index entry with the search-key value of the deleted record, nothing
needs to be done to the index.
2. Otherwise the system takes the following actions:
a. If the deleted record was the only record with its search key, the system replaces the corresponding
index record with an index record for the next search-key value (in search-key order). If the next search-
key value already has an index entry, the entry is deleted
instead of being replaced.
b. Otherwise, if the index entry for the search-key value points to the record being deleted, the system
updates the index entry to point to the next record with the same search-key value.

Secondary Indices:
Secondary indices must be dense, with an index entry for every search-key value, and a pointer
to every record in the file. A secondary index on a candidate key looks just like a dense clustering index,
except that the records pointed to by successive values in the index are not stored sequentially. In
general, however, secondary indices may have a different structure from clustering indices. If the search
key of a clustering index is not a candidate key, it suffices if the index points to the first record with a
particular value for the search key, since the other records can be fetched by a sequential scan of the
file.
In contrast, if the search key of a secondary index is not a candidate key, it is not enough to
point to just the first record with each search-key value. The remaining records with the same search-
key value could be anywhere in the file, since the records are ordered by the search key of the clustering
index, rather than by the search key of the secondary index. Therefore, a secondary index must contain
pointers to all the records.
Advantages:
Secondary indices improve the performance of queries that use keys other than the search key of the
clustering index.
 B+-Tree Index Files

Disadvantage of indexed-sequential files:

Performance degrades as file grows, since many overflow blocks get created. Periodic
reorganization of entire file is required.
Advantage of B+-tree index files:
Automatically reorganizes itself with small, local, changes, in the face of insertions and
deletions. Reorganization of entire file is not required to maintain performance.
Disadvantage of B+-trees:
Extra insertion and deletion overhead, space overhead.

Defn: A B+-tree index takes the form of a balanced tree in which every path from the root of the tree
to a leaf of the tree is of the same length. Each non leaf node in the tree has between n/2 and n children,
where n is fixed for a particular tree.
Structure of a B+-Tree:
Figure shows a typical node of a B+-tree. It contains up to n − 1 search-key values K1, K2, . . .
, Kn−1, and n pointers P1, P2, . . . , Pn. The search-key values within a node are kept in sorted order;
thus,if i < j, then Ki < K j .

Sample account file

Properties of a leaf node:

 1. For i = 1, 2, . . ., n–1, pointer Pi either points to a file record with search-key value Ki, or to a
bucket of pointers to file records, each record having search-key value Ki. Only need bucket
structure if search-key does not form a primary key.
2. If Li, Lj are leaf nodes and i < j, Li’s search-key values are less than Lj’s search-key values.
3. Pn points to next leaf node in search-key order.

Non-Leaf Nodes:
Non leaf nodes form a multi-level sparse index on the leaf nodes.
For a non-leaf node with m pointers:
▪ All the search-keys in the sub tree to which P1 points are less than K1
▪ For 2  i  n – 1, all the search-keys in the subtree to which Pi points have
values greater than or equal to Ki–1 and less than Km–1
• The number of pointers in a nodeis called the fanout of the node. Nonleaf nodes are also
referred to as internal nodes.

Example of a B+-tree

B+-tree for account file (n = 3)

B+-tree for account file (n = 5)
Queries on B+-Trees
Find all records with a search-key value of k.
1. Start with the root node
1. Examine the node for the smallest search-key value > k.
2. If such a value exists, assume it is Kj. Then follow Pi to the child node
3. Otherwise k  Km–1, where there are m pointers in the node. Then follow Pm to
the child node.
2. If the node reached by following the pointer above is not a leaf node, repeat the above
procedure on the node, and follow the corresponding pointer.
3. Eventually reach a leaf node. If for some i, key Ki = k follow pointer Pi to the desired
record or bucket. Else no record with search-key value k exists.
Updates on B+-Trees:
When a record is inserted into, or deleted from a relation, indices on the relation must be updated
correspondingly. Recall that updates to a record can be modelled as a deletion of the old record followed
by insertion of the updated record.
Insertion:
1. Find the leaf node in which the search-key value would appear.
2. If the search-key value is already there in the leaf node, record is added to file and if necessary
a pointer is inserted into the bucket.
3. If the search-key value is not there, then add the record to the main file and create a bucket if
necessary. Then:
o If there is room in the leaf node, insert (key-value, pointer) pair in the leaf node
o Otherwise, split the node
• Splitting a node:
o take the n(search-key value, pointer) pairs (including the one being inserted) in sorted
order. Place the first  n/2  in the original node, and the rest in a new node.
 o let the new node be p, and let k be the least key value in p. Insert (k,p) in the parent of
the node being split. If the parent is full, split it and propagate the split further up.

Fig:Result of splitting node containing Brighton and Downtown on inserting Clearview

Fig:B+-Tree before and after insertion of “Clearview”

Deletion:
1. Find the record to be deleted, and remove it from the main file and from the bucket (if present)
2. Remove (search-key value, pointer) from the leaf node if there is no bucket or if the bucket has
become empty
3. If the node has too few entries due to the removal, and the entries in the node and a sibling fit
into a single node, then
o Insert all the search-key values in the two nodes into a single node (the one on the left),
and delete the other node.
o Delete the pair (Ki–1, Pi), where Pi is the pointer to the deleted node, from its parent,
recursively using the above procedure
• Otherwise, if the node has too few entries due to the removal, and the entries in the node and a
sibling fit into a single node, then
o Redistribute the pointers between the node and a sibling such that both have more than
the minimum number of entries.
o Update the corresponding search-key value in the parent of the node.
• The node deletions may cascade upwards till a node which has n/2  or more pointers is found.
If the root node has only one pointer after deletion, it is deleted and the sole child becomes the
root.
Fig: Before and after deleting “Downtown
B-Tree Index Files:

B-tree indices are similar to B+-tree indices. The primary distinction between the two
approaches is that a B-tree eliminates the redundant storage of search-key values. Search keys in nonleaf
nodes appear nowhere else in the B-tree; an additional pointer field for each search key in a nonleaf
node must be included.

Figure Typical nodes of a B-tree. (a) Leaf node. (b) Nonleaf node.

Fig:B-tree and B+-tree on same data

Advantages of B-Tree indices:
➢ May use less tree nodes than a corresponding B+-Tree.
➢ Sometimes possible to find search-key value before reaching leaf node.
Disadvantages of B-Tree indices:
➢ Only small fraction of all search-key values are found early
➢ Non-leaf nodes are larger, so fan-out is reduced. Thus B-Trees typically have greater
depth than corresponding B+-Tree
➢ Insertion and deletion more complicated than in B+-Trees
➢ Implementation is harder than B+-Trees.

HASHING

Static Hashing:
One disadvantage of sequential file organization is that we must access an index structure to
locate data, or must use binary search, and that results in more I/O operations. File organizations based
on the technique of hashing allow us to avoid accessing an index structure. Hashing also provides a
way of constructing indices.
1. A bucket is a unit of storage containing one or more records (a bucket is typically a disk block).
2. In a hash file organization we obtain the bucket of a record directly from its search-key value
using a hash function.
3. Hash function h is a function from the set of all search-key values K to the set of all bucket
addresses B.
4. Hash function is used to locate records for access, insertion as well as deletion.
5. Records with different search-key values may be mapped to the same bucket; thus entire bucket
has to be searched sequentially to locate a record.
Hash file organization of account file, using branch-name as key.
 a) There are 10 buckets,
b) The binary representation of the ith character is assumed to be the integer i.
c) The hash function returns the sum of the binary representations of the characters modulo 10
➢ E.g. h(Perryridge) = 5 h(Round Hill) = 3 h(Brighton) = 3
Hash function:
The worst possible hash function maps all search-key values to the same bucket. Such a function is
undesirable because all the records have to be kept in the same bucket. An ideal hash function distributes
the stored keys uniformly across all the buckets, so that every bucket has the same number of records.
The distribution is uniform: That is, the hash function assigns each bucket the same number of search-
key values from the set of all possible search-key values.
The distribution is random: That is, in the average case, each bucket will have nearly the same number
of values assigned to it, regardless of the actual distribution of search-key values.
Typical hash functions perform computation on the internal binary representation of the search-
key. For example, for a string search-key, the binary representations of all the characters in the string
could be added and the sum modulo the number of buckets could be returned.
Handling of Bucket Overflows:
If the bucket does not have enough space, a bucket overflow is said to occur. Bucket overflow
can occur for several reasons:
Bucket overflow can occur because of
➢ Insufficient buckets
➢ Skew in distribution of records. This can occur due to two reasons:
 multiple records have same search-key value
 chosen hash function produces non-uniform distribution of key values
Although the probability of bucket overflow can be reduced, it cannot be eliminated; it is handled by
using overflow buckets.
There are two types of hashing
1. Closed hashing: other name is Overflow chaining – the overflow buckets of a given bucket
are chained together in a linked list.

2. Open hashing: The set of buckets is fixed, and there are no overflow chains. Instead, if a bucket
is full, the system inserts records in some other bucket in the initial set of buckets B. It is not
suitable for database applications.
Hash Indices:
Hashing can be used not only for file organization, but also for index-structure creation.
Defn: A hash index organizes the search keys, with their associated pointers, into a hash file structure.
Hash indices are always secondary indices
o if the file itself is organized using hashing, a separate primary hash index on it using the
same search-key is unnecessary.
o However, we use the term hash index to refer to both secondary index structures and
hash organized files.
Deficiencies of Static Hashing
In static hashing, function h maps search-key values to a fixed set of B of bucket addresses.
➢ Databases grow with time. If initial number of buckets is too small, performance will
degrade due to too much overflows.
➢ If file size at some point in the future is anticipated and number of buckets allocated
accordingly, significant amount of space will be wasted initially.
➢ If database shrinks, again space will be wasted.
➢ One option is periodic re-organization of the file with a new hash function, but it is very
expensive.
These problems can be avoided by using techniques that allow the number of buckets to be modified
dynamically.

Dynamic Hashing:
Defn: Dynamic hashing techniques allow the hash function to be modified dynamically to
accommodate the growth or shrinkage of the database.
Extendable hashing – one form of dynamic hashing
➢ Hash function generates values over a large range — typically b-bit integers, with b =
32.
 ➢ At any time use only a prefix of the hash function to index into a table of bucket
addresses.
➢ Let the length of the prefix be i bits, 0  i  32.
➢ Bucket address table size = 2i. Initially i = 0
➢ Value of i grows and shrinks as the size of the database grows and shrinks.
➢ Multiple entries in the bucket address table may point to a bucket.
➢ Thus, actual number of buckets is < 2i
 The number of buckets also changes dynamically due to coalescing and
splitting of buckets.

Figure General extendable hash structure.

- Each bucket j stores a value ij; all the entries that point to the same bucket have the same
values on the first ij bits.
- To locate the bucket containing search-key Kj:
1. Compute h(Kj) = X
2. Use the first i high order bits of X as a displacement into bucket address table, and follow the
pointer to appropriate bucket
To insert a record with search-key value Kj
➢ follow same procedure as look-up and locate the bucket, say j.
➢ If there is room in the bucket j insert record in the bucket.
➢ Else the bucket must be split and insertion re-attempted
To split a bucket j when inserting record with search-key value Kj
1. If i > ij (more than one pointer to bucket j)
 ➢ allocate a new bucket z, and set ij and iz to the old ij -+ 1.
➢ make the second half of the bucket address table entries pointing to j to point to z
➢ remove and reinsert each record in bucket j.
➢ recompute new bucket for Kj and insert record in the bucket (further splitting is
required if the bucket is till full)
2. If i = ij (only one pointer to bucket j)
➢ increment i and double the size of the bucket address table.
➢ replace each entry in the table by two entries that point to the same bucket.
➢ recompute new bucket address table entry for Kj
Now i > ij so use the first case above.
When inserting a value, if the bucket is full after several splits (that is, i reaches some limit b) create
an overflow bucket instead of splitting bucket entry table further.
To delete a key value,
➢ locate it in its bucket and remove it.
➢ The bucket itself can be removed if it becomes empty (with appropriate updates to the
bucket address table).
➢ Coalescing of buckets can be done (can coalesce only with a “buddy” bucket having
same value of ij and same ij –1 prefix, if it is present)
➢ Decreasing bucket address table size is also possible
n Note: decreasing bucket address table size is an expensive operation and should
be done only if number of buckets becomes much smaller than the size of the
table
Use of Extendable Hash Structure: Example

Initial Hash structure, bucket size = 2

Hash structure after insertion of one Brighton and two Downtown records

Hash structure after insertion of Mianus record

Hash structure after insertion of three Perryridge records

Hash structure after insertion of Redwood and Round Hill records

Extendable Hashing vs. Other Schemes

Benefits of extendable hashing:

➢ Hash performance does not degrade with growth of file
➢ Minimal space overhead
Disadvantages of extendable hashing:
 ➢ Extra level of indirection to find desired record
➢ Bucket address table may itself become very big (larger than memory)
 Need a tree structure to locate desired record in the structure!
➢ Changing size of bucket address table is an expensive operation
Linear hashing is an alternative mechanism which avoids these disadvantages at the possible cost of
more bucket overflows.
Comparison of Ordered Indexing and Hashing:
➢ Cost of periodic re-organization
➢ Relative frequency of insertions and deletions
➢ Is it desirable to optimize average access time at the expense of worst-case access
time?
➢ Expected type of queries:
Hashing is generally better at retrieving records having a specified value of the key.
If range queries are common, ordered indices are to be preferred.

ALGORITHMS FOR QUERY PROCESSING AND OPTIMIZATION

QUERY PROCESSING

A query expressed in a high-level query language such as SQL must first be scanned, parsed, and
validated. The scanner identifies the language tokens-such as SQL keywords, attribute names, and
relation names, whereas the parser checks the query syntax . The query must also be validated, by
checking that all attribute and relation names are valid .

An internal representation of the query is then created, usually as a tree data structure called a query
tree. It is also possible to represent the query using a graph data structure called a query graph.
Then DBMS select the execution strategy to execute the query.
Defn: Query Optimization
A query typically has many possible execution strategies, and the process of choosing a suit-able one
for processing a query is known as query optimization.
The query optimizer module has the task of producing an execution plan, and the code
generator generates the code to execute that plan. The runtime database processor has the task of
running the query code,whether in compiled or interpreted mode, to produce the query result.

QUERY OPTIMIZATION
There are two main techniques for implementing query optimization.
1. Heuristic rules for ordering the operations in a query execution strategy
A heuristic is a rule that works well in most cases but is not guaranteed to work well in every
possible case. The rules typically reorder the operations in a query tree.
2. Systematically estimating the cost of different execution strategies
To select a lowest cost estimate.
This two techniques are usually combined in a query optimizer.
TRANSLATING SQL QUERIES INTO RELATIONAL ALGEBRA

An SQL query is first translated into an equivalent extended relational algebra expression represented
as a query tree data structure.
SQL queries are decomposed into query blocks. Nested queries within a query are identified as separate
query blocks.
Consider the following SQL query on the EMPLOYEE relation in the above Figure
SELECT LNAME, FNAME FROM EMPLOYEE WHERE SALARY > (SELECT
MAX (SALARY) FROM EMPLOYEE WHERE DNO=5);
This query includes a nested sub query and hence would be decomposed into two blocks.
The inner block is
(SELECT MAX (SALARY) FROM EMPLOYEE WHERE DNO=5)
and the outer block is
SELECT LNAME, FNAME FROM EMPLOYEE WHERE SALARY > C
where c represents the result returned from the inner block. The inner block could be
translated into the extended relational algebra expression
ℱMAX SALARY (σDNO=5 (EMPLOYEE))
outer block into the expression
πLNAME, FNAME (σSALARY>C(EMPLOYEE))

ALGORITHMS FOR EXTERNAL SORTING:

Sorting is one of the primary algorithms used in query processing. eg:order-by clause
External sorting:
Refers to sorting algorithms that are suitable for large files of records stored on disk that do not fit
entirely in main memory, such as most database files. The typical external sorting algorithm uses a sort-
merge strategy, which starts by sorting small subfiles-called runs-of the main file and then merges the
sorted runs, creating larger sorted subfiles that are merged in turn.
Sorting phase:
The size of a run and number of initial runs (nR) is dictated by the number of file blocks (b) and the
available buffer space (nB).For example, if nB = 5 blocks and the size of the file b = 1024 blocks, then
nR = (b/nB) , or 205 initial runs each of size 5 blocks. Hence, after the sort phase, 205 sorted runs
are stored as temporary subfiles on disk.
In the merging phase, the sorted runs are merged during one or more passes. The degree
of merging (d M ) is the number of runs that can be merged together in each pass.
dM = Min (nB-1, nR);
no of passes nP = (logdM(nR))

ALGORITHMS FOR SELECT AND JOIN OPERATIONS:

I. Implementing the SELECT Operation:
There are many options for executing a SELECT operation.
(OP1): SSN='123456789' (EMPLOYEE)

(OP2): DNUMBER>5(DEPARTMENT)

(OP3): DNO=5(EMPLOYEE)

(OP4): DNO=5 AND SALARY>30000 AND SEX=F(EMPLOYEE)

(OP5): ESSN=123456789 AND PNO=10(WORKS_ON)

Search Methods for Simple Selection:

S1 Linear search (brute force):
Retrieve every record in the file, and test whether its attribute values satisfy the selection condition.
S2 Binary search:
If the selection condition involves an equality comparison on a key attribute on which the file is ordered,
binary search (which is more efficient than linear search) can be used. (See OP1).
S3 Using a primary index or hash key to retrieve a single record:
If the selection condition involves an equality comparison on a key attribute with a primary index (or a
hash key), use the primary index (or the hash key) to retrieve the record.
S4 Using a primary index to retrieve multiple records:
If the comparison condition is >, ≥, <, or ≤ on a key field with a primary index, use the index to find
the record satisfying the corresponding equality condition, then retrieve all subsequent records in the
(ordered) file.
S5 Using a clustering index to retrieve multiple records:
If the selection condition involves an equality comparison on a non-key attribute with a clustering index,
use the clustering index to retrieve all the records satisfying the selection condition.
S6 Using a secondary (B+-tree) index:
On an equality comparison, this search method can be used to retrieve a single record if the indexing
field has unique values (is a key) or to retrieve multiple records if the indexing field is not a key.
Search Methods for Complex Selection:
S7 Conjunctive selection:
If an attribute involved in any single simple condition in the conjunctive condition has an access path
that permits the use of one of the methods S2 to S6, use that condition to retrieve the records and then
check whether each retrieved record satisfies the remaining simple conditions in the conjunctive
condition.
S8 Conjunctive selection using a composite index
If two or more attributes are involved in equality conditions in the conjunctive condition and a
composite index (or hash structure) exists on the combined field, we can use the index directly.
S9 Conjunctive selection by intersection of record pointers:
This method is possible if secondary indexes are available on all (or some of) the fields involved in
equality comparison conditions in the conjunctive condition and if the indexes include record pointers
(rather than block pointers).
II. Implementing the JOIN Operation
Join (EQUIJOIN, NATURAL JOIN)
two–way join: a join on two files
e.g. R A=B S
multi-way joins: joins involving more than two files.
e.g. R A=B S C=D T
Examples
(OP6): EMPLOYEE DNO=DNUMBER DEPARTMENT
(OP7): DEPARTMENT MGRSSN=SSN EMPLOYEE
Methods for Implementing Joins:
J1 Nested-loop join (brute force):
For each record t in R (outer loop), retrieve every record s from S (inner loop) and test whether the two
records satisfy the join condition t[A] = s[B].
J2 Single-loop join (Using an access structure to retrieve the matching records):
If an index (or hash key) exists for one of the two join attributes — say, B of S — retrieve each record
t in R, one at a time, and then use the access structure to retrieve directly all matching records s from S
that satisfy s[B] = t[A].
J3 Sort-merge join:
If the records of R and S are physically sorted (ordered) by value of the join attributes A and B,
respectively, we can implement the join in the most efficient way possible.
J4 Hash-join :
The records of files R and S are both hashed to the same hash file , using the same hashing function on
the join attributes A of R and B of S as hash keys.
A single pass through the file with fewer records (say, R) hashes its records to the hash file buckets.
Factors affecting JOIN performance
•Available buffer space
•Join selection factor
•Choice of inner VS outer relation
Other types of JOIN algorithms
Partition hash join
Partitioning phase:
Each file (R and S) is first partitioned into M partitions using a partitioning hash function on the join
attributes:
R1 , R2 , R3 , ...... Rm and S1 , S2 , S3 , ...... Sm
Minimum number of in-memory buffers needed for the partitioning phase: M+1.
A disk sub-file is created per partition to store the tuples for that partition.
Joining or probing phase:
Involves M iterations, one per partitioned file.
Iteration i involves joining partitions Ri and Si.
Using Heuristics in Query Optimization
Process for heuristics optimization
•The parser of a high-level query generates an initial internal representation;
•Apply heuristics rules to optimize the internal representation.
•A query execution plan is generated to execute groups of operations based on the access paths available
on the files involved in the query.
The main heuristic is to apply first the operations that reduce the size of intermediate results.
E.g., Apply SELECT and PROJECT operations before applying the JOIN or other binary operations.
Query tree:
A tree data structure that corresponds to a relational algebra expression. It represents the input relations
of the query as leaf nodes of the tree, and represents the relational algebra operations as internal nodes.
An execution of the query tree consists of executing an internal node operation whenever its operands
are available and then replacing that internal node by the relation that results from executing the
operation.
Query graph:
A graph data structure that corresponds to a relational calculus expression. It does not indicate an order
on which operations to perform first. There is only a single graph corresponding to each query.
Example:
For every project located in ‘Stafford’, retrieve the project number, the controlling department number
and the department manager’s last name, address and birthdate.
Relation algebra:
PNUMBER, DNUM, LNAME, ADDRESS, BDATE ((( PLOCATION=_STAFFORD_(PROJECT))

DNUM=DNUMBER (DEPARTMENT)) MGRSSN=SSN (EMPLOYEE))

SQL query:
Q2: SELECT P.NUMBER,P.DNUM,E.LNAME,
E.ADDRESS, E.BDATE
FROM PROJECT AS P,DEPARTMENT AS D,
EMPLOYEE AS E
WHERE P.DNUM=D.DNUMBER AND
D.MGRSSN=E.SSN AND
P.PLOCATION=‘STAFFORD’;
Heuristic Optimization of Query Trees:
The same query could correspond to many different relational algebra expressions — and hence many
different query trees.
The task of heuristic optimization of query trees is to find a final query tree that is efficient to execute.
Example:
Q: SELECT LNAME
FROM EMPLOYEE, WORKS_ON, PROJECT
WHERE PNAME = ‘AQUARIUS’ AND
PNMUBER=PNO AND ESSN=SSN
AND BDATE > ‘1957-12-31’;
General Transformation Rules for Relational Algebra Operations:
1. Cascade of : A conjunctive selection condition can be broken up into a cascade (sequence) of
individual operations:
c1 AND c2 AND ... AND cn(R) = c1 ( c2 (...( cn(R))...) )
2. Commutativity of : The operation is commutative:
c1 ( c2(R)) = c2 ( c1(R))
3. Cascade of : In a cascade (sequence) of operations, all but the last one can be ignored:
List1 ( List2 (...( Listn(R))...) ) = List1(R)
4. Commuting with : If the selection condition c involves only the attributes A1, ..., An in the
projection list, the two operations can be commuted:
 ( c (R)) = c ( A1, A2, ..., An (R))
A1, A2, ..., An

5. Commutativity of ( and x ): The operation is commutative as is the x operation:

R C S = S C R; R x S = S x R
6. Commuting with (or x ): If all the attributes in the selection condition c involve only the attributes
of one of the relations being joined—say, R—the two operations can be commuted as follows:
c(R S ) = ( c (R)) S
Alternatively, if the selection condition c can be written as (c1 and c2), where condition c1 involves
only the attributes of R and condition c2 involves only the attributes of S, the operations commute as
follows:
c(R S ) = ( c1 (R)) ( c2 (S))
7. Commuting with (or x): Suppose that the projection list is L = {A1, ..., An, B1, ..., Bm}, where
A1, ..., An are attributes of R and B1, ..., Bm are attributes of S. If the join condition c involves only
attributes in L, the two operations can be commuted as follows:
L (R C S) =( A1, ..., An (R)) C ( B1, ..., Bm (S))
If the join condition C contains additional attributes not in L, these must be added to the projection list,
and a final operation is needed.
8. Commutativity of set operations: The set operations υ and ∩ are commutative but “–” is not.
9. Associativity of , x, υ, and ∩ : These four operations are individually associative; that is, if
stands for any one of these four operations (throughout the expression), we have
(R S) T = R (S T)
10. Commuting with set operations: The operation commutes with υ , ∩ , and –. If stands for
any one of these three operations, we have
c(R S ) = ( c (R)) ( c (S))

Summary of Heuristics for Algebraic Optimization:

I.The main heuristic is to apply first the operations that reduce the size of intermediate results.
2. Perform select operations as early as possible to reduce the number of tuples and perform project
operations as early as possible to reduce the number of attributes.
3. The select and join operations that are most restrictive should be executed before other similar
operations.
Query Execution Plans
An execution plan for a relational algebra query consists of a combination of the relational algebra
query tree and information about the access methods to be used for each relation as well as the methods
to be used in computing the relational operators stored in the tree.
Materialized evaluation: the result of an operation is stored as a temporary relation.
Pipelined evaluation: as the result of an operator is produced, it is forwarded to the next operator in
sequence.
Using Selectivity and Cost Estimates in Query Optimization
Cost-based query optimization :
Estimate and compare the costs of executing a query using different execution strategies and choose
the strategy with the lowest cost estimate.
Issues
Cost function
Number of execution strategies to be considered
Cost Components for Query Execution
•Access cost to secondary storage
•Storage cost
•Computation cost
•Memory usage cost
•Communication cost

Catalog Information Used in Cost Functions

Information about the size of a file
•number of records (tuples) (r),
•record size (R),
•number of blocks (b)
•blocking factor (bfr)
Information about indexes and indexing attributes of a file
•Number of levels (x) of each multilevel index
•Number of first-level index blocks (bI1)
•Number of distinct values (d) of an attribute
•Selectivity (sl) of an attribute
•Selection cardinality (s) of an attribute. (s = sl * r)
Examples of Cost Functions for SELECT
S1. Linear search (brute force) approach
CS1a = b;
For an equality condition on a key, CS1a = (b/2) if the record is found; otherwise CS1a = b.
S2. Binary search:
CS2 = log2b + (s/bfr) _1
For an equality condition on a unique (key) attribute, CS2 =log2b
S3. Using a primary index (S3a) or hash key (S3b) to retrieve a single record
CS3a = x + 1; CS3b = 1 for static or linear hashing;
CS3b = 1 for extendible hashing;
S4. Using an ordering index to retrieve multiple records:
For the comparison condition on a key field with an ordering index, CS4 = x + (b/2)
S5. Using a clustering index to retrieve multiple records:
CS5 = x + ┌ (s/bfr) ┐
S6. Using a secondary (B+-tree) index:
For an equality comparison, CS6a = x + s;
For an comparison condition such as >, <, >=, or <=,
CS6a = x + (bI1/2) + (r/2)
S7. Conjunctive selection:
Use either S1 or one of the methods S2 to S6 to solve.
S8. Conjunctive selection using a composite index:
Same as S3a, S5 or S6a, depending on the type of index
Examples of Cost Functions for JOIN
Join selectivity (js)
js = | (R C S) | / | R x S | = | (R C S) | / (|R| * |S |)
If condition C does not exist, js = 1;
If no tuples from the relations satisfy condition C, js = 0;
Usually, 0 <= js <= 1;
Size of the result file after join operation
| (R C S) | = js * |R| * |S |
J1. Nested-loop join:
CJ1 = bR + (bR*bS) + ((js* |R|* |S|)/bfrRS)
(Use R for outer loop)
J2. Single-loop join (using an access structure to retrieve the matching record(s))
If an index exists for the join attribute B of S with index levels xB, we can retrieve each record s in R
and then use the index to retrieve all the matching records t from S that satisfy t[B] = s[A].
The cost depends on the type of index.
For a secondary index,
CJ2a = bR + (|R| * (xB + sB)) + ((js* |R|* |S|)/bfrRS);
For a clustering index,
CJ2b = bR + (|R| * (xB + (sB/bfrB))) + ((js* |R|* |S|)/bfrRS);
For a primary index,
CJ2c = bR + (|R| * (xB + 1)) + ((js* |R|* |S|)/bfrRS);
If a hash key exists for one of the two join attributes — B of S
CJ2d = bR + (|R| * h) + ((js* |R|* |S|)/bfrRS);
J3. Sort-merge join:
CJ3a = CS + bR + bS + ((js* |R|* |S|)/bfrRS);
(CS: Cost for sorting files)