RAC 12c optimization

By
Riyaj Shamsudeen

©OraInternals Riyaj Shamsudeen

Me

 20+ years using Oracle

products/DBA
 OakTable member
 Oracle ACE Director
 Specializes in RAC,
performance
tuning and Internals.
 Slowly in to BigData

 Email:
rshamsud@orainternals.com
 Blog :
orainternals.wordpress.com
 Web: www.orainternals .com
©OraInternals Riyaj Shamsudeen 2
Process architecture
RAC message flow
BL resources

 BL locks protects database blocks.

 BL resource follows a naming convention of

[Block_id ] [file_id], BL

[0xe7c2] [0x1] , [BL]

Block_id = 59330. File_id = 1

[ 0xe7c2 = 59330]

©OraInternals Riyaj Shamsudeen 5

Single block read

 Block is not in any buffer cache. LMS grants a PR mode lock on the
resource and asks FG to read from the disk.

FG FG FG
1
LMD LMD LMD
2
LMS LMS LMS

GRD Buffer 3 GRD Buffer GRD Buffer

SGA SGA SGA

Directory node for

[0x1ad3][0x7],[BL] the resource

FG – Foreground Process
PR mode lock LMD – Lock Manager Daemon
GRD – Global Resource Directory
Disk files

Demo: demo_01a.sql
©OraInternals Riyaj Shamsudeen 6
Trace lines

WAIT #18446741324875049632: nam='gc cr grant 2-way' ela= 499 p1=7 p2=6867 p3=1 obj#=76484
tim=4597940025
WAIT #18446741324875049632: nam='db file sequential read' ela= 758 file#=7 block#=6867 blocks=1
obj#=76484 tim=4597941129

 PR mode

KJBLNAME KJBLNAME2 KJBLGRANT KJBLROLE KJBLREQUES

-------------------- -------------------- ---------- -------- ----------
[0x1ad3][0x7],[BL] 6867,7,BL KJUSERPR 0 KJUSERNL

©OraInternals Riyaj Shamsudeen 7

 GCS structures
A resource structure created in the directory
instance, a lock created in instance 2

X$bh BH Buffer

X$le LE

Shadow Shadow
X$kjbl [0x1ac4][0x7],[BL] [0x1ac4][0x7],[BL]

X$kjbr Resource
[0x1ac4][0x7],[BL] A shadow structure setup in instance
1 to keep track of the resource.
Instance 2 (directory instance) Instance 1

Demo: tc_one_row.sql, gcs_locks.sql gcs_resources.sql

©OraInternals Riyaj Shamsudeen 8
Single block transfer -2 way
 Block is in the directory instance in a compatible mode. Both block
transfer and grant performed by the LMS process running in instance 2.

FG FG FG
2
LMD LMD LMD
1
LMS LMS LMS

GRD Buffer GRD Buffer GRD Buffer

SGA SGA SGA

[0x1ad3][0x7],[BL]

[0x1ad3][0x7],[BL]
PR mode lock

PR mode lock
Disk files FG – Foreground Process
LMD – Lock Manager Daemon
GRD – Global Resource Directory

©OraInternals Riyaj Shamsudeen 9

 GCS Directory instance

X$bh BH Buffer

X$le LE

Client
X$kjbl [0x1ac4][0x7],[BL]

X$kjbr Resource
[0x1ac4][0x7],[BL] A resource structure already exists still
that block is in the buffer cache.
Instance 2 (Directory instance)

Demo: demo_01a.sql and demo_01b.sql

©OraInternals Riyaj Shamsudeen 10
 GCS shadows

X$bh BH Buffer BH Buffer

X$le LE LE

Client Shadow Shadow

X$kjbl [0x1ac4][0x7],[BL] [0x1ac4][0x7],[BL] [0x1ac4][0x7],[BL]

A set of GCS shadows (locks) setup

X$kjbr Resource on the resource in directory instance
[0x1ac4][0x7],[BL] and the requesting instance.

Instance 2 (directory instance) Instance 1

Demo: demo_01a.sql and demo_0a.sql

©OraInternals Riyaj Shamsudeen 11
Single block transfer -3 way
 Block is in the buffer cache of instance 3. Instance 2 is the directory
instance of the resource. LMS process transfers the blocks from instance
3 over the interconnect.
FG
3 FG FG

LMD 1 LMD LMD

LMS LMS LMS

2
GRD Buffer GRD Buffer GRD Buffer
SGA SGA SGA

Directory instance [0x1ad3][0x7],[BL]

[0x1ad3][0x7],[BL] for the resource
PR mode lock
PR mode lock
Disk files FG – Foreground Process
LMD – Lock Manager Daemon
GRD – Global Resource Directory

©OraInternals Riyaj Shamsudeen 12

CUR mode

 Concurrent changes to the same block.

 Row level lock vs BL lock.

 EX grants must be acquired to change blocks.

Demo: upd_one_row_100.sql, upd_one_row_101.sql

©OraInternals Riyaj Shamsudeen 13
CUR mode

 Two pending transactions in the same block.

FG LMS FG LMS

EX
PI EX

Buffer cache Buffer cache

©OraInternals Riyaj Shamsudeen 14

Buffer changes

 Changes under EX mode.

 Downgrade by other instances.

KJBLNAME KJBLNAME2 KJBLGRANT KJBLROLE KJBLREQUES

------------------------------ -------------------- ---------- -------- ----------
[0x1ac4][0x7],[BL][ext 0x0,0x0 6852,7,BL KJUSEREX 0 KJUSERNL

Enter value for block: 6852

STATE MODE_HELD LE_ADDR DBARFIL DBABLK CR_SCN_BAS CR_SCN_WRP CLASS
---------- ---------- ---------------- ---------- ---------- ---------- ---------- ----------
1 0 000000006D3E3AB0 7 6852 0 0 1

©OraInternals Riyaj Shamsudeen 15

CR buffers

 Instance 1 acquired EX mode lock.

 Instance 2 requests the block, and LMS in instance 1 ships CR copy.

FG LGWR LMS FG LMS

EX
Apply undo
CR CR

CR CR

Buffer cache Buffer cache

[0x1ad3][0x7],[BL]

EX mode lock

Demo: demo_03a.sql ,demo_03b.sql

©OraInternals Riyaj Shamsudeen 16
Busy

 gc cr block busy, gc current block

 LMS constructed buffer applying undo records.

 Excessive *busy events = No application affinity.

 Application affinity will reduce *busy events as the buffers will be

modified in the same instance.

©OraInternals Riyaj Shamsudeen 17

Congested

 Congested wait events also imply concurrency, but at an higher

level.

 If LMS process can not get to a request in 1ms time, then the
response for the request will be marked with ‘congestion’ wait
event.

 Review RT priority and LMS session/process metrics.

©OraInternals Riyaj Shamsudeen 18

RAC cluster root causes

 CPU / Memory starvation in a node.

 Log file write/ LGWR slowness.
 Network issues / packet drops.

 Excessive Dynamic Remastering activities.

 Excessive amount of global cache traffic- due to workload or

execution plan regression.

 Localized inserts in to a few blocks.

 Localized, excessive accesses to a few blocks.

©OraInternals Riyaj Shamsudeen

Golden rules of RAC diagnostics

1. Beware of top event tunnel vision.

2. Eliminate infrastructure as an issue.

3. Identify problem instance(s).

4. Review Send side metrics, and not just Receive side metrics.

5. Use histograms to identify any abnormalities, and not just

averages.

©OraInternals Riyaj Shamsudeen

Top event tunnel vision

 In RAC, the root cause can be hidden in other node and might
not show up in the Top-5 wait events.

 Impact of side effect events can be much, much worse then

actual root cause event itself.

 Performance of the instances are intertwined.

 Performance of background processes in one node, can affect

the application performance in the all other nodes.

©OraInternals Riyaj Shamsudeen

Top event tunnel vision ..2

Example 1

©OraInternals Riyaj Shamsudeen 22

Top event tunnel vision ..3

Example 2

©OraInternals Riyaj Shamsudeen 23

Top event tunnel vision ..4

Example 3

©OraInternals Riyaj Shamsudeen 24

Problem node

Example 3 (continued)

©OraInternals Riyaj Shamsudeen 25

gc buffer busy
A BL request for file #5,
block #8 is pending. gc buffer busy acquire

FG2
gc current request charge time to actual
wait event.

Inst 1
FG1
Wire
Open request Block
File #5, blk #8 send
LMS

Inst 2
Flush request Post

LGWR
gc buffer busy release

FG3

©OraInternals Riyaj Shamsudeen

gc buffer busy

 Event ‘gc buffer busy [acquire|release]’ is a symptom (usually).

 means there is a pending request for a BL lock for that block in

progress.

 All other processes wait for the BL lock to be available and charge
the time to gc buffer busy event.

©OraInternals Riyaj Shamsudeen

Gc buffer busy – Identify object

@ash_gcwait_to_obj.sql

INST_ID EVENT OWNER OBJECT_NAME OBJECT_TYPE CNT

---------- ------------------------ ------ ---------------- ----------------
1 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 3
2 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 10
2 gc buffer busy release RS T_GEN_INS_01 TABLE 4

©OraInternals Riyaj Shamsudeen 28

Gc buffer busy – Identify block

@ash_gcwait_to_block.sql
INST_ID EVENT CURRENT_FILE# CURRENT_BLOCK# CNT
---------- ------------------------------ ------------- -------------- ----------
2 gc buffer busy acquire 4 103582 4
2 gc buffer busy acquire 4 103607 3
2 gc buffer busy acquire 4 103603 3
1 gc buffer busy acquire 4 103582 2
2 gc buffer busy release 4 103615 2
1 gc buffer busy acquire 4 103583 1
2 gc buffer busy release 4 103586 1
2 gc buffer busy release 4 103613 1

In this example, many different blocks are involved in ‘gc buffer

busy’ waits. A symptom of heavy insert load.

Demo: ash_gcwait_to_block.sql
©OraInternals Riyaj Shamsudeen 29
Gc buffer busy – Identify block type

 Dump the block to identify the type and contents of the block.
alter system dump datafile 4 block min 103582 block max 103582;

 In this example, it is a leaf block of an index.

Block header dump: 0x0101949e
Object id on Block? Y
seg/obj: 0x13266 csc: 0x00.11c65d1 itc: 9 flg: E typ: 2 - INDEX
brn: 0 bdba: 0x1019498 ver: 0x01 opc: 0
inc: 0 exflg: 0
Itl Xid Uba Flag Lck Scn/Fsc
..
Leaf block dump
===============

Demo: dump_blocks.sql, tracefile.sql

©OraInternals Riyaj Shamsudeen 30
Object Block type Possible issue(s)
type
Table Segment header Freelists, freelist groups, deletes from
one node,ASSM bugs etc
Segment blocks Heavy insert workload, heavy update to
few blocks, SQL performance issues
scanning few objects aggressively.
Index Leaf block Numerous inserts on indexed columns,
sequence generated keys
Root Insert + numerous index scans,
block/branch concurrent inserts and deletes etc
block
Undo Undo header Numerous short transactions
block
Undo block Long pending transaction and CR block
generation
Dictionary SEQ$ (object) Sequences with nocache or order set
and excessive access to sequence.
©OraInternals Riyaj Shamsudeen 31
Golden rules of RAC diagnostics

1. Beware of top event tunnel vision.

2. Eliminate infrastructure as an issue.

3. Identify problem instance(s).

4. Review Send side metrics, and not just Receive side metrics.

5. Use histograms to identify any abnormalities, and not just

averages.

©OraInternals Riyaj Shamsudeen

gc current block 2-way

gc current request charge time to

‘gc current block 2-way’

Inst 1
FG1
Wire
Open request Block
File #5, blk #8 send
LMS

Inst 2
©OraInternals Riyaj Shamsudeen
Eliminate infra-structure as an issue

©OraInternals Riyaj Shamsudeen 34

Eliminate infra-structure as an issue

 Wait events ‘gc cr block 2-way’ , ‘gc cr block 3-way’ , ‘gc current
block 2-way’, and ‘gc current block 3-way’, ‘gc cr grant 2-way’, ‘gc
cr grant 2-way’.

 No concurrency or congestion issues factored in these events.

 The lowest wait time

 A wait time of <2ms is normal. In Exadata, <0.5ms

©OraInternals Riyaj Shamsudeen 35

Golden rules of RAC diagnostics

1. Beware of top event tunnel vision.

2. Eliminate infrastructure as an issue.

3. Identify problem instance(s).

4. Review Send side metrics, and not just Receive side metrics.

5. Use histograms to identify any abnormalities, and not just

averages.

©OraInternals Riyaj Shamsudeen

Identify problem-inducing instance

 Gv$instance_cache_transfer keeps track of the RX metrics at

both instance and class level.

©OraInternals Riyaj Shamsudeen 37

Identify problem inducing instance ..2
Receiving side analysis.

SELECT INSTANCE ||'->’ || inst_id transfer,

class, cr_block cr_blk,
Trunc(cr_block_time / cr_block / 1000, 2) avg_Cr,
current_block cur_blk,
Trunc(current_block_time / current_block / 1000, 2) avg_cur
FROM gv$instance_cache_transfer
WHERE cr_block > 0 AND current_block > 0
ORDER BY INSTANCE, inst_id, class
/

TRANS CLASS CR_BLK AVG_CR CUR_BLK AVG_CUR

----- ---------- ---------- ---------- ---------- ----------
1->2 data block 87934887 1.23 9834152 1.8
2->1 data block 28392332 1.30 1764932 2.1
...
3->1 data block 12519985 11.57 2231921 21.6
...
3->2 undo block 4676398 8.85 320 27.82

 Use script gc_identify_slow_inst.sql

©OraInternals Riyaj Shamsudeen 38

Golden rules of RAC diagnostics

1. Beware of top event tunnel vision.

2. Eliminate infrastructure as an issue.

3. Identify problem instance(s).

4. Review Send side metrics, and not just Receive side metrics.

5. Use histograms to identify any abnormalities, and not just

averages.

©OraInternals Riyaj Shamsudeen

Send side in an oval
A BL request for file #5,
block #8 is pending. gc buffer busy acquire

FG2
gc current request charge time to actual
wait event.

Inst 1
FG1
Wire
Open request Block
File #5, blk #8 send
LMS

Inst 2
Flush request Post

LGWR
gc buffer busy release

FG3

©OraInternals Riyaj Shamsudeen

Sending side analysis

gc cr block receive time=

Time to send message to a remote LMS process by FG
+ Time taken by LMS to build block (statistics: gc cr block build
time)
+ LMS wait for LGWR latency ( statistics: gc cr block flush time)
+ LMS send time (Statistics: gc cr block send time)
+ Wire latency. RX side
Instance 1:
----------
Avg global cache cr block receive time (ms) : 222.1
Avg global cache current block receive time (ms) : 27.5
Send side
Instance 2:
----------
Avg global cache cr block build time (ms) : 0.0
Avg global cache cr block send time (ms) : 0.1
Global cache log flushes for cr blocks served % : 2.7
Avg global cache cr block flush time (ms) : 15879.9

©OraInternals Riyaj Shamsudeen 41

 Problem node

Global Cache and Enqueue Services - Workload Characteristics

Avg global enqueue get time (ms) : 0.1

Avg global cache cr block receive time (ms) : 14.4

Avg global cache current block receive time (ms): 23.4
Avg global cache cr block build time (ms) : 0.0
Avg global cache cr block send time (ms) : 0.0
Global cache log flushes for cr blocks served %: 25.0

Avg global cache cr block flush time (ms) : 2.5

Avg global cache current block pin time (ms) : 0.0
Avg global cache current block send time (ms): 0.0
Global cache log flushes for current blocks served %: 6.3
Avg global cache current block flush time (ms): 5.9

©OraInternals Riyaj Shamsudeen 42

Sending side analysis ..2

Avg message sent queue time (ms) : 0.1

Avg message sent queue time on ksxp (ms): 0.6
Avg message received queue time (ms): 0.0
Avg GCS message process time (ms) : 0.0
Avg GES message process time (ms) : 0.0
% of direct sent messages : 46.62
% of indirect sent messages : 43.76
% of flow controlled messages : 9.62

Avg message sent queue time (ms) : 0.0

Avg message sent queue time on ksxp (ms): 0.4
Avg message received queue time (ms) : 0.0
Avg GCS message process time (ms) : 0.0
Avg GES message process time (ms) : 0.0
% of direct sent messages : 60.86
% of indirect sent messages : 38.48
% of flow controlled messages : 0.66

©OraInternals Riyaj Shamsudeen 43

 Problem node

Global Cache and Enqueue Services - Messaging Statistics

Avg message sent queue time (ms) : 344.7

Avg message sent queue time on ksxp (ms): 0.3
Avg message received queue time (ms): 0.0
Avg GCS message process time (ms) : 0.0
Avg GES message process time (ms) : 0.0
% of direct sent messages : 31.76
% of indirect sent messages : 46.12
% of flow controlled messages : 22.13

©OraInternals Riyaj Shamsudeen 44

Multi-tenancy

 Buffer cache and Global cache processing is common to all

PDBs.

 A PDB generating lot more GC workload can affect others.

 No methods to constrain the buffer cache or global resource

usage of a PDB.

 Resource manager only controls CPU, PX, and I/O(exadata).

©OraInternals Riyaj Shamsudeen 45

Prepare for multi-tenancy (RAC specific)

 Keep the buffer cache as large (for OLTP). Why?

 Make LGWR efficient. LMS waits for LGWR to ship blocks.

 Tune the SQL statements to reduce global cache traffic.

 Connect to the container database to review global cache metrics.

©OraInternals Riyaj Shamsudeen 46

Resource names

 From 12 onwards, resource names are coded with con_id.

select resource_name from gv$ges_resource

where resource_name like '[0x15f29][0x0],[TM]%'
/

RESOURCE_NAME
------------------------------
[0x15f29][0x0],[TM][ext 0x3,0x0] <- hrdb1 PDB – GSTEST table –object_id=89897
[0x15f29][0x0],[TM][ext 0x4,0x0] <- hrdb2 PDB – GSTEST table –object_id=89897

 Comes handy if you are debugging RAC trace files, to identify

the PDB generating the errors.

©OraInternals Riyaj Shamsudeen 47

GC statistics and PDBs

 Most Global cache statistics views do not provide granularity to a

pluggable database. For example,

select con_id, inst_id, count(*) from gv$instance_cache_transfer

group by con_id, inst_id
/
CON_ID INST_ID COUNT(*)
---------- ---------- ----------
0 2 4096
0 1 4096

 Review AWR reports from the container database to review

global cache performance issues.

©OraInternals Riyaj Shamsudeen 48

High Priority Processes (from 12c)

Parameter: _high_priority_processes
Value : LMS*|VKTM|LM*|LCK0|GCR*|DIAG

…
kjfmPriCheck: 0 RTLMS taken, 1 RTLMS available
kjfmPriCheck: automatic priority changes enabled - multi-inst mode
kjfmPriCheck: # of real time LMS increasing from 0 to 1
…

Parameter: _lm_lms_priority_dynamic
Value : TRUE

©OraInternals Riyaj Shamsudeen 49

gc cr grants 2-way

gc cr request charge time to

‘gc cr grant 2-way’

Inst 1
FG1
Wire
Open request Read from
File #5, blk #8 the disk
LMS

Inst 2
©OraInternals Riyaj Shamsudeen
Why DRM?

charge time to
‘gc cr grant 2-way’

Inst 1
FG1

LMS

Inst 2
Read from
the disk
Excessive amount of GC activity for
One object!

©OraInternals Riyaj Shamsudeen

DRM & grants

 Reduces RAC Tax: gc cr grants 2-way

 If the object is mastered locally, processes can read the block

directly without incurring any GC waits.

 Ash_gcwait_to_obj to identify objects suffering from ‘gc cr

grants 2-way’.

INST_ID OWNER OBJECT_NAME OBJECT_TYPE CNT

---------- -------------------- -------------------------------- -------------------- ----------
1 AR RA_CUSTOMER_TRX_ALL TABLE 9
1 INV MTL_MATERIAL_TRANSACTIONS TABLE 141
1 INV MTL_SERIAL_NUMBERS TABLE 155
1 Undo Block 253
1 INV MTL_SYSTEM_ITEMS_B TABLE 476
1 AP AP_CHECKS_ALL TABLE 785

Demo: remaster.sql tc_one_row remaster2.sql

©OraInternals Riyaj Shamsudeen 52
Local vs Remote

 74% remote grants. DRM can be useful here.

gc local grants 6,570,658 3,643.6 21.8
gc remote grants 19,262,806 10,681.7 63.9

 Increase SGA size to reduce single block I/O reads.

 Tuning SQL statements to reduce logical I/O, optimal indexes,

etc can help to improve the performance.

©OraInternals Riyaj Shamsudeen 53

DRM effect

Top 5 Timed Events Avg %Total

~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
gc buffer busy 1,826,073 152,415 83 62.0 Cluster
CPU time 30,192 12.3
enq: TX - index contention 34,332 15,535 453 6.3 Concurrency
gcs drm freeze in enter server 22,789 11,279 495 4.6 Other
enq: TX - row lock contention 46,926 4,493 96 1.8 Applicatio

Global Cache and Enqueue Services - Workload Characteristics

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg global enqueue get time (ms) : 16.8
Avg global cache cr block receive time (ms) : 17.1
Avg global cache current block receive time (ms): 14.9

©OraInternals Riyaj Shamsudeen 54

 DRM (11g)
In 11g, all resources are frozen
Request queue during the reconfiguration.
LCK

LMD LMD LMD

LMON LMON LMON

LMS0 LMS1 LMSn LMS0 LMS1 LMSn LMS0 LMS1 LMSn

Resources

Inst 1 Inst 2 Inst 3

©OraInternals Riyaj Shamsudeen 55

 DRM (12c)
In 12c, only set of resources in
Request queue a window is frozen.
LCK

LMD LMD LMD

LMON LMON LMON

LMS0 LMS1 LMSn LMS0 LMS1 LMSn LMS0 LMS1 LMSn

Resources

Inst 1 Inst 2 Inst 3

©OraInternals Riyaj Shamsudeen 56

DRM in 12c
DRM impact is decreased in 12c by freezing a resource partition, at a time..
* drm quiesce
2014-02-15 16:03:45.131840 : DRM(3) resources quiesced [0-1023], rescount 1719
2014-02-15 16:03:45.132173 : DRM(3) local converts quiesced [0-1023], lockcount 0, bucket 0
* drm sync 1
* drm freeze
* drm cleanup
* drm sync 2
* drm replay
* drm sync 3
* drm fix writes
* drm sync 4
* drm end

* DRM RCFG called (swin 0)

CGS recovery timeout = 85 sec
* drm quiesce
2014-02-15 16:03:45.172867 : DRM(3) resources quiesced [1024-2047], rescount 1754
2014-02-15 16:03:45.172932 : DRM(3) local converts quiesced [1024-2047], lockcount 0, bucket 0
* drm sync 1
* drm freeze

©OraInternals Riyaj Shamsudeen 57

PQ trace file
 PQ Co-ordinator trace shows Object checkpoint.
select /*+ parallel (a 4) */ min(length(v1)) from rs.hugetable a;
WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 866
name|mode=1263468550 2=65581 0=1 obj#=-1 tim=6965638229
...
WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 454
name|mode=1263468545 2=65581 0=2 obj#=-1 tim=6965638986
...

 PQ server process will perform direct reads to PGA. Notice that

there is no GC waits between the reads.
WAIT #18446741324874985288: nam='direct path read' ela= 4782 file number=4 first dba=11584
block cnt=16 obj#=76739 tim=3653439149
WAIT #18446741324874985288: nam='direct path read' ela= 16157 file number=4 first dba=11600
block cnt=16 obj#=76739 tim=3653456065
WAIT #18446741324874985288: nam='direct path read' ela= 3675 file number=4 first dba=11616
block cnt=16 obj#=76739 tim=3653462065
WAIT #18446741324874985288: nam='direct path read' ela= 8162 file number=4 first dba=11632
block cnt=16 obj#=76739 tim=3653471358
WAIT #18446741324874985288: nam='direct path read' ela= 10270 file number=4 first dba=11650
block cnt=14 obj#=76739 tim=3653482566

©OraInternals Riyaj Shamsudeen 58

Application issues

 There is no easy fix in 12c either 

 Avoid localized inserts.

©OraInternals Riyaj Shamsudeen 59

Application issues

 Hash partitioning the index/table might be helpful.

©OraInternals Riyaj Shamsudeen 60

Application issues

 Reduce localized accesses:

 Frequent access to a lookup table.

 Aggressive scanning of smaller tables.

 Aggressive nested loops joins accessing index leaf blocks.

 Ordered cached sequences – Excessive SV resource contention.

©OraInternals Riyaj Shamsudeen 61

 Use Services

 While creating a service, you Inst 1

define a preferred and available
instances. PO
FIN

PO Inst 2 FIN

PO
FIN

Inst 3
For PO service: Instance 1 & 2 are
Preferred
preferred instances, and instance PO
3 is an available instance. Available
FIN
©OraInternals Riyaj Shamsudeen 62
 Or PDBs

 PDB1 is open in inst1 & inst2. Inst 1

 PDB2 is open in inst2 & inst3
PDB1
PDB2

PO Inst 2 FIN

PDB1
PDB2

Inst 3
Online
PDB1
PDB2 Failover

©OraInternals Riyaj Shamsudeen 63

HugePages

 By default, pagesize is 4KB in Linux. PageTables size will be

huge, in the order of, GigaBytes.

 Kswapd consumes CPU heavily.

 Node reboots due to CRS issues.

 Pagesize is 2MB and pagetable usage is minimal.

©OraInternals Riyaj Shamsudeen 64

THANK YOU

 Email: rshamsud@orainternals.com
 Blog : orainternals.wordpress.com
 Web: www.orainternals.com

©OraInternals Riyaj Shamsudeen 65