AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

1.Let me first create a test case:I

will take different OS statistics
and run time statistics to
compare with AWR report
generated after execution.
March 15, 2018 by

AWR report analysis in depth-part 1

A.About test case in lower CPU:-

I will create a table first

create table t (id number, sometext varchar2(50),my_date date) tablespace

data;

Now I will create a simple procedure to load bulk data

create or replace procedure manyinserts as

v_m number;
begin
for i in 1..10000000 loop
select round(dbms_random.value() * 44444444444) + 1 into v_m from dual
t;
insert /*+ new2 */ into t values (v_m, ‘DOES
THIS’||dbms_random.value(),sysdate);

commit;

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 1 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

end loop;
end;

This insert will be executed in 10 parallel sessions using

dbms_job.This will fictitiously increasded load on database.

create or replace procedure manysessions as

v_jobno number:=0;
begin
FOR i in 1..10 LOOP
dbms_job.submit(v_jobno,’manyinserts;’, sysdate);
END LOOP;
commit;
end;
/

Now I will execute manysessions which will fork 10 parallel

sessions

exec manysessions;

B.About test case in higer CPU:-

I will create a table first

create table t (id number, sometext varchar2(50),my_date date) tablespace

data;

Now I will create a simple procedure to load bulk data.I will make
commit out of loop to decrease frequent commit.

create or replace procedure manyinserts as

v_m number;

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 2 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

begin
for i in 1..10000000 loop
select round(dbms_random.value() * 44444444444) + 1 into v_m from dual
t;
insert /*+ new2 */ into t values (v_m, ‘DOES
THIS’||dbms_random.value(),sysdate);

–commit;
end loop;

commit;
end;

This insert will be executed in 10 parallel sessions using

dbms_job.This will fictitiously increasded load on database.

create or replace procedure manysessions as

v_jobno number:=0;
begin
FOR i in 1..10 LOOP
dbms_job.submit(v_jobno,’manyinserts;’, sysdate);
END LOOP;
commit;
end;
/

Now I will execute manysessions which will fork 10 parallel

sessions

exec manysessions;

I will execute the test case on server with low CPU (A) as

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 3 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

well as high CPU (B).

2.I will use oratop tool to monitor the sessions

oratop – Utility for Near Real-time Monitoring of Databases, RAC and Single
Instance (Doc ID 1500864.1)

./oratop -f -d -i 10 / as sysdba

optionally you press x

Then you can put sql_id to see the execution plan as well.

For A.

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 4 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

For B

Section 1 - database

Global Database information

Version : Oracle major version

role : database_role

db name : db_unique_name

time [s]: time as of the most recent stats (hh24:mi:ss)

up [T]: database uptime

ins [N]: total number of instance(s)

sn [c,N]: total user sessions (active/inactive)

us [c,N]: number of distinct users

mt [s,N]: global database memory total (sga+pga)

fra [N]: flashback recovery area %used, (red > 90%)

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 5 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

er [N]: diag active problem count (faults)

% db [s,N]: database time as %(dbtime/cpu) (red if >

99%)

Section 2 - instance

Top 5 Instance(s) Activity

o Ordered by Database time desc

ID [c,N]: inst_id (instance id)

%CPU [m,N]: host cpu busy %(busy/busy+idle). (red if >

90%)

LOAD [m,N]: current os load. (red if > 2*#cpu & high

cpu)

%DCU [m,N]: db cpu otusef as %host cpu. (red if > 99%

& high AAS)

AAS [s,N]: Average Active Sessions. (red if > #cpu)

ASC [c,N]: active Sessions on CPU

ASI [c,N]: active Sessions waiting on user I/O

ASW [c,N]: active Sessions Waiting, non-ASI (red if >

ASC+ASI)

ASP [m,N]: active parallel sessions (F/G)

AST [c,N]: Active user Sessions Total (ASC+ASI+ASW)

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 6 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

UST [c,N]: user Sessions Total (ACT/INA)

MBPS [m,N]: i/o megabytes per second (throughput)

IOPS [m,N]: i/o requests per second

IORL [m,T]: avg synchronous single-block read latency. (red

> 20ms)

LOGR [s,N]: logical reads per sec

PHYR [s,N]: physical reads per sec)

PHYW [s,N]: physical writes per sec

%FR [s,N]: shared pool free %

PGA [s,N]: total pga allocated

TEMP [s,N]: temp space used

UTPS [s,N]: user transactions per sec

UCPS [c,m,N]: user calls per sec

SSRT [c,m,T]: sql service response time (T/call)

DCTR [m,N]: database cpu time ratio

DWTR [m,N]: database wait time ratio. (red if > 50 &

high ASW)

%DBT [s,N]: instance %Database Time (e.g. non-rac shows

100%)

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 7 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

Section 3 - db wait events

Top 5 Timed Events

o Cluster-wide, non-idle

o Ordered by wait time desc

EVENT : wait event name. (red if active)

(RT) : Real-Time mode

WAITS : total waits

TIME(s) : total wait time in seconds)

AVG_MS : average wait time in milliseconds

PCT : percent of wait time (all events)

WAIT_CLASS : name of the wait class

Section 4 - process

o Non-Idle processes

o Ordered by event wait time desc

ID [N]: inst_id. (red if blocking)

SID [N]: session identifier. (red if blocking)

SPID [N]: server process os id

USERNAME : Oracle user name

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 8 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

PROGRAM : process program name

SRV : SERVER (dedicated, shared, etc.)

SERVICE : db service_name

PGA [N]: pga_used_mem. (red if continuously growing)

SQL_ID/BLOCKER : sql_id or the final blocker's (inst:sid, in

red)

OPN : operation name, e.g. select

E/T [T]: session elapsed time (active/inactive)

STA : ACTive|INActive|KILled|CAChed|SNIped

STE : process state, e.g. on CPU or user I/O or

WAIting

WAIT_CLASS : wait_class for the named event

EVENT/*LATCH : session wait event name. Auto toggle with

*latch name.

(red if process is hung/spin)

W/T [T]: event wait time. (red if > 1s)

3.TOP output during activity

For A:-

We had 4 CPU,so our 6 jobs (status :-S means Sleeping) had to wait for
CPU.And other 4 jobs are running (Status :-R means Running). Also note
https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 9 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

86% CPU used because we are consuming 4 CPU out of 4 CPU).There is

almost no idle CPU.

For B:-

We had 20 CPU,so our all 10 jobs are running (Status :-R means
Running).Also note 44.4 % CPU is used because out of 20 CPU we are using
10 CPU.Hence there is 50% idle CPU.

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 10 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

4.VMSTAT report
For A:-

For B:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 11 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

5.The table size after 10 jobs completed.

SQL> select bytes/1024/1024/1024 from dba_segments where
segment_name=’T’;

BYTES/1024/1024/1024

——————-

7.375

6.AWR report 360 degree analysis

A.Header Section

For A:-

For B:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 12 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

Same load completed 5 times faster when we have more CPU.

Sessions:=No of active sessions

Cursors/Session:=No of open cursor per session.If this value is

increasing or high indicates potential problem of cursor leaking.

Elapsed :=The elapsed time indicates the duration of the report

between the 2 selected snapshots. Any other duration figures can
be compared back to this. A 30-60 minute reporting period is
usually recommended.

DB time:=db time is the time spent in the database.For example,If

we have 10 parallel session each taking 20 minutes to execute,our
DB time will be 200 minutes+other query execution time in 20
minutes snapshot period.The DB Time is the time spent in the
database for the period of the report. If this is significantly higher
than the Elapsed time then this is a good indicator of a heavily
loaded system.

B.Load Profile

For A:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 13 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

For B:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 14 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

DB Time:-No of active sessions average during snapshot

period.This is derived by DB time/Elapsed time in previous
section.If the number his high,then probably many active session
are there in database on particular point which may indicate
bottleneck or opportunity to deep analysis.

DB Cpu:-DB CPU is quite less than DB time ,so sessions are

waiting not much working on cpu.This may indicate a problem if
your DB cpu percent is quite less than DB time.

Redo size:-11 MB per second.An increase in Redo size and Block

changes indicate increased DML(INSERT/UPDATE/DELETE)
https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 15 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

activity.

Logical reads:-Logical reads is simply the number of blocks read

by the database, including physical (i.e. disk) reads, and block
changes is fairly self-descriptive.If you find those number higher
than expected (based on usual numbers for this database, current
application workload etc.), then you can drill down to the “SQL by
logical reads” and “SQL by physical reads” to see if you can
identify specific SQL responsible.

Block Changes:-This indicates high block change means many

transactions going on.

User Calls:-In particular, when the database is executing many

times per a user call, this could be an indication of excessive
context switching (e.g. a PL/SQL function in a SQL statement
called too often because of a bad plan). In such cases looking into
“SQL ordered by executions” will be the logical next step.

Hard parse:-A hard parse occurs when a SQL statement is

executed and is not currently in the shared pool.A hard parse rate
greater than 100 per second could indicate that bind variables are
not being used effectively; the CURSOR_SHARING initialization
parameter should be used; or you have a shared pool–sizing
problem.

Parse:-This is soft parse+hard parse.A soft parse occurs when a

SQL statement is executed and it is currently in the shared pool. A
very high soft parse rate could lead to problem of programmed
application.

Logons:-Establishing a new database connection is also expensive

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 16 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

(and even more expensive in case of audit or triggers). “Logon

storms” are known to create very serious problem.In fact log off is
more expensive.This indicates problem of not using proper
connection pooling.

Executes:-High no of executes always indicates potential load to

database.

Transactions:-High number of transactions indicates it is OLTP

system.

C.Time Model Statistics

for A:

For B:

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 17 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

sql execute elapsed time is 60%.That means DB spent 60% time of

DB_TIME is executing SQL query
which is OK for now .
DB CPU is used 20% of DB_TIME.
Now We can see PL/SQL execution elapsed time is 11% means
many PL/SQL procedure is being executed.
You can see which PL/SQL executed in SQL ordered by Elapsed
Time.
Parse time elapsed and hard parse elapsed time is less which is
good sign.If there are bigger,we
need to check why query is taking more time to parse.

D.Instance Efficiency Percentages

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 18 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

For A:-

For B:-

Buffer Nowait %:-Less than 99 percent may indicate a problem .

This value is the ratio of hits on a request for a specific buffer
where the buffer was immediately available in memory. If the ratio
is low, then there are (hot) blocks being contented for that should
be found in the Buffer Wait section.

Buffer Hit %:- Less than 95 percent may indicate a problem. This
value is the ratio of hits on a request for a specific buffer when the
buffer was in memory and no physical I/O was needed.A hit ratio
that is steadily at 95 percent and then drops to 45 percent should
be checked for bad SQL or a dropped index (check the top physical
reads SQL) causing a surge in physical reads that are not using an
index or an index that has been dropped

Library Hit % :-Less than 95 percent indicates problem. A lower

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 19 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

library hit ratio usually indicates that SQL is being pushed out of
the shared pool early (could be due to a shared pool that is too
small). A lower ratio could also indicate that bind variables are not
used or some other issue is causing SQL not to be reused (in which
case, a smaller shared pool may only be a bandage that potentially
fixes a resulting library latch problem)

In-memory Sort % :- less than 95 percent in OLTP indicates

problem. In an OLTP system, you really don’t want to do disk
sorts. Setting the MEMORY_TARGET or
PGA_AGGREGATE_TARGET (or SORT_AREA_SIZE in previous
versions) initialization parameter effectively eliminates this
Problem.

Soft Parse %:- Less than 95 percent indicates problem. A soft parse
ratio that is less than 80 percent indicates that SQL is not being
reused and needs to be investigated.

Latch Hit %:- Less than 99 percent is usually a big problem.

Finding the specific latch will lead you to solving this issue.

E.Foreground Events by total wait time

For A:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 20 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

For B:-

This is one of the most important section.You must be concerned

if any wait event take abnormally high %DB time apart for DB
CPU.If DB CPU is high like 80%,then your application is CPU
bound.

DB CPU :-It is showing 19.5% means it is may be CPU bound but

because we have only 4 CPU and 10 parallel sessions,CPU is not
able to do much work.

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 21 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

Resmgr:cpu quantum:-Indicates it may be problem of allocating

sufficient number of CPU cycles.

F.Background wait events

For A:-

For B:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 22 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

The log file parallel write shows LGWR is waiting for blocks to be
written to all online redo log members in one group. LGWR will
wait until all blocks have been written to all members. So here we
had 80% of background total time spent on log file parallel write.
The parameter of interest here is Avg wait (ms). In our case it is
1ms which is a perfectly respectable figure. Obviously larger
average wait times point to slower I/O subsystem or poor I/O
configurations.As a rule of thumb, an average time for ‘log file
parallel write’ over 20 milliseconds suggests a problem with IO
subsystem.
Another thing is log file parallel write is background wait event of
log file sync.You need to check if large commit
is happening in “user commits” section of “Instance Activity Stats”

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 23 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

G.Host CPU

For A:-

For B:-

Host CPU

Here you can notice %user is 84%.This really gives you sign that
your CPU are highly used.Load average 2.77 (after execution
completed) means you are having average 3 sessions running at
one time and other 7 sessions are waiting for CPU.But most of the
time load average was around 10 (refer oratop and vmstat) as
there was 10 sessions running all the time during snapshot period.

Instance CPU

%Total cpu indicates that Database is using 82.6 % cpu of total CPU

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 24 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

of the database server. This indicates there are not many other
database’s process running currently.If you see less value here it
may indicates the database server total cpu may be used by other
application or database instances.

H.IO Profile

For A:-

For B:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 25 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

Here Total requests:=Database requests + Redo

requests=1046+322=1381(around) which is actually IOPS.

Total MB=Database (MB) + Redo (MB)=5.9+22.9=30.4 MB which

denotes database is generating 30 MB data per second.

I.Memory Statistics

For A:-

For B:-

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 26 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

Host Mem(MB):-Total memory in the system

SGA use (MB):-Total SGA used

PGA use(MB):-Total PGA used.

Cache size mainly distributed Buffer cache(where data buffer

stored in memory)+Shared pool size.

A low value for the % SQL with exec>1 statistic indicates databasee
is not re-using shared SQL statements, usually because the SQL is

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 27 of 28
AWR report analysis in depth-part 1 | clouddba 3/25/18, 6*22 PM

not using bind variables.

J.OS Statistics

For A:-

For B:-

This is like vmstat report.Almost 83% cpu was used during

execution which is quite high.%busy high is good means cpu are
not using simultaneous multi threading.

https://www.clouddba.co/awr-report-analysis-depth-part-1/ Page 28 of 28