Nmon and Topas
Nmon and Topas
Nigel Griffiths
IBM Power Systems
Advanced Technology Support, Europe
p
To
2010 IBM
as
&
on
nm
nmon
Year 16
2 of 33
p
To
2010 IBM
as
on
nm
3 of 33
2000
4.3.3
1999
2003
2006
2009
nmon9f AIX4
Last AIX 4.3.3
Used /dev/kmem
nmon10 AIX5
Uses public API
& libperfstat
nmon12f AIX6
nmon2web
nmon
analyser
nmon2rrd
Lotus 1-2-3
Then Excel
nmon
Consolidator
Multiple machines
2011
2010
AIX command
topas_nmon
2008
2005
2003
60+ flavours!
Open
Sourced
2007
2001
on
nm
&
1997
2010 IBM
as
x86
POWER
x86
mainframe
p
To
Timeline 15 years
embedded
x86_64
merged in to topas
nmon12f function
Released with AIX
AIX 5.3 TL09 +
AIX 6.1 TL02 +
topas CEC
reports
analyser
pGraph
Website generator
4 of 33
2010 IBM
Stephens
nmon
file
nmon
5
Consolidator
as
&
on
nm
nmon
-f or F option
p
To
Screen
Stephens
Others
Federicos
pGraph
Bruces
nmon2web
Perl
nmon2rrd
C filter
nmon
Analyser
18 0
16 0
14 0
12 0
10 0
80
60
40
20
0
Excel graphs
Scripts
Scripts
rrdtool scripts
+ CPU & RAM
aggregation
- Create rrd
- Load rrd
- Graph rrd
index.html
10 0
90
80
70
60
50
40
30
20
10
0
nmon flow
Website +.jpg graphs
rrdtool
rrdtool
open
source
open
source
5 of 33
p
To
2010 IBM
as
&
on
nm
6 of 33
7
ER
W
PO
2013 IBM
ce
an
m
or
rf
Pe
Email 6:
What 1st things do I look for in an nmon file?
Firmware level
(BBBP lsconf)
AIX level - release, TL and SP (BBBP oslevel -s)
Not using classic nmon
(AAA)
If any over 2 years old the nmon user are either a newbie
or a time waster.
From this I know how professional they are!!
If you dont service your car for three+ years:
Why should the car maker spend time investigating
your problem or take the blame for the top speed !
Then period of time covered and the interval
Long intervals hide peaks
2013 IBM
ce
an
m
or
rf
Pe
7
ER
W
PO
Then
What 2nd things do I look for in an nmon file?
Shared Uncapped how often over Entitlement
Most utilisation numbers are pointless now IMHO
CPU_SUMM tab
Shows use of SMT
DISKBUSY tab
No disks over 50%
Most disks active i.e. not old school disk layout
IOADAPT tab
Disk I/O even across FC adapters
MEM tab
Check free memory is NOT large. If it is tune to use it
MEMNEW tab
Is filesystem cache use as expected. RDBMS DIO/CIO should be small
2 of 3
7
ER
W
PO
Then
What 3rd things do I look for in an nmon file?
2013 IBM
ce
an
m
or
rf
Pe
NET tab
Is it pretty!
Does it peak at a well know network speed?
Page tab
Should be low but do see high filesystem paging
these days = memory mapped files.
POOL tab
See if Pool is exhausted
Proc
Compare RunQ with SMT thread available
Top - if present
Check is the top CPU processes expected
3 of 3
p
To
2010 IBM
as
&
on
nm
10
p
To
2010 IBM
as
&
11
on
nm
11 of 33
as
on
nm
DETAIL
&
topas
default
12
Nigels Monitor
mode
~
topas
p
To
Mode Changing
nmon
topas_nmon
2010 IBM
C
CEC topas -C
12 of 33
p
To
2010 IBM
as
&
13
on
nm
topas
13 of 33
&
14
on
nm
14 of 33
2010 IBM
as
No marketing
No Redbook
Manual is 30+ pages
p
To
topas problems
KB-W
AQW
MWT
PSz
Vcsw
%bdon
MPuse
hpit
physb
pmem
RTFM
p
To
2010 IBM
as
Modes
&
on
nm
Topas
Local AIX mode
15
Cross-partition Mode
nmon mode
nmon
Classic for
AIX & Linux
[CEC/whole machine]
Default
View
Detailed
View
2
WPAR
7
CEC
Save
nmon
View
nmon
View
LPAR View
5
6
nmon
Save
AMS
10
VIOS View
8
WPAR
nmon
Save
CSV file
Other Local
Save Tools
topasout
Text
Reports
nmon Analyser
nmon Consolidator
nmon2rrd/nmon2web
Other tools or databases
15 of 33
p
To
2010 IBM
as
&
on
nm
16
16 of 33
p
To
2010 IBM
as
&
17
on
nm
cndfp
17 of 33
p
To
2010 IBM
as
&
18
on
nm
E
D
F
18 of 33
p
To
2010 IBM
as
&
19
on
nm
19 of 33
p
To
2010 IBM
as
&
on
nm
20
VOIS
[SEA]
Physical Network
13.9KB/s
31.3KB/s
Client
Virtual Network
14.3KB/s
20 of 33
10
p
To
2010 IBM
as
&
21
on
nm
21 of 33
p
To
2010 IBM
as
&
22
on
nm
Client
vhost
22 of 33
Server
vhost1
Logical
Volume
sissas0
hdisk0 to 4
11
p
To
as
CEC
Whole machine
Cross Partition
All LPAR
View
2010 IBM
&
23
on
nm
23 of 33
p
To
2010 IBM
as
&
HMC
LPAR: topas C
VIOS
Once only
- Can take 20 seconds
- Newer version polls
the local network for
AIX LPARs
on
nm
(ugly)
topas
xmtopas
xmwlm
xmperf
24
Over network
Once every 10 seconds
Linux
AIX
AIX
AIX
12
p
To
2010 IBM
as
&
25
on
nm
Active
Memory
Sharing view
AMS has a renewed focus due to new De-Duplication features
25 of 33
p
To
2010 IBM
as
&
26
on
nm
Monitoring 1 LPAR
for AMS is insane
26 of 33
13
2010 IBM
as
p
To
topas C
&
27
on
nm
Interval: 10
Memory (GB)
Processor
Monitored : 8.0
Monitored :2.0
UnMonitored:
UnMonitored: Available :
Available : UnAllocated:
UnAllocated: Consumed
: 6.5
Shared
: 2
Dedicated : 0
Donated
: 0
Pool Size : 4
Host
OS M Mem InU Lp Us Sy Wa Id PhysB Vcsw Ent %EntC PhI pmem
-------------------------------------shared------------------------------------silver_vios1 A61 U 2.0 1.9 8 08 1 0 90
0.01 873 0.40
2.6
0
silver_lpar2 A61 UM 2.0 1.4 2 49 1 0 49
0.26 244 0.50 51.3
1 1.30
silver_lpar3 A61 UM 2.0 1.6 2
0 0 0 99
0.01 294 0.50
1.4
0 0.78
silver_lpar4 A61 UM 2.0 1.7 2
0 0 0 99
0.01 220 0.50
1.3
0 0.87
silver_lpar5 A61 UM 2.0 1.7 2
0 0 0 99
0.01 218 0.50
1.1
0 1.01
Inu
InUse LPAR Working Set
Logical Memory Mem
Dedicated
Memory
Shared memory LPAR
LPAR
Physical Memory pmem
27 of 33
p
To
2010 IBM
as
&
on
nm
28
mpid
mpsz mpus
mem memu iome iomu
hpi
hpit
-------------------------------------------------------------------------------0
4.00 3.99 8.00 6.82 308.0 47.9
32
31
Machine memory
14
p
To
as
topasout
2010 IBM
&
29
on
nm
-c
v
p
To
as
topasout
2010 IBM
&
30
on
nm
c = comma separated
but one timestamp and value per line
s = spreadsheet
space separated 100s of stats per line
a = analyser
.csv file for nmon analyser or use topas_nmon directly
Machine or LPAR - Text reports
topasout -R OPTION [-i MM -b HHMM -e HHMM] /etc/perf/daily/hostname_YYMMDD.topas
OPTION = detailed|summary|poolinfo|disk |lan|mempool|adapter|vadapter|vios|vios_adapter
Example: topasout R summary /etc/perf/daily/silver8_090304.topas >/tmp/LPARsummary.txt
Text reports via standard out, so you need to redirect the output to a file
Cross-Partition Text reports
topasout -R detailed | -R summary [-i MM -b HHMM -e HHMM] /etc/perf/topas_cec.YYMMDD
Example: topasout -R detailed topas_cec.090305 >/tmp/CECdetailed.txt
Text reports via standard out, so you need to redirect the output to a file
For system use only, do NOT run these manually.
topasout -R daily | -R weekly
Notes: YYMMDD = year, month, day
MM is minutes
30 of 33
15
p
To
2010 IBM
as
31
&
on
nm
Creates sample_cec_101222_2149.topas
3. topasout -a sample_cec_101222_2149.topas
Creates sample_cec_101222_21492.topas.csv
4. Start CEC Analyser & load the .csv output file & you get
31 of 33
p
To
2010 IBM
as
&
32
on
nm
32 of 33
16
p
To
2010 IBM
as
&
33
on
nm
New Stuff
2.
3.
4.
5.
6.
7.
8.
33 of 33
purple7.aixncc.uk.ibm.com
p
To
2010 IBM
as
&
on
nm
34
34 of 33
17
as
35
&
on
nm
Memory Pages
p
To
2010 IBM
Compressed
Memory pages
p
To
2010 IBM
as
&
on
nm
36
Memory Growing
for RAM optimisation & performance
10GB
10GB
But want 14GB to
improve performance
Looks like 10GB but
is actually 8GB,
thanks to AME
2GB released
for another LPAR
36 of 33
18
p
To
2010 IBM
as
&
on
nm
topas
37
EF = Expansion Factor
T = Target
A = Actual
p
To
2010 IBM
as
&
on
nm
38
MEM tab
Size of the Compressed pool (MB)
Size of true memory (MB)
Expanded memory size (MB),
Size of the Uncompressed pool (MB)
MEMNEW tab
Compressed Pool%
PAGE tab
Paging rates (pages per second) but this time very quickly
in & out of the compressed memory area
Compressed pool pgins - other tools like topas call this CI
Compressed pool pgouts - other tools like topas call this CO
38 of 33
19
p
To
2010 IBM
as
&
39
on
nm
2.00G
39 of 33
p
To
2010 IBM
as
&
on
nm
40
See my session
PE07
POWER7, AIX, Affinity and Performance
40 of 33
20
p
To
2010 IBM
as
&
41
on
nm
topas M or
topas and the hit M or
nmon then ~ then M
Power 770 again
# lssrad -av
REF1
SRAD
0
0
3
1
1
2
MEM
CPU
25054.75
6705.50
17679.00
14193.00
41 of 33
p
To
2010 IBM
as
&
42
on
nm
42 of 33
21
p
To
2010 IBM
as
&
43
on
nm
43 of 33
p
To
2010 IBM
as
&
44
on
nm
22
p
To
2010 IBM
as
&
45
on
nm
Power 770
LPAR = 64 GB
Across 3 SRADs
CPUs are
0- 7
8 = 2
8-11
4 = 1
16-23
4 = 1
VP = 4
Excellent
Good
Blisteringly
Fast
Very little
use so 100%
not that
important
45 of 33
p
To
2010 IBM
as
&
on
nm
46
46 of 33
23
p
To
2010 IBM
as
&
47
on
nm
Memory stats
Kernel stats
numperm=
filesystem cache
Process switches
Fork new process
Type =
System or App
State =
Active or Defined
47 of 33
p
To
2010 IBM
as
&
48
on
nm
Busy with
WPAR work
48 of 33
24
p
To
2010 IBM
as
&
49
on
nm
Inside a WPAR
Bonkers like driving a car with your eyes closed
p
To
2010 IBM
as
&
on
nm
50
50 of 33
25
p
To
2010 IBM
as
&
51
on
nm
p
To
2010 IBM
as
&
52
on
nm
Last snapshot
Max=2 physical CPUs
Average of what
is on screen
52 of 33
26
p
To
Max=2
2010 IBM
as
&
53
on
nm
Max=0.5
Max=0.031
Max=0.06
53 of 33
p
To
2010 IBM
as
&
on
nm
54
54 of 33
27
p
To
2010 IBM
as
&
55
on
nm
MEM
1699.37
CPU"
0-7"
p
To
2010 IBM
as
&
on
nm
56
Experimental
DISKAVGRIO - Disk IO Average Reads per second
DISKAVGWIO - Disk IO Average Writes per second
56 of 33
28
p
To
2010 IBM
as
&
57
on
nm
Undocumented
No power saving
Over-clocking
57 of 33
p
To
2010 IBM
as
&
58
on
nm
topas_nmon PURR/SPURR
for AIX7 TL1 on POWER7 E=0.8 SMT=4 VP=2
with Power Saving On via Systems Director AEM
Note:
Goes over the VP. It has more clock ticks due to over clocking by ~9% - I think!!
58 of 33
29
p
To
2010 IBM
as
&
59
on
nm
Online Disks
nmon then
DDDD
New columns
59 of 33
p
To
2010 IBM
as
&
on
nm
nmon
60
60 of 33
30
p
To
2010 IBM
as
&
on
nm
topas
61
61 of 33
31