The Power of

IPMI
Werner Fischer
Technology Specialist
Thomas-Krenn.AG

2nd Open Source Data Center Conference – OSDC 2010

Nuremberg, 23rd and 24th June 2010
Agenda

1) About Thomas Krenn

2) IPMI basics

3) IPMI details

4) Example: Nagios/Icinga IPMI Plugin

5) Conclusions

slide 2/26
1) About Thomas Krenn

• Server systems, virtualization

and accessories
"Made in Germany"
• Unique service and support
and 24h express delivery
• History
– 2002: founded by Max Wittenzellner and Thomas Krenn
– 2005: turned into stock corporation
– 2008: DIN ISO 9001:2000 certification
– today:
• 70 employees – 20 of them being technicians ;-)
• over 9.000 customers

slide 3/26
Agenda

2) IPMI basics
● IPMI main features
● IPMI Illustration
● IPMI Messaging Interfaces
● Channel Privilege Levels
● IPMI Software overview

3) IPMI details

4) Example: Nagios/Icinga IPMI Plugin

5) Conclusions

slide 4/26
2) IPMI basics

• IPMI = Intelligent Platform Management Interface,

developed by Intel, HP, NEC, Dell
– 1998: IPMI v1.0
– 2001: IPMI v1.5
– 2004: IPMI v2.0

• IPMI main features:

– Monitoring (temperatures, fans, voltages, etc.)
– Recovery Control (power on/off/reset a server)
– Logging (System Event Log)
– Inventory (FRU information)

slide 5/26
2) IPMI basics
Remote Mmgt. Card
ICMB
(KVM over IP, ...)

LAN Serial Auxillary ICMB

Connector Connector IPMB Connector bridge

Chassis
PCI mgmt. bus IPMB mgmt.
NVS Storage (Satellite
SDR
Controller)
Network
LAN SEL
(LAN)
interface FRU
Controller
Baseboard FRU Temp.
Sensors & Controls
Management sensor
Fan sensor …
Controller Temp. sensor
(BMC) Power control Chassis board
Reset control
…
Serial BMC
Serial/Modem
Port Serial private mgmt. busses FRU
interface
Sharing Controller
FRU FRU
Redundant Power
M/B board
Temp. s.
Serial System
Controller interface Memory Processor
board board
System bus
Motherboard

slide 6/26
2) IPMI basics

• IPMI Messaging Interfaces (request/response protocol)

– System Interfaces
• local access
• requires root privileges
– Serial/Modem Interface
• access via serial interface or modem
• requires IPMI user name/password (deactivate auth. NONE)
– LAN Interface
• access via network
• requires IPMI user name/password (deactivate auth. NONE)
– (ICMB and PCI Management Bus)

slide 7/26
2) IPMI basics

• Channel Privilege Levels (for LAN/Serial access)

Privilege Level Description

Callback Lowest Privilege Level.
Allows only initiating a callback.
User Allows only IPMI 'begin' commands (query sensors).
Changing the BMC configuration, writing data to the
BMC, executing power on/off or reset commands is
prohibited.
Operator Allows nearly all IPMI commands. Only changes of
out-of-band interfaces are prohibited.
Administrator Allows all IPMI commands.

• use privilege level 'User' for monitoring purposes

slide 8/26
2) IPMI basics

• IPMI Software overview

ipmitool ipmiutil freeipmi OpenIPMI
(openipmish)
OS Support Linux, Linux, Linux, Linux
BSD, BSD, BSD,
Solaris, Solaris, Solaris,
(Windows) Windows, (Windows)
EFI
Target Market admins, admins, HPC, (Kernel-driver)
developers, developers, universities,
oem's oem's admins, ...
Included in Debian, Gentoo Debian Debian,
Linux Distros Gentoo, (Squeeze), Gentoo,
RedHat, Gentoo, RedHat,
SLES, RedHat (exp.) SLES,
Ubuntu (univ.) Ubuntu (univ.), Ubuntu

slide 9/26
Agenda

3) IPMI details
● IPMI example configuration of a LAN interface
● Remote Control
● Sensors
• Sensor Classes
• Sensor Types
• SDR (Sensor Data Record) Types
• example query with ipmitool/freeipmi
● System Event Log (SEL)
● Platform Event Filtering (PEF)
● Serial over LAN (SOL)
● Field Replaceable Unit (FRU) data

4) Example: Nagios/Icinga IPMI Plugin

5) Conclusions
slide 10/26
3) IPMI details

• IPMI example configuration of a LAN interface

[root@testserver
[root@testserver ~]#
~]# ipmitool
ipmitool lan
lan print
print 11
Set
Set in
in Progress
Progress :: Set
Set Complete
Complete
Auth Type Support
Auth Type Support :: NONE MD5
NONE MD5 PASSWORD
PASSWORD
Auth Type Enable
Auth Type Enable : Callback
: Callback : :
:: User
User :: MD5
MD5
:: Operator
Operator ::
:: Admin
Admin :: MD5
MD5
:: OEM
OEM ::
IP Address Source
IP Address Source : Static Address
: Static Address
IP Address
IP Address :: 192.168.1.211
192.168.1.211
Subnet Mask
Subnet Mask : 255.255.255.0
: 255.255.255.0
MAC
MAC Address
Address :: 00:0e:0c:ea:92:a2
00:0e:0c:ea:92:a2
[...]
[...]

slide 11/26
3) IPMI details

• Remote Control
[user@adminpc
[user@adminpc ~]$
~]$ ipmitool
ipmitool ­I­I lan
lan ­H
­H 192.168.1.211
192.168.1.211 \\
­U
­U admin
admin ­P
­P relation
relation power
power status
status
Chassis Power is off
Chassis Power is off
[user@adminpc
[user@adminpc ~]$
~]$

[user@adminpc
[user@adminpc ~]$
~]$ ipmitool
ipmitool ­I­I lan
lan ­H
­H 192.168.1.211
192.168.1.211 \\
­U
­U admin
admin ­P
­P relation
relation power
power on
on
Chassis Power Control: Up/On
Chassis Power Control: Up/On
[user@adminpc
[user@adminpc ~]$
~]$

[user@adminpc
[user@adminpc ~]$
~]$ ipmitool
ipmitool ­I­I lan
lan ­H
­H 192.168.1.211
192.168.1.211 \\
­U
­U admin
admin ­P
­P relation
relation power
power status
status
Chassis Power is on
Chassis Power is on
[user@adminpc
[user@adminpc ~]$
~]$

slide 12/26
3) IPMI details

• Sensors: Sensor Classes (1/2)

Discrete Threshold
multiple states possible: changes event status on analog
● up to 15 states reading comparison to threshold
● each state is reflected by a bit values
● multiple state bits can active at a time

can provide: provides:

● generic states (Table 42-2) or ● analog reading of the sensor and

● sensor-specific states (Table 42-3) ● discr. threshold comparison status

bit (generic state, Table 42-2)

other classes similar to discrete:
● Digital: term often used for discrete

sensors with two possible states

● OEM: discrete sensor where the

meaning of the states (offsets) are

OEM defined

slide 13/26
3) IPMI details

• Sensors: Sensor Classes (2/2)

Discrete Threshold
[root@test ~]# ipmitool sdr get "PS2 Status" [root@test ~]# ipmitool sdr get "Fan 1"
[root@test ~]# ipmitool sdr get "PS2 Status" [root@test ~]# ipmitool sdr get "Fan 1"
Sensor ID : PS2 Status (0x71) Sensor ID : Fan 1 (0x50)
Sensor ID : PS2 Status (0x71) Sensor ID : Fan 1 (0x50)
Entity ID : 10.2 (Power Supply) Entity ID : 29.1 (Fan Device)
Entity ID : 10.2 (Power Supply) Entity ID : 29.1 (Fan Device)
Sensor Type (Discrete): Power Supply Sensor Type (Analog) : Fan
Sensor Type (Discrete): Power Supply Sensor Type (Analog) : Fan
States Asserted : Power Supply Sensor Reading : 5719 (+/­ 0) RPM
States Asserted : Power Supply Sensor Reading : 5719 (+/­ 0) RPM
[Presence detected] Status : ok
[Presence detected] Status : ok
[Power Supply AC lost] Nominal Reading : 6708.000
[Power Supply AC lost] Nominal Reading : 6708.000
Assertion Events : Power Supply Normal Minimum : 2451.000
Assertion Events : Power Supply Normal Minimum : 2451.000
[Presence detected] Normal Maximum : 10965.000
[Presence detected] Normal Maximum : 10965.000
[Power Supply AC lost] Lower critical : 1720.000
[Power Supply AC lost] Lower critical : 1720.000
Assertions Enabled : Power Supply Lower non­critical : 1978.000
Assertions Enabled : Power Supply Lower non­critical : 1978.000
[Presence detected] Positive Hysteresis : 86.000
[Presence detected] Positive Hysteresis : 86.000
[Failure detected] Negative Hysteresis : 86.000
[Failure detected] Negative Hysteresis : 86.000
[Predictive failure] Minimum sensor range : Unspecified
[Predictive failure] Minimum sensor range : Unspecified
[Power Supply AC lost] Maximum sensor range : Unspecified
[Power Supply AC lost] Maximum sensor range : Unspecified
[...] Event Message Control : Per­threshold
[...] Event Message Control : Per­threshold
Deassertions Enabled : Power Supply Readable Thresholds : lcr lnc
Deassertions Enabled : Power Supply Readable Thresholds : lcr lnc
[...] Settable Thresholds : lcr lnc
[...] Settable Thresholds : lcr lnc
Threshold Read Mask : lcr lnc
Threshold Read Mask : lcr lnc
Assertion Events :
Assertion Events :
Assertions Enabled : lnc­ lcr­
Assertions Enabled : lnc­ lcr­
Deassertions Enabled : lnc­ lcr­
Deassertions Enabled : lnc­ lcr­

slide 14/26
3) IPMI details

• Sensors: Sensor Types

Sensor Types (Table 42-3)
[root@testserver ~]# ipmitool sdr type list
[root@testserver ~]# ipmitool sdr type list
Sensor Types:
Sensor Types:
Temperature Voltage
Temperature Voltage
Current Fan
Current Fan
Physical Security Platform Security
Physical Security Platform Security
Processor Power Supply
Processor Power Supply
Power Unit Cooling Device
Power Unit Cooling Device
Other Memory
Other Memory
Drive Slot / Bay POST Memory Resize
Drive Slot / Bay POST Memory Resize
System Firmwares Event Logging Disabled
System Firmwares Event Logging Disabled
Watchdog System Event
Watchdog System Event
Critical Interrupt Button
Critical Interrupt Button
Module / Board Microcontroller
Module / Board Microcontroller
Add­in Card Chassis
Add­in Card Chassis
Chip Set Other FRU
Chip Set Other FRU
Cable / Interconnect Terminator
Cable / Interconnect Terminator
System Boot Initiated Boot Error
System Boot Initiated Boot Error
OS Boot OS Critical Stop
OS Boot OS Critical Stop
Slot / Connector System ACPI Power State
Slot / Connector System ACPI Power State
Watchdog Platform Alert
Watchdog Platform Alert
Entity Presence Monitor ASIC
Entity Presence Monitor ASIC
LAN Management Subsystem Health
LAN Management Subsystem Health
Battery Session Audit
Battery Session Audit
Version Change FRU State
Version Change FRU State
slide 15/26
3) IPMI details

• Sensors: SDR (Sensor Data Record) Types

Name (Chapter 43) SDR Type
Full Sensor Record 01h
Compact Sensor Record 02h
Event-Only Record 03h
Entity Association Record 08h
Device-relative Entity Association Record 09h
Reserved Records 0Ah:0Fh
Generic Device Locator Record 10h
FRU Device Locator Record 11h
Management Controller Device Locator Record 12h
Management Controller Confirmation Record 13h
BMC Message Channel Info Record 14h
OEM Record C0h
slide 16/26
3) IPMI details

• Sensors: example query with ipmitool

– use 'ipmitool sdr type […]' for sensor type
– use 'ipmitool sdr elist […]' for SDR type
[root@testserver
[root@testserver ~]#
~]# ipmitool
ipmitool sdr
sdr type
type Other
Other
PS1 +12V Power
PS1 +12V Power || 7Ch | ok | 10.1 | 80 Watts
7Ch | ok | 10.1 | 80 Watts
PS2 +12V Power
PS2 +12V Power || 7Dh | ok | 10.2 | 104 Watts
7Dh | ok | 10.2 | 104 Watts
[root@testserver
[root@testserver ~]#
~]# ipmitool
ipmitool sdr
sdr type
type Other
Other ­v
­v
Sensor ID
Sensor ID : PS1 +12V Power (0x7c)
: PS1 +12V Power (0x7c)
Entity ID
Entity ID :: 10.1
10.1 (Power
(Power Supply)
Supply)
Sensor Type (Analog) : Other
Sensor Type (Analog) : Other
Sensor
Sensor Reading
Reading :: 80
80 (+/­
(+/­ 6)
6) Watts
Watts
Status
Status :: ok
ok
Nominal Reading
Nominal Reading : 372.000
: 372.000
Normal Minimum
Normal Minimum :: 100.000
100.000
Normal Maximum
Normal Maximum : 744.000
: 744.000
Upper critical
Upper critical :: 840.000
840.000
Upper
Upper non­critical
non­critical :: 792.000
792.000
[...]
[...]
slide 17/26
3) IPMI details

• Sensors: example query with freeipmi

[root@testserver
[root@testserver ~]# ~]# ipmimonitoring
ipmimonitoring
Record_ID
Record_ID | Sensor Name || Sensor
| Sensor Name Sensor Group
Group || Monitoring
Monitoring Status|
Status|
Sensor Units | Sensor
Sensor Units | Sensor Reading Reading
[...]
[...]
17
17 || Fan
Fan 55 || Fan
Fan || Nominal
Nominal || RPMRPM || 9052.000000
9052.000000
18 | Fan 6 | Fan | Nominal | RPM |
18 | Fan 6 | Fan | Nominal | RPM | 8060.000000 8060.000000
19
19 || PS1
PS1 AC
AC Current
Current || Current
Current || Nominal
Nominal || AA || 0.124000
0.124000
20 | PS2 AC Current | Current | Nominal |
20 | PS2 AC Current | Current | Nominal | A | 0.992000 A | 0.992000
[...]
[...]
33
33 || Power
Power Redundancy
Redundancy || PowerPower Unit
Unit || Critical
Critical || N/A
N/A || 'Redundancy
'Redundancy
Lost' 'Non­redundant:Sufficient Resources from
Lost' 'Non­redundant:Sufficient Resources from Redundant' Redundant'
34
34 || BMC
BMC Watchdog
Watchdog || Watchdog
Watchdog 22 || Nominal
Nominal || N/A
N/A || 'OK'
'OK'
35
35 | Platform Secu V | Platform Security Violation Attempt
| Platform Secu V | Platform Security Violation Attempt ||
Nominal
Nominal || N/AN/A || 'OK'
'OK'
36
36 | Physical Scrty || Physical
| Physical Scrty Physical Security
Security || Critical
Critical || N/A
N/A ||
'General
'General Chassis
Chassis Intrusion'
Intrusion'

slide 18/26
3) IPMI details

• System Event Log (SEL)

– stored in non-volatile storage

[root@testserver ~]# ipmitool sel elist

[root@testserver ~]# ipmitool sel elist
40 | 06/21/2010 | 14:29:29 | Power Supply PS1 Status | Power Supply AC lost | Asserted
40 | 06/21/2010 | 14:29:29 | Power Supply PS1 Status | Power Supply AC lost | Asserted
54 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Fully Redundant
54 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Fully Redundant
68 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Redundancy Lost
68 | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Redundancy Lost
7c | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Non­Redundant: Sufficient from Redundant
7c | 06/21/2010 | 14:29:29 | Power Unit Power Redundancy | Non­Redundant: Sufficient from Redundant
[...]
[...]
2fc | 06/21/2010 | 15:20:32 | Physical Security Physical Scrty | General Chassis intrusion | Asserted
2fc | 06/21/2010 | 15:20:32 | Physical Security Physical Scrty | General Chassis intrusion | Asserted

[root@testserver
[root@testserver ~]#
~]# ipmitool
ipmitool sel
sel elist
elist
Power
Power Supply
Supply PS1
PS1 Status
Status || Power
Power Supply
Supply AC
AC lost
lost || Asserted
Asserted
Power Unit Power Redundancy | Fully Redundant
Power Unit Power Redundancy | Fully Redundant
Power
Power Unit
Unit Power
Power Redundancy
Redundancy || Redundancy
Redundancy Lost
Lost
Power
Power Unit Power Redundancy | Non­Redundant: Sufficient
Unit Power Redundancy | Non­Redundant: Sufficient from
from Redundant
Redundant
[...]
[...]
Physical
Physical Security
Security Physical
Physical Scrty
Scrty || General
General Chassis
Chassis intrusion
intrusion || Asserted
Asserted

slide 19/26
3) IPMI details

• Platform Event Filtering (PEF)

– BMC takes selected actions on event messages
– actions can be:
• system power off
• system reset
• generating SNMP trap
– tools: freeipmi (pef-config), ipmiutil
[root@testserver
[root@testserver ~]#
~]# pef­config
pef­config ­­info
­­info
PEF
PEF version:
version: 2.0
2.0
Alert
Alert action
action support:
support: Yes
Yes
Power
Power down action support:
down action support: Yes
Yes
Power
Power reset action support:
reset action support: Yes
Yes
Power cycle action support:
Power cycle action support: Yes
Yes
OEM
OEM action
action support:
support: Yes
Yes
Diagnostic
Diagnostic interrupt action
interrupt action support:
support: Yes
Yes
[...]
[...]
slide 20/26
3) IPMI details

• Serial over LAN (SOL) (1/2)

– allows text-based access to
• BIOS
• GRUB
• operating systems
– Linux serial console
– Windows debugger
(http://support.microsoft.com/kb/151981/)
– SOL is implemented as a payload type in RMCP+

slide 21/26
3) IPMI details

• Serial over LAN (SOL) (2/2)

slide 22/26
3) IPMI details

• Field Replaceable Unit (FRU) data

[root@testserver
[root@testserver ~]#
~]# ipmitool
ipmitool fru
fru print
print
FRU
FRU Device Description : Builtin FRU Device
Device Description : Builtin FRU Device (ID
(ID 0)
0)
Chassis Type
Chassis Type : Rack Mount Chassis
: Rack Mount Chassis
Chassis Serial
Chassis Serial :: 9000041568
9000041568
Chassis Extra
Chassis Extra : SR2500LX
: SR2500LX
Board Mfg Date
Board Mfg Date :: Sat
Sat Apr
Apr 28
28 11:37:00
11:37:00 2007
2007
Board Mfg
Board Mfg : Intel
: Intel
Board Product
Board Product :: S5000PAL0
S5000PAL0
Board Serial
Board Serial : BZAU71700054
: BZAU71700054
Board Part Number
Board Part Number :: D13607­805
D13607­805
Product Manufacturer : Intel
Product Manufacturer : Intel
Product
Product Name
Name :: S5000PAL
S5000PAL
Product Part Number
Product Part Number : ..........
: ..........
Product Serial
Product Serial :: .........
.........
[...]
[...]
FRU
FRU Device
Device Description
Description :: Pwr
Pwr Supply
Supply 11 FRU
FRU (ID
(ID 2)
2)
Product Manufacturer : DELTA
Product Manufacturer : DELTA
Product
Product Name
Name :: DPS­750EBA
DPS­750EBA
Product Part Number
Product Part Number : D20850­006
: D20850­006
Product Version
Product Version :: 02
02
Product Serial
Product Serial : DLD0712004978
: DLD0712004978
slide 23/26
Agenda

1) About Thomas Krenn

2) IPMI basics

3) IPMI details

4) Example: Nagios/Icinga IPMI Plugin

5) Conclusions

slide 24/26
4) Example: Nagios/Icinga IPMI Plugin

• how the Nagios IPMI Sensor Monitoring Plugin works

– it's a shell script (Bash)
– it uses ipmitool, gawk
– you can use the plugin with every IPMI-compatible server
– it follows the Nagios plug-in development guidelines
– clear illustration within the Nagios web interface

slide 25/26
5) Conclusions

• IPMI has multiple very useful features

• IPMI is built into most modern Servers
• IPMI has rarely been used until now

→ so start using the Power of IPMI today ;-)

• Further information
– http://download.intel.com/design/servers/ipmi/
IPMI2_0E4_Markup_061209.pdf
– http://www.thomas-krenn.com/de/wiki/IPMI_Grundlagen
– http://www.thomas-krenn.com/ipmi-plugin
– http://lists.thomas-krenn.com
slide 26/26