Sys Admin Pocket Survival Guide
Sys Admin Pocket Survival Guide
HOW-TO
AIX ADMIN 101
PMR - Problem Management Record.
PTF - Program Temporary Fix eg Unnnnnn
APAR - Authorized Program Analysis Report, specific release of PTF. eg IYnnnnn, IXnnnnn
ML - Maintenance Level, grouping of APAR. eg 5300-04 is ML 4
SP - Service Pack (starting aix 5.3 ML 3) eg 5300-03-01 is SP 1 on ML3
TL -
SMIT
A lot of the sys admin task in AIX can be done thru smit (or smitty for TUI version). It can accept a parameter to get to a
specific menu section right away.
Here are some example:
smitty fastpath:
Keystrokes:
F3 or ESC, 3 Cancel, go back
F4 or ESC, 4 Provide list of items to choose from
F6 or ESC, 6 Show command that smit will actually run
F10 or ESC, 10 Exit smitty
LINUX AFFINITY
AIX is affectionally known as It Ain't uniX. With version 5L that added Linux Affinity, it maybe called Ain't It linuX?
The linux toolkit does add lot of open source utilities for AIX, it even provides an rpm package manager to manipulate
these open source tools. However, it is stuck in the past, circa 2002, and so many of the newer Linux updates are not
available. For example, there is no rpmbuild but use the old rpm --recompile. There is no automatic patch to update the
source codes from Source RPM. These deficiencies makes development in AIX extremely painful.
AIX Idiosyncrasies
If a file is held open by the OS with a running program, it will not allowed to be overwritten (think Windows). For
example, /lib/security/NSS_LDAP and /usr/lib/netsvc/dynload/nss_ldap.so are modules loaded by the authentication
subsystem, and these files are locked and cannot be upgraded.
FS LAYOUT
AIX uses a pretty good volume manager out of the box. Every FS can be expanded on the fly, so no special work is
really needed on initial setup. If you use /home as NFS mounted home dir, you may wish to update /etc/filesystems so
that the the local disk is mounted as /localhoome or something.
BUNDLE INSTALL
AIX that comes with the machine often don't have all the desired software packages installed. It maybe good to add some
bundles before releasing the machine for general use.
1. AllDevicesKernels
2. Alt_Disk_Install
3. App-Dev
4. CC_EVAL.Graphics
5. CDE
6. GNOME
7. Graphics
8. KDE
9. Kerberos_5 (Expansion Pack)
10. Media-Defined
11. Mozilla (AIX Toolbox for Linux Applications, Mozilla CD)
12. PerfTools
13. Server
Note that not everything is in the first CD. Time for IBM to get with DVD media, no? :)
FIRMWARE/BIOS
When upgrading an older machine to new version of AIX, be sure to check the AIX release notes to ensure the machine
in question is still supported, and whether any firmware update maybe needed.
RS/6000 / pSeries machine firmware is typically updated when AIX is still running. Many older machines does not
support firmware update in SMS. Old hardware or firmware have tendency to crap out in a boot process giving error
20EE000B (unable to find boot device), even when boot from the CD for the initial install works fine. The saving grace
is that one can boot and reinstall the old OS should machine become unbootable.
Platform Firmware:
ROM Level.(alterable).......3R040323
Version.....................RS6K
System Info Specific.(YL)...U0.1-P1/Y1
Physical Location: U0.1-P1/Y1
# perform the update, it will reboot the machine automatically when done
cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/3R041029.img
/etc/rc.d/rcN.d/
same as solaris, Snn--- script, but there is nothing in here from basic OS.
normal run level is 2. Placed S74autofs and S89sshd in rc2.d .
/etc/rc.d/rc script is the one that source all the rcN.d,
but this script doesn't seems to exist by default,
had to be copied from another 4.3.3 machine and then it was fine.
For some unknown reason, init script for AIX 5.3 doesn't seems to run
correctly!! (eg, opensshd script will create log, but fail to start)
To install new OS to system, change system to get into System Management Service (SMS) mode. Then choose to boot
from cdrom (first CD for OS install).
A note about the old desktop machines (eg 43P-140 604e-332 PowerPC 760):
1. When machine first boot, it will display various number on the various terminal.
eg. 0 for ASCII console (dumb terminal), 1 on first graphics video, 2 on second graphics video.
Press the number corresponding to the video that you wish to use.
This setting is "memorized" by the system, prompt again only when a there is hardware config change.
2. To get into the graphical System Management Service (SMS) mode, press F1 as soon as the double beep "dirip-
dirip" chirps happens (and the keyboard icons is displayed).
You better be quick, do it before the long chime of the speaker test happens (the fifth/last icon in startup: speaker
icon).
If using ASCII console, F-keys are simulated via ESC+Number keys. The graphical icons won't appear, but the
equivalent text will be displayed progressively across the screen:
3. Pressing F8 will go to the open firmware OK prompt. There are only a few commands accepted:
devalias
printenv
setenv
boot [DEVICE]
boot disk will do a normal boot from hard drive. This is the default action if open firmware is not explicitly
invoked.
boot cd will boot from the cdrom and run the installer program, whereby it will prompt for install config info.
There is also option to get into maintenance mode (?? It is NOT boot -s !!)
More info about open firmware: IEEE 1275.
4. SMS has entry to define boot device scan sequence, which is based on SCSI (lowest ID first). However, actual
boot is according to Open Firmware boot command, so which device is booted depends on what the alias for disk
is defined as. In summary, ignore the SCSI id boot sequence in SMS.
5. Some system require pressing F5 instead of F1 to get into SMS mode. If you are lucky, some will display what key
to press to get into what mode right before the hardware scan progress, eg:
6. IBM RS/6000 hardware is pretty finiky and OS support may not be available for some machines. AIX 5.2 will no
support any machines with ISA bus. Some 43P have such bus and thus can only run AIX 5.1 as latest OS. Other
43P that don't have ISA bus can run the newer AIX. If you wish to install Linux, expect even more finickyness, as
there are lot of small modifications between motherboards of different serial numbers. Linux may run on one 43P,
but not another that looks the same. ie: don't install Linux on these putters!
---
---
EMC navisphere agent also started in inittab, I guess it is the AIX way.
it calls /etc/rc.agent
----
ibm option number for universal (oem) cabinet rack mount rail part. OEM means 3rd party cabinet. IBM cabinet rails are
fixed length!
p5 9113-550 :: 7162 IBM/OEM RACK-MOUNT DR.RAIL KIT 1 168.00
STORAGE
DISK/VOLUME MANAGEMENT/FILESYSTEM
AIX has probably the most admin-friendly Logical Volume Manager and Filesystem of all the major Unices out there. It
use a single LVM for every FS. Every FS is extendable online without the need to unmount. It does have lot of
commands that are needed to manipulate the system though. But if you just need to get more space on a given
FS/partition, just use smit chfs.
TLA name eg sample cmd
--- ------------- ------------------------ --------------
PP physical part
LP logical part
lsdev -Cc adapter # list many adapters, include fc, scsi, ide, pci, LPAR virtual, usb
lsdev -Cc disk # list hard disks. eg hdisk0 hdisk1
rmdev -dl hdisk2 # remove a named disk from config, often needed when san disk has b
# changed on the storage server side. after rm, rescan the bus.
mklv -y'appvol00' appvg 1 # create logical volume (after volume group defined via mkvg)
# if -y is not given, a generic volume name will be assigned.
# This step isn't strictly needed, mkfs will create generic vol as
mkvg -f -y'appvg' hdisk2 hdisk3 # create a VG appvg from 2 physical hd
mkvg # original Vol Grp (Max 32 PV, 255 LV)
mkvg -B # Big Volume Group (Max 128 PV, 512 LV, 4.3.1+)
mkvg -S # Scalable Vol Group (Max 1024 PV, 256 LV, 5.2+)
chfs -a size=+1 /dev/hd2 # grow a file system, size mult of 512 bytes, but
# always rounded up to integer number of PP
# 8 mb, 64 mb, etc (see lsvg rootvg pp size)
# to add 256mb, use size=+524288
# 256 * 1024*1024 / 512 = 524288 blocks
# ie, use size val of [ X MB * 2048 (constant) ]
chfs -a size=+1 /tmp # /dev/hd2 can be substituted by the fs mount pt.
smitty chfs # will see SIZE for more accurate resizing.
# count is # of 512 bytes block,
# value is absolute num only.
# Can also strink fs while mounted (at least tested in 5.3)
cplv # copy LV
chlv # change LV,
# cplv + chlv = move Log Vol b/w Vol Grp.
migratepv -l LVname oldPV newPV # move a fs w/in same vol grp, but diff hd. eg
migratepv -l paging00 hdisk1 hdisk2 # migrate pagin, can take a while!
getlvodm -u rootvg # Determine if ODM and VGDA data are correct (in sync)
chvg -u rootvg # unlick rootvg (maybe locked during crash, ODM problems)
getlvodm/putlvodm # disk ODM repair related.
mirrorvg rootvg # mirror whole volume group, simple one command does it.
# aix book p 202
SWAP / PAGING
/etc/swpaspaces # config file
swapon -a # activate all swap spaces
swapon /dev/paging00 # activate additional device
swapoff /dev/paging00 # deactivate
CONNECTIVITY (NETWORK)
SERIAL/CONSOLE PORT
DB9 connector wiring is slightly different than Sun. Use a real null modem cable. "Psseudo" cable from sun DB9-RJ45
adaptors and ribbon cable don't work. One will see text and be able to type text, but it won't accept the login, even when
password is right. can't loging!! Terminal is VT100, none of the flow control stuff, but that probably don't matter. Also,
for running system, when NULL modem is connected to serial, it should automatically display new text and prompt for
login. ribbon cable probably don't work at all here.
P4 and older machines use standard 9600 for serial port. But new POWER5 machines, they changed serial port to use
19200!! All other settings are same as before.
DUMB TERMINAL
The tip command exist on AIX, but it is slightly different than Solaris.
# aix tip host spec, typically placed in /etc/remote-file
# with a /etc/remote file defining its location
# alternatively, env var REMOTE can define location of this remote-file
#
# aix tip escape sequence is same as solaris.
# ~? produces full list.
# ~^D terminate the connection.
#
# tty0 is the system console, like Solaris Serial A, not usually usable as dumb terminal em
# tty1 is the serial port labeled T2, the equi of serial B in Solaris.
#
# IBM AIX serial connection is slightly diff than Sun, req NULL modem cable.
# This emulated TIP does not fully act as dumb terminal :(
# Depending on the state of the machine, and whether tip session is established first,
# or the cable is connected first, text may or may not be visible :(
# So, tip in aix can probably only work as dumb terminal for sun and hp machines,
# and the most reliable console is still a dumb terminal.
#
# eg usage, runnable as normal user:
# export REMOTE=$HOME/etc/remote-file
# tip dumb19200
# tip -9600 serial1
#
#
dumb9600|direct connect at 9600 bps,br#9600:dv=/dev/tty1
dumb19200|direct connect at 19200 bps,br#19200:dv=/dev/tty1
serial0|dumbterminal:dv=/dev/tty0
serial1|tty1|dumbterminal:dv=/dev/tty1
tty|dumbterminal:br#19200:dv=/dev/tty
#
# The serial ports ttyp0...ttyp9,ttypa...ttypf do not usually exist
#
ttyp0|dumbterminal:br#19200:dv=/dev/ttyp0
ttyp1|dumbterminal:br#19200:dv=/dev/ttyp1
ttyp2|dumbterminal:br#19200:dv=/dev/ttyp2
ttyp3|dumbterminal:br#19200:dv=/dev/ttyp3
ttyp9|dumbterminal:br#19200:dv=/dev/ttyp9
ttypa|dumbterminal:br#19200:dv=/dev/ttypa
ttypb|dumbterminal:br#19200:dv=/dev/ttypb
ttypc|dumbterminal:br#19200:dv=/dev/ttypc
ttypf|dumbterminal:br#19200:dv=/dev/ttypf
cu Connect Unix.
Opening a serial line may need some file config, cli below get error :(
I guess need to get minicom/telix or some other similar program from the Linux world.
cu -s9600 -l/dev/ttyp1
-s speed
-l device line
cu commands:
~%b send a break
~%! give shell
~. end session, logout user.
NIC
ifconfig
netstat -i
NETWORK SERVICES
lssrc -a # list all services and their status
startsrc # start network service
stopsrc # stop network service
-g # group
-s # subsystem, g or s is largely how AIX define the service.
-a #
-D # Debug
DNS
/etc/resolv.conf # main config file, like other Unix.
NIS
chypdom -B titaniumleg.com # set the NIS domain for the machine
# it proabably edit /etc/rc.nfs to run cmd domainname
domainname # show the NIS domain
smitty yp # general YP NIS config
smitty mkclient # configure machine as NIS client
rmyp -c # remove yp client config
lssrc -s ypbind # display active vs inoperative NIS
stopsrc -s ypbind # nis client process
startsrc -s ypbind
# /etc/rc.nfs is where the ypbind is set to use specific server (if not broadbcast)
CD
AIX does not have a /etc/nsswitch.conf, instead use /etc/netsvc.conf
and /etc/irs.conf (Information Retrieval System, bind 8 code base, used in FreeBSD also).
eg of irs.conf
Okay, here a real sample of /etc/irs.conf from an AIX 5.3 machine:
hosts dns continue
hosts nss_ldap continue
hosts nis continue
hosts local
services nss_ldap continue
services nis continue
services local
networks dns continue
networks nss_ldap continue
networks nis continue
networks local
netgroup nss_ldap continue
netgroup nis continue
netgroup local
protocols nss_ldap continue
protocols nis continue
protocols local
###
### http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.doc/
ount.htm
###
automount nis_ldap
##automount nis
##automount files
##automount nis_ldap nis files
eg of /etc/netsvc.conf
NFS
nfso -a # show all nfs options and current value
nfso -o varname[=val] # display/set nfs option.
nfso -o nfs_use_reserved_ports=1
# AIX default does not use port <1024, which result in auth error.
# http://www.faqs.org/faqs/aix-faq/part3/
# only needed for /net with certain solaris version?
/nethome:
dev = "/vol/vol1/users"
vfs = nfs
nodename = netapp
mount = true
options = rw,fg,soft,intr,nosuid
account = false
/Import:
dev = "/vol/vol1/depot"
vfs = nfs
nodename = netapp
mount = false
options = ro,fg,hard,intr,nosuid
account = false
/usr/sbin/automount -M /home /-
CLI:
/usr/lib/lpd/pio/etc/piomisc_ext mkpq_remote_ext -q 'QueueName' -h 'Hostname' -r 'RemoteQueueName'
SYS CONFIG
GENERAL SA COMMANDS
oslevel -rq # list all known and recommended ML
oslevel -r # which ML (maint level) machine is at.
oslevel -l 4.3.3.0 # list fileset missing to bring machine to specified level.
oslevel -l 5200-02 -r # aix 5L is weired wrt oslevel -l
oslevel -s # service pack, in addition to ML, not avail on all sys.
# ( -s is new flag starting ca 2006, aix 5.3 ML 3? not in base 5.3)
watch cmd -o logfile # similar to truss cmd, but can't watch existing pid
truss # exist for 5.x, in /bin
----
NIM Network Install Manager
- Network installation of aix, equiv of solaris jumpstart
- r-cmd to control remote machine
CONFIG FILES
/etc/passwd
/etc/security/passwd # similar to shadow file
/etc/security/login # dictate shells that are valid shell for various process.
# if user shell not listed here, ftp will fail.
/etc/ftpaccess # set ftp access priviledges, allow/deny access, etc.
/etc/nologin # tmp disable login, presence of file set this behaviour
/etc/filesystems # instead of /etc/fstab
# /etc/syslog.conf
# IBM don't seems to have any standard, there are no default settings.
# *.info /var/adm/syslog_info.log
*.notice /var/adm/syslog_notice.log rotate size 100k files 4
# *.warning /var/adm/syslog_warning.log
# *.err /var/adm/syslog_err.log
*.crit /var/adm/syslog_crit.log
# /etc/filesystems
/:
dev = /dev/hd4
vol = "root"
mount = automatic
check = false
free = true
vfs = jfs2
log = /dev/hd8
type = bootfs
/home-old:
dev = /dev/hd1
vol = "/home"
mount = true
check = true
free = false
vfs = jfs2
log = /dev/hd8
/usr:
dev = /dev/hd2
vfs = jfs2
log = /dev/hd8
mount = automatic
check = false
type = bootfs
vol = /usr
free = false
/var:
dev = /dev/hd9var
vfs = jfs2
log = /dev/hd8
mount = automatic
check = false
type = bootfs
vol = /var
free = false
/tmp:
dev = /dev/hd3
vfs = jfs2
log = /dev/hd8
mount = automatic
check = false
vol = /tmp
free = false
/proc:
dev = /proc
vol = "/proc"
mount = true
check = false
free = false
vfs = procfs
/opt:
dev = /dev/hd10opt
vol = "/opt"
mount = true
check = true
free = false
vfs = jfs2
log = /dev/hd8
/lhome:
dev = /dev/vol1
vfs = jfs2
log = /dev/loglv00
mount = true
options = rw
account = false
/home_na:
dev = "/vol/vol1/users"
vfs = nfs
nodename = na_data
mount = true
options = rw,fg,hard,intr,nosuid
account = false
/mnt/import:
dev = "/vol/vol1/depot"
vfs = nfs
nodename = na_data
mount = true
options = rw,fg,hard,intr,nosuid
account = false
/mnt/cdrom:
dev = "/dev/cd0"
vfs = cdrfs
mount = false
check = false
options = ro
vol = "/mnt/cdrom"
free = false
HARDWARE COMMANDS
lspci
lsattr
Hot swap PCI cards, or hot remove. Essentailly, issue command to identify, then power off a given pci slot, then
reactivate it again.
smit, devices, pci hot plug manager
lsdev -Cc processor # see number of processor on machine (may not be numbered in sequen
lsattr -El proc0 # find processor speed
----
LPAR sysmtem
From HMC (hardware manager console), cmds in /opt/hsc/bin
PACKAGE/SOFTWARE MANAGEMENT
lslpp -h bos.rte # show fileset history (from package mgnt?)
lslpp -L bos.rte # snapshot of main levele and statet of fileset
lslpp -f bos.rte # see what files are installed by a given fileset.
lslpp -l bos.net.nfs.client # show fileset revision level
lppchk -v # chk see if any lpp product are broken, obsolete, missing
lslpp -l | grep vacpp # show all fileset info about the ibm compiler
installp flags
a apply update (def action)
c commit
g recursively install prereq
X eXpand (local) FS is more space is needed.
d file specify the location of the media file ( . for current dir).
p preview only, nothing will actually be done.
eg:
-d # delete license
-v # opt vendor info (for some custom lic, used by VA C++
-p # opt product info
-U # update license
-T # specify number of user, cn license only
eg
14blt -a -f /usr/vac/c44aix_cn.lic -T 10 -R "registration_comment_avoid_space"
# enroll a license. -T 10 inidcates number of licenses.
i4blt -U -v "'IBM Software Solutions Toronto'" -p "'VisualAge C++ Professional' 6.0.n" -T 100
# Update license to allow 100 users. Only for _cn license type.
# Be very careful about quotes! Name with spaces need additional
# single quotes around them!!! Awful syntax and very cryptic error messge!!
i4blt -d -n server_name -v "'vendor name with space'" -p "'product name' product_version" -t timest
i4blt -d -v "'IBM Software Solutions Toronto'" -p "'VisualAge C++ Professional' 6.0.n" -t 102229180
i4blt -d -v "'IBM Software Solutions Toronto'" -p "'C for AIX' 6.0.n" -t 1022291952
# delete a license.
# -t is timestapm retrieved from output of i4blt -lp -i
Visual Age compiler license file installation requires the package bos.adt.libm For 5.1L, it is in CD 4/6. However, its
install need to kick started from CD 1, by requesting the install of all all componets of bos.adt, which is not installed by
default.
APPLYING ML
http://techsupport.services.ibm.com/server/mlfixes/43/
http://techsupport.services.ibm.com/server/mlfixes/43/11/00to11.html
inutoc ./usr/sys/inst.images/
installp -acgXd ./usr/sys/inst.images bos.rte.install
smit update_all #
cd /home_sc/sa/share/aix43.ml/433.noLevel_to_11
inutoc . # this creates the .toc file
installp -acgXd . bos.rte.install # install the basic os patch ?
smit update_all # add the other patches
oslevel -r # + reboot
SOFTWARE PATCHES
aka Program Temporary Fix
instfix -i # show applied APAR fix
instfix -i | grep AIX_ML # fix done by ML patch
instfix -i -k IY78933 # see if a given IY APAR fix had been applied.
instfix -ivq | grep xlC.aix5.*rte | sort -u # list installed xlC runtime environment
/usr/lib/instl/sm_inst installp_cmd -a -Q -c -N -g -X -G \
-d '/nfshome/sa/share/aixX.smit/aix_vacpp_rte7_nov2005/extract' -f _all_latest
# install the whole RunTimeEnv v8 for "xlC v7"
/usr/lib/instl/sm_inst installp_cmd -a -c -N -g -X \
-d '/nfshome/sa/share/aixX.smit/aix_vacpp_rte7_nov2005/extract' -f _update_all
# apply relevant updates from RunTimeEnv v8 for "xlC v7"
# it will not display failure msg when optional debugger
# components are not installed.
use dir of
cdrom/usr/sys/inst.images
eg /nethome/sa/software/VisualAge_5.0.2_cd/usr/sys/inst.images
/nethome/sa/software/VisualAge_5.0.2_cd/usr/sys/mvCD
mvCD may have complain of wrong path, but it generally list more stuff and install more programs, e
mvCD may be listing all software in the whole CD set, not just a given CD.
mvCD would fail if content is network path and no CD media change is possible.
actually, aix5l book suggest use of inst.images dir (build .toc before install if obtaining as tar
Applying patches :
/nethome/sa/software/VisualAge_patch_to_5.0.2.7
it is like installing normal software, but the tar didn't have a .toc file.
thru smit, there was an entry to list software in media, run it so that it created a .toc file.
After that, install the patch as software (via smit update_all), which generate commands like:
File:
xlC.adt.include 6.0.0.0
xlC.aix50.rte 6.0.0.0
xlC.aix50.rte 6.0.0.5
xlC.msg.en_US.rte 6.0.0.0
xlC.rte 6.0.0.0
Instead of putting CD into machine CD-ROM drive, one can specify network shared CD-ROM
such as /net/tin-sun/cdrom, but it will NOT prompt to change CD,
and any language locale pack selected not avail on network path will simply not be queue for
install, so not even error message. If the CD has nothing about the related
language, it will display error. Most International language locale start
with CD #5.
----
Can use Remove Lang Env to see list of installed lang translation locale.
Sometime adding a language locale complain missing file set pre-req. do:
cd directy to the dir with the bos.loc.com file, and install it manually.
smitty installp on any of the CD doesn't show option to add this fileset!
cd /share/aix5.2/disk1/usr/sys/inst.images
installp -acgXd . bos.loc.com
May also need to put in CD #1 and use smitty installp, and add the necessary
bos.iconv set appropriate to the desired language.
---
chkbd
lsfont
lskbd
LOOSE ENDS
AIX dev env does not use LD_LIBRARY_PATH, but instead use LIBPATH
portmon
REFERENCE
rootvg.net AIX portal, lot of sutff!
AIX CD CATALOG
AIX does not have a "install everything" option. So, one often has to dig thru CDs to find the right package to add a
desired feature. This file list what is in some of the CD. The vary by releases, so pretty long list. But easier to search here
than keep on swapping CDs :)