http://www.scribd.com/archive/plans?

doc=61299788

[ohasd(3941)]CRS-8017:location: /etc/oracle/lastgasp has 60 reboot advisory log

files, 0 were announced and 0 errors occurred
2012-08-30 17:44:05.798
[ohasd(3941)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-08-30 17:44:12.551
[cssd(4195)]CRS-1713:CSSD daemon is started in clustered mode
[client(4319)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4330)]CRS-10001:ACFS-9322: done.
2012-08-30 17:44:23.070
[cssd(4195)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 17:44:23.112
[cssd(4195)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 17:44:33.665
[cssd(4195)]CRS-1625:Node rac2, number 2, was manually shut down
2012-08-30 17:44:41.272
[cssd(4195)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-08-30 17:44:43.051
[ctssd(4392)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-08-30 17:44:43.827
[ctssd(4392)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-08-30 17:45:24.292
[crsd(4616)]CRS-1012:The OCR service started on node rac1.
2012-08-30 17:45:43.656
[crsd(4616)]CRS-1201:CRSD started on node rac1.
2012-08-30 17:46:17.446
[crsd(4616)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-08-30 17:46:17.450
[crsd(4616)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-08-30 17:52:14.903
[ohasd(3941)]CRS-2765:Resource 'ora.drivers.acfs' has failed on server 'rac1'.
[client(6302)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(6306)]CRS-10001:ACFS-9322: done.
2012-08-30 18:00:57.536
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5818:Abor
ted command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers
.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log
/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 18:01:07.008
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5014:Agen
t "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% " timed out
starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for actio
n "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1
/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 18:01:54.316
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5818:Abor
ted command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers
.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log

/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 18:01:56.055
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

% (4077)]CRS-5014:Agen
t "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% " timed out
starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for actio
n "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1
/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 18:02:54.940
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5818:Abor
ted command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers
.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log
/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 18:02:57.826
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5014:Agen
t "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% " timed out
starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for actio
n "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1
/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 18:05:07.011
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5818:Abor
ted command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers
.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log
/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 18:05:19.604
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% (4077)]CRS-5014:Agen
t "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
% " timed out
starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for actio
n "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1
/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 18:08:22.207
[ohasd(4048)]CRS-2112:The OLR service started on node rac1.
2012-08-30 18:08:22.679
[ohasd(4048)]CRS-8017:location: /etc/oracle/lastgasp has 62 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-08-30 18:08:30.078
[ohasd(4048)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-08-30 18:08:33.294
[ohasd(4048)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-08-30 18:08:36.947
[cssd(4312)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 18:08:47.678
[cssd(4312)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 18:08:47.779
[cssd(4312)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 18:08:58.515
[cssd(4312)]CRS-1625:Node rac2, number 2, was manually shut down
2012-08-30 18:09:17.200

[cssd(4312)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

2012-08-30 18:09:22.011
[ctssd(4378)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-08-30 18:09:22.158
[ctssd(4378)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
[client(4570)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4585)]CRS-10001:ACFS-9322: done.
2012-08-30 18:12:02.623
[crsd(4887)]CRS-1012:The OCR service started on node rac1.
2012-08-30 18:12:11.218
[crsd(4887)]CRS-1201:CRSD started on node rac1.
2012-08-30 18:12:29.003
[crsd(4887)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-08-30 18:12:29.004
[crsd(4887)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-08-30 18:27:48.950
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
u( (4270)]CRS-5818:Ab
orted command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drive
rs.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 18:27:56.937
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
u( (4270)]CRS-5014:Ag
ent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
u( " timed ou
t starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for act
ion "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid
_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 18:40:41.147
[crsd(4887)]CRS-2765:Resource 'ora.scan1.vip' has failed on server 'rac1'.
2012-08-30 18:40:41.440
[crsd(4887)]CRS-2771:Maximum restart attempts reached for resource 'ora.scan1.vi
p'; will not restart.
2012-08-30 18:40:41.664
[crsd(4887)]CRS-2765:Resource 'ora.rac1.vip' has failed on server 'rac1'.
2012-08-30 18:40:41.664
[crsd(4887)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac1.vip
'; will not restart.
2012-08-30 18:40:41.677
[crsd(4887)]CRS-2765:Resource 'ora.rac2.vip' has failed on server 'rac1'.
2012-08-30 18:40:41.677
[crsd(4887)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac2.vip
'; will not restart.
2012-08-30 18:40:41.679
[crsd(4887)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
2012-08-30 18:40:47.151
[cssd(4312)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-08-30 18:40:47.682
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor(4246)]CRS-5822:Agent '/u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root' disconnected from ser
ver. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/ra
c1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-08-30 18:40:47.694
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4268)]CRS-5822:Agent '/u01
/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root' disconnected from server.
Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/a

gent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-08-30 18:44:01.125
[ohasd(4018)]CRS-2112:The OLR service started on node rac1.
2012-08-30 18:44:01.608
[ohasd(4018)]CRS-8017:location: /etc/oracle/lastgasp has 64 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-08-30 18:44:09.752
[ohasd(4018)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-08-30 18:44:16.686
[cssd(4283)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 18:44:25.299
[cssd(4283)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 18:44:25.340
[cssd(4283)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 18:44:37.031
[cssd(4283)]CRS-1625:Node rac2, number 2, was manually shut down
2012-08-30 18:44:43.949
[cssd(4283)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-08-30 18:44:45.243
[ctssd(4550)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-08-30 18:44:46.033
[ctssd(4550)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
[client(4732)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4739)]CRS-10001:ACFS-9322: done.
2012-08-30 18:45:47.787
[crsd(4987)]CRS-1012:The OCR service started on node rac1.
2012-08-30 18:45:49.980
[crsd(4987)]CRS-1201:CRSD started on node rac1.
2012-08-30 18:45:55.082
[crsd(4987)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-08-30 18:45:55.083
[crsd(4987)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-08-30 19:12:19.438
[cssd(4283)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-08-30 19:12:46.559
[cssd(4283)]CRS-1625:Node rac2, number 2, was manually shut down
2012-08-30 19:12:48.741
[cssd(4283)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-08-30 19:13:29.920
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:13:35.436
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:14:26.977
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a

cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r

ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:14:35.148
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:15:26.447
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:15:35.630
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:16:26.371
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:16:39.677
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:17:27.333
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:17:37.417
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:18:29.843
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a

cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r

ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:18:42.515
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:19:26.493
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:19:45.673
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:21:29.155
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:21:42.259
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:22:27.417
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:22:28.021
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:23:26.732
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a

cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r

ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-08-30 19:23:26.774
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4242)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 19:46:27.029
[crsd(4987)]CRS-2765:Resource 'ora.scan1.vip' has failed on server 'rac1'.
2012-08-30 19:46:27.510
[crsd(4987)]CRS-2771:Maximum restart attempts reached for resource 'ora.scan1.vi
p'; will not restart.
2012-08-30 19:46:28.000
[crsd(4987)]CRS-2765:Resource 'ora.rac2.vip' has failed on server 'rac1'.
2012-08-30 19:46:28.016
[crsd(4987)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac2.vip
'; will not restart.
2012-08-30 19:46:28.129
[crsd(4987)]CRS-2765:Resource 'ora.rac1.vip' has failed on server 'rac1'.
2012-08-30 19:46:28.129
[crsd(4987)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac1.vip
'; will not restart.
2012-08-30 19:46:28.132
[crsd(4987)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
2012-08-30 19:46:32.873
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4240)]CRS-5822:Agent '/u01
/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root' disconnected from server.
Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/a
gent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-08-30 19:49:11.940
[ohasd(4018)]CRS-2112:The OLR service started on node rac1.
2012-08-30 19:49:12.409
[ohasd(4018)]CRS-8017:location: /etc/oracle/lastgasp has 66 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-08-30 19:49:20.341
[ohasd(4018)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-08-30 19:49:23.177
[ohasd(4018)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
[client(4486)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4495)]CRS-10001:ACFS-9322: done.
2012-08-30 19:49:28.723
[cssd(4457)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 19:49:37.820
[cssd(4457)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 19:49:37.938
[cssd(4457)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 19:49:49.480
[cssd(4457)]CRS-1625:Node rac2, number 2, was manually shut down
2012-08-30 19:49:56.366
[cssd(4457)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-08-30 19:49:57.437
[ctssd(4711)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-08-30 19:49:58.175

[ctssd(4711)]CRS-2401:The Cluster Time Synchronization Service started on host r

ac1.
2012-08-30 19:50:46.417
[crsd(4983)]CRS-1012:The OCR service started on node rac1.
2012-08-30 19:50:48.762
[crsd(4983)]CRS-1201:CRSD started on node rac1.
2012-08-30 19:50:53.787
[crsd(4983)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-08-30 19:50:53.788
[crsd(4983)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-08-30 20:20:16.555
[crsd(4983)]CRS-2765:Resource 'ora.rac2.vip' has failed on server 'rac1'.
2012-08-30 20:20:16.967
[crsd(4983)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac2.vip
'; will not restart.
2012-08-30 20:20:17.645
[crsd(4983)]CRS-2765:Resource 'ora.scan1.vip' has failed on server 'rac1'.
2012-08-30 20:20:17.670
[crsd(4983)]CRS-2771:Maximum restart attempts reached for resource 'ora.scan1.vi
p'; will not restart.
2012-08-30 20:20:18.268
[crsd(4983)]CRS-2765:Resource 'ora.rac1.vip' has failed on server 'rac1'.
2012-08-30 20:20:18.271
[crsd(4983)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac1.vip
'; will not restart.
2012-08-30 20:20:18.296
[crsd(4983)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
2012-08-30 20:20:21.819
[cssd(4457)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-08-30 20:20:22.486
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4332)]CRS-5822:Agent '/u01
/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root' disconnected from server.
Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/a
gent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-08-30 20:20:22.482
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor(4265)]CRS-5822:Agent '/u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root' disconnected from ser
ver. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/ra
c1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-08-30 20:44:57.658
[ohasd(4009)]CRS-2112:The OLR service started on node rac1.
2012-08-30 20:44:58.126
[ohasd(4009)]CRS-8017:location: /etc/oracle/lastgasp has 68 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-08-30 20:45:06.167
[ohasd(4009)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
[client(4351)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4399)]CRS-10001:ACFS-9322: done.
2012-08-30 20:45:14.083
[cssd(4409)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 20:45:22.072
[cssd(4409)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 20:45:22.105
[cssd(4409)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 20:45:33.515
[cssd(4409)]CRS-1625:Node rac2, number 2, was manually shut down
2012-08-30 20:45:40.406
[cssd(4409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-08-30 20:45:41.480

[ctssd(4472)]CRS-2407:The new Cluster Time Synchronization Service reference nod

e is host rac1.
2012-08-30 20:45:42.320
[ctssd(4472)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-08-30 20:46:22.278
[crsd(4699)]CRS-1012:The OCR service started on node rac1.
2012-08-30 20:46:24.527
[crsd(4699)]CRS-1201:CRSD started on node rac1.
2012-08-30 20:46:28.487
[crsd(4699)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-08-30 20:46:28.496
[crsd(4699)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-08-30 20:52:34.973
[cssd(4409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-08-30 21:05:17.819
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4867)]CRS-5822:Agent '/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_r
oot' disconnected from server. Details at (:CRSAGF00117:) in /u01/grid/oracle/pr
oduct/11.2.0/grid_1/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2012-08-30 21:05:18.620
[ctssd(4472)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is s
hutdown by user
[client(6970)]CRS-10001:Waiting for ASM to shutdown.
[client(6994)]CRS-10001:Waiting for ASM to shutdown.
2012-08-30 21:05:37.009
[cssd(4409)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-08-30 21:11:01.468
[ohasd(4030)]CRS-2112:The OLR service started on node rac1.
2012-08-30 21:11:02.528
[ohasd(4030)]CRS-8017:location: /etc/oracle/lastgasp has 70 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-08-30 21:11:16.551
[ohasd(4030)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-08-30 21:11:22.349
[ohasd(4030)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-08-30 21:11:28.456
[cssd(4290)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 21:11:38.848
[cssd(4290)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 21:11:38.920
[cssd(4290)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 21:11:51.761
[cssd(4290)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-08-30 21:11:54.146
[ctssd(4374)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.
2012-08-30 21:11:54.960
[ctssd(4374)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-08-30 21:11:54.964
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:12:03.814
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
[client(4481)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

[client(4535)]CRS-10001:ACFS-9322: done.
2012-08-30 21:13:52.486
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:14:16.918
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:14:56.844
[crsd(5165)]CRS-1012:The OCR service started on node rac1.
2012-08-30 21:15:15.596
[crsd(5165)]CRS-1201:CRSD started on node rac1.
2012-08-30 21:15:31.649
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:16:04.145
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:16:20.384
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:16:36.475
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:16:52.975
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:17:09.392
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:17:41.647
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:18:06.403
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:18:55.914
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:20:12.107
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization ServicPuTTYe to be synchronous with the mean cluster time.
2012-08-30 21:20:39.017
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:21:50.474
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:22:27.553
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:23:00.369
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:24:15.849
[ctssd(4374)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:25:27.217
[cssd(4290)]CRS-1612:Network communication with node rac2 (2) missing for 50% of
timeout interval. Removal of this node from cluster in 44.980 seconds
2012-08-30 21:25:27.678

[cssd(4290)]CRS-1625:Node rac2, number 2, was manually shut down

2012-08-30 21:25:28.318
[cssd(4290)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-08-30 21:25:29.873
[ctssd(4374)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-08-30 21:25:49.229
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
e?
(5383)]CRS-5822:Agent '/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_r
oot' disconnected from server. Details at (:CRSAGF00117:) in /u01/grid/oracle/pr
oduct/11.2.0/grid_1/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2012-08-30 21:25:53.198
[cssd(4290)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-08-30 21:25:53.509
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4254)]CRS-5822:Agent '/u01
/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root' disconnected from server.
Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/a
gent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-08-30 21:25:53.509
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor(4232)]CRS-5822:Agent '/u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root' disconnected from ser
ver. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/ra
c1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-08-30 21:44:08.181
[ohasd(4233)]CRS-2112:The OLR service started on node rac1.
2012-08-30 21:44:09.002
[ohasd(4233)]CRS-8017:location: /etc/oracle/lastgasp has 72 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-08-30 21:44:19.949
[ohasd(4233)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
[client(4486)]CRS-10001:version mismatch:
kernel = 2.6.32-200.13.1.el5uek
driver = 2.6.18-274.el5
[client(4493)]CRS-10001:ACFS-9226: ADVM/ACFS drivers not correct for this OS - c
an not load.
2012-08-30 21:44:29.983
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
X
(4376)]CRS-5016:Process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" sp
awned by agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
X
" for action "start" failed: details at "(:CLSN00010:)" in "/u01/grid/oracle/pro
duct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-08-30 21:44:33.479
[cssd(4525)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 21:44:40.806
[cssd(4525)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 21:44:40.862
[cssd(4525)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 21:44:53.177
[cssd(4525)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-08-30 21:44:55.015
[ctssd(4587)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.
2012-08-30 21:44:56.686
[ctssd(4587)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.

2012-08-30 21:44:56.978
[ctssd(4587)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-08-30 21:44:57.532
[ctssd(4587)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-08-30 21:46:06.048
[crsd(4856)]CRS-1012:The OCR service started on node rac1.
2012-08-30 21:46:19.945
[crsd(4856)]CRS-1201:CRSD started on node rac1.
2012-08-30 23:09:04.799
[ohasd(4033)]CRS-2112:The OLR service started on node rac1.
2012-08-30 23:09:05.035
[ohasd(4033)]CRS-8011:reboot advisory message from host: rac1, component: ag2144
41, with time stamp: L-2012-08-30-21:51:25.572
[ohasd(4033)]CRS-8013:reboot advisory message text: clsnomon_status: need to reb
oot, unexpected failure 8 received from CSS
2012-08-30 23:09:05.248
[ohasd(4033)]CRS-8011:reboot advisory message from host: rac1, component: mo2144
41, with time stamp: L-2012-08-30-21:51:29.444
[ohasd(4033)]CRS-8013:reboot advisory message text: clsnomon_status: need to reb
oot, unexpected failure 8 received from CSS
2012-08-30 23:09:05.297
[ohasd(4033)]CRS-8017:location: /etc/oracle/lastgasp has 74 reboot advisory log
files, 2 were announced and 0 errors occurred
2012-08-30 23:09:13.087
[ohasd(4033)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
[client(4395)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4412)]CRS-10001:ACFS-9322: done.
2012-08-30 23:09:21.028
[cssd(4440)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-30 23:09:29.492
[cssd(4440)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-30 23:09:29.550
[cssd(4440)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-08-30 23:19:19.936
[cssd(4440)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-08-30 23:19:19.935
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4367)]CRS-5818:Aborted com
mand 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CR
SAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/agent/ohasd/oracs
sdagent_root/oracssdagent_root.log.
2012-08-30 23:19:20.890
[cssd(4440)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-08-30 23:19:25.336
[ohasd(4033)]CRS-2757:Command 'Start' timed out waiting for response from the re
source 'ora.cssd'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/
grid_1/log/rac1/ohasd/ohasd.log.
2012-08-30 23:19:25.450
[ohasd(4033)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.
2012-08-31 18:31:28.386
[ohasd(4070)]CRS-2112:The OLR service started on node rac1.
2012-08-31 18:31:28.999
[ohasd(4070)]CRS-8011:reboot advisory message from host: rac1, component: ag2144
41, with time stamp: L-2012-08-30-21:51:25.572
[ohasd(4070)]CRS-8013:reboot advisory message text: clsnomon_status: need to reb
oot, unexpected failure 8 received from CSS
2012-08-31 18:31:29.645
[ohasd(4070)]CRS-8011:reboot advisory message from host: rac1, component: mo2144

41, with time stamp: L-2012-08-30-21:51:29.444

[ohasd(4070)]CRS-8013:reboot advisory message text: clsnomon_status: need to reb
oot, unexpected failure 8 received from CSS
2012-08-31 18:31:29.756
[ohasd(4070)]CRS-8017:location: /etc/oracle/lastgasp has 76 reboot advisory log
files, 2 were announced and 0 errors occurred
2012-08-31 18:31:46.814
[ohasd(4070)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-08-31 18:31:50.188
[ohasd(4070)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-08-31 18:31:55.558
[cssd(4544)]CRS-1713:CSSD daemon is started in clustered mode
2012-08-31 18:32:04.567
[cssd(4544)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-08-31 18:32:04.713
[cssd(4544)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
[client(4719)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4744)]CRS-10001:ACFS-9322: done.
2012-08-31 18:36:36.684
[cssd(4544)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-08-31 18:36:36.735
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor(4399)]CRS-5822:Agent '/u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root' disconnected from ser
ver. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/ra
c1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-08-31 18:36:36.740
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4463)]CRS-5822:Agent '/u01
/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root' disconnected from server.
Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/a
gent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-09-03 00:48:11.359
[ohasd(4046)]CRS-2112:The OLR service started on node rac1.
2012-09-03 00:48:12.811
[ohasd(4046)]CRS-8017:location: /etc/oracle/lastgasp has 78 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 00:48:34.260
[ohasd(4046)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 00:48:40.206
[ohasd(4046)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-09-03 00:48:48.605
[cssd(4328)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 00:48:57.287
[cssd(4328)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 00:48:57.496
[cssd(4328)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 00:50:29.923
[cssd(4328)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2012-09-03 00:50:33.870
[cssd(4328)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 00:50:36.304
[ctssd(4463)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-09-03 00:50:36.782
[ctssd(4463)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
[client(4632)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

[client(4642)]CRS-10001:ACFS-9322: done.
2012-09-03 00:52:45.784
[crsd(4920)]CRS-1012:The OCR service started on node rac1.
2012-09-03 00:52:57.891
[crsd(4920)]CRS-1201:CRSD started on node rac1.
2012-09-03 00:53:24.334
[crsd(4920)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-09-03 00:53:24.484
[crsd(4920)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-09-03 01:01:28.478
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:01:30.435
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:04:28.777
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:04:28.986
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:07:31.152
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:07:31.556
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:08:28.930
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:08:28.979
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:09:31.572
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:09:31.820
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:10:28.817
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:10:28.846
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:16:32.821
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:16:34.461
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1

1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:18:32.746
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:18:32.773
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:20:34.426
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:20:34.489
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:22:46.752
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5818:Aborted command 'check for resource: ora.drivers.acfs 1 1' for r

esource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/produ
ct/11.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 01:22:46.771
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(4336)]CRS-5014:Agent "/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.b
in

" timed out starting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsloa

d" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/1
1.2.0/grid_1/log/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 01:41:39.645
[ohasd(4046)]CRS-2112:The OLR service started on node rac1.
2012-09-03 01:41:40.126
[ohasd(4046)]CRS-8017:location: /etc/oracle/lastgasp has 80 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 01:41:45.924
[ohasd(4046)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 01:41:50.913
[cssd(4312)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 01:42:00.909
[cssd(4312)]CRS-1707:Lease acquisition for node rac1 number 1 completed

2012-09-03 01:42:00.938
[cssd(4312)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 01:51:50.515
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(4274)]CRS-5818:Aborted com
mand 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CR
SAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/agent/ohasd/oracs
sdagent_root/oracssdagent_root.log.
2012-09-03 01:51:50.516
[cssd(4312)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 01:51:55.888
[ohasd(4046)]CRS-2757:Command 'Start' timed out waiting for response from the re
source 'ora.cssd'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/
grid_1/log/rac1/ohasd/ohasd.log.
2012-09-03 01:51:56.108
[ohasd(4046)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.
[client(4850)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4875)]CRS-10001:ACFS-9322: done.
2012-09-03 01:52:08.525
[cssd(4929)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 01:52:13.881
[cssd(4929)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 01:52:13.915
[cssd(4929)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 01:52:16.945
[ohasd(4046)]CRS-2765:Resource 'ora.diskmon' has failed on server 'rac1'.
2012-09-03 01:52:16.945
[ohasd(4046)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover
.
2012-09-03 01:54:52.742
[cssd(4929)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 01:55:01.926
[cssd(4929)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 01:55:55.257
[ohasd(5586)]CRS-2112:The OLR service started on node rac1.
2012-09-03 01:55:55.544
[ohasd(5586)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 01:55:59.142
[ohasd(5586)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
[client(5864)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(5885)]CRS-10001:ACFS-9322: done.
2012-09-03 01:56:05.443
[cssd(5985)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 01:56:17.439
[cssd(5985)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 01:56:17.490
[cssd(5985)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 01:56:29.158
[cssd(5985)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 01:56:31.079
[ctssd(6033)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.
2012-09-03 01:56:31.271
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:56:31.814
[ctssd(6033)]CRS-2401:The Cluster Time Synchronization Service started on host r

ac1.
2012-09-03 01:56:32.146
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:57:36.726
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:57:37.073
[crsd(6298)]CRS-1012:The OCR service started on node rac1.
2012-09-03 01:57:45.531
[crsd(6298)]CRS-1201:CRSD started on node rac1.
2012-09-03 01:57:52.934
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:58:25.263
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:59:05.715
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:59:21.931
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:59:37.960
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 01:59:54.038
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:00:18.418
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:00:42.485
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:01:30.817
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:01:47.024
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:02:11.515
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:02:43.815
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:02:59.979
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:03:32.316
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:03:48.680
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:04:13.688
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:04:29.781

Tim
Tim

Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim
Tim

[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:06:08.356
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:06:32.568
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:07:21.355
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:07:37.491
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:07:53.700
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:08:09.859
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:08:42.636
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:09:07.239
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:09:55.947
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:10:28.363
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:11:24.669
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:11:48.954
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 02:12:04.968
[ctssd(6033)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:22:24.169
[ohasd(4056)]CRS-2112:The OLR service started on node rac1.
2012-09-03 10:22:25.470
[ohasd(4056)]CRS-8017:location: /etc/oracle/lastgasp has 86 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 10:22:39.765
[ohasd(4056)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 10:22:44.571
[cssd(4305)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 10:22:54.950
[cssd(4305)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 10:22:55.057
[cssd(4305)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 10:23:27.917
[cssd(4305)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 10:23:29.709
[ctssd(4391)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.

2012-09-03 10:23:29.589
[ctssd(4391)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:23:30.200
[ctssd(4391)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-09-03 10:23:38.613
[ctssd(4391)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
[client(4550)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4560)]CRS-10001:ACFS-9322: done.
2012-09-03 10:25:51.990
[crsd(4867)]CRS-1012:The OCR service started on node rac1.
2012-09-03 10:26:00.953
[crsd(4867)]CRS-1201:CRSD started on node rac1.
2012-09-03 10:38:45.228
[ohasd(4032)]CRS-2112:The OLR service started on node rac1.
2012-09-03 10:38:45.854
[ohasd(4032)]CRS-8017:location: /etc/oracle/lastgasp has 88 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 10:38:51.058
[ohasd(4032)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 10:38:53.657
[ohasd(4032)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-09-03 10:38:55.655
[cssd(4298)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 10:39:05.697
[cssd(4298)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 10:39:05.730
[cssd(4298)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 10:42:25.004
[cssd(4298)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 10:42:26.689
[ctssd(4738)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.
2012-09-03 10:42:26.808
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:42:27.116
[ctssd(4738)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-09-03 10:42:43.851
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
[client(4951)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4959)]CRS-10001:ACFS-9322: done.
2012-09-03 10:43:40.211
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:43:56.720
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:44:20.769
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:44:45.186
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.

2012-09-03 10:44:55.688
[crsd(5295)]CRS-1012:The OCR service started on node rac1.
2012-09-03 10:45:01.266
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:45:09.415
[crsd(5295)]CRS-1201:CRSD started on node rac1.
2012-09-03 10:45:29.891
[crsd(5295)]CRS-2772:Server 'rac2' has been assigned to pool 'Generic'.
2012-09-03 10:45:29.910
[crsd(5295)]CRS-2772:Server 'rac2' has been assigned to pool 'ora.RAC'.
2012-09-03 10:45:33.708
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:45:49.737
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:46:13.893
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:46:30.057
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:47:18.300
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:47:34.369
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:48:06.643
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:48:32.394
[crsd(5295)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-09-03 10:48:32.395
[crsd(5295)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-09-03 10:49:29.254
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:51:08.848
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:53:39.965
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:55:18.682
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:55:34.782
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:56:44.935
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:58:18.677
[ctssd(4738)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 10:58:43.506
[crsd(5295)]CRS-2757:Command 'Start' timed out waiting for response from the res
ource 'ora.rac.db'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0

/grid_1/log/rac1/crsd/crsd.log.
2012-09-03 11:07:41.140
[ohasd(4575)]CRS-2112:The OLR service started on node rac1.
2012-09-03 11:07:41.721
[ohasd(4575)]CRS-8017:location: /etc/oracle/lastgasp has 90 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 11:07:47.896
[ohasd(4575)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 11:07:50.655
[ohasd(4575)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
[client(4963)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4976)]CRS-10001:ACFS-9322: done.
2012-09-03 11:07:53.179
[cssd(4858)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 11:08:03.156
[cssd(4858)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 11:08:03.206
[cssd(4858)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 11:08:17.785
[cssd(4858)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 11:08:19.731
[ctssd(5090)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.
2012-09-03 11:08:19.755
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:08:20.495
[ctssd(5090)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-09-03 11:08:45.132
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:09:35.287
[crsd(5378)]CRS-1012:The OCR service started on node rac1.
2012-09-03 11:09:44.641
[crsd(5378)]CRS-1201:CRSD started on node rac1.
2012-09-03 11:09:49.640
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:10:05.671
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:11:01.789
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:12:54.189
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:13:10.221
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:14:06.324
[ctssd(5090)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:16:00.944
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin

(5552)]CRS-5822:Agent '/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_r

oot' disconnected from server. Details at (:CRSAGF00117:) in /u01/grid/oracle/pr

oduct/11.2.0/grid_1/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2012-09-03 11:16:02.411
[ctssd(5090)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is s
hutdown by user
[client(6439)]CRS-10001:Waiting for ASM to shutdown.
[client(6471)]CRS-10001:Waiting for ASM to shutdown.
2012-09-03 11:16:25.460
[cssd(4858)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 11:22:21.221
[ohasd(4181)]CRS-2112:The OLR service started on node rac1.
2012-09-03 11:22:21.856
[ohasd(4181)]CRS-8017:location: /etc/oracle/lastgasp has 92 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 11:22:31.326
[ohasd(4181)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 11:22:34.168
[ohasd(4181)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
[client(4612)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4620)]CRS-10001:ACFS-9322: done.
2012-09-03 11:22:37.121
[cssd(4488)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 11:22:47.148
[cssd(4488)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 11:22:47.199
[cssd(4488)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 11:23:05.347
[cssd(4488)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 11:23:06.531
[ctssd(4728)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac1.
2012-09-03 11:23:07.276
[ctssd(4728)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-09-03 11:23:56.821
[crsd(4966)]CRS-1012:The OCR service started on node rac1.
2012-09-03 11:24:01.608
[crsd(4966)]CRS-1201:CRSD started on node rac1.
2012-09-03 11:24:18.508
[crsd(4966)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-09-03 11:24:18.548
[crsd(4966)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-09-03 11:25:06.331
[crsd(4966)]CRS-2772:Server 'rac2' has been assigned to pool 'Generic'.
2012-09-03 11:25:06.331
[crsd(4966)]CRS-2772:Server 'rac2' has been assigned to pool 'ora.RAC'.
2012-09-03 11:32:12.505
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
p (5142)]CRS-5822:Agen
t '/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_root' disconnected fr
om server. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/
log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2012-09-03 11:32:13.934
[ctssd(4728)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is s
hutdown by user
2012-09-03 11:32:19.118
[cssd(4488)]CRS-1625:Node rac2, number 2, was manually shut down
2012-09-03 11:32:19.407

[cssd(4488)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

2012-09-03 11:32:28.463
[cssd(4488)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 11:40:57.970
[ohasd(4336)]CRS-2112:The OLR service started on node rac1.
2012-09-03 11:41:02.103
[ohasd(4336)]CRS-8017:location: /etc/oracle/lastgasp has 94 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 11:41:23.236
[ohasd(4336)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 11:41:40.096
[cssd(4615)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 11:42:03.335
[cssd(4615)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 11:42:03.440
[cssd(4615)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
[client(4785)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4808)]CRS-10001:ACFS-9322: done.
2012-09-03 11:42:21.724
[cssd(4615)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2012-09-03 11:42:27.164
[ctssd(4826)]CRS-2407:The new Cluster Time Synchronization Service reference nod
e is host rac2.
2012-09-03 11:42:27.439
[ctssd(4826)]CRS-2401:The Cluster Time Synchronization Service started on host r
ac1.
2012-09-03 11:42:27.660
[ctssd(4826)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:42:28.715
[ctssd(4826)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:43:01.173
[ctssd(4826)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:45:57.245
[crsd(5202)]CRS-1012:The OCR service started on node rac1.
2012-09-03 11:47:06.361
[crsd(5202)]CRS-1201:CRSD started on node rac1.
2012-09-03 11:49:24.437
[crsd(5202)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2012-09-03 11:49:24.614
[crsd(5202)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.RAC'.
2012-09-03 11:50:48.168
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4496)]CRS-5818:Aborte
d command 'check for resource: ora.drivers.acfs 1 1' for resource 'ora.drivers.a
cfs'. Details at (:CRSAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/r
ac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-09-03 11:50:48.317
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(4496)]CRS-5014:Agent
"/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
" timed out st
arting process "/u01/grid/oracle/product/11.2.0/grid_1/bin/acfsload" for action
"check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0/grid_1/l
og/rac1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-09-03 11:51:23.296

[ctssd(4826)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:52:02.971
[ctssd(4826)]CRS-2408:The clock on host rac1 has been updated by the Cluster Tim
e Synchronization Service to be synchronous with the mean cluster time.
2012-09-03 11:54:45.842
[/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent.bin
(5518)]CRS-5822:Agen
t '/u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_root' disconnected fr
om server. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/
log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2012-09-03 11:54:54.993
[ctssd(4826)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is s
hutdown by user
[client(6105)]CRS-10001:Waiting for ASM to shutdown.
[client(6128)]CRS-10001:Waiting for ASM to shutdown.
[client(6134)]CRS-10001:Waiting for ASM to shutdown.
[client(6155)]CRS-10001:Waiting for ASM to shutdown.
2012-09-03 11:56:21.004
[cssd(4615)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 11:59:18.683
[ohasd(6404)]CRS-2112:The OLR service started on node rac1.
2012-09-03 11:59:19.711
[ohasd(6404)]CRS-8017:location: /etc/oracle/lastgasp has 96 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 11:59:31.507
[ohasd(6404)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 11:59:40.844
[ohasd(6404)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-09-03 11:59:51.402
[cssd(6679)]CRS-1713:CSSD daemon is started in clustered mode
2012-09-03 12:00:00.630
[cssd(6679)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 12:00:00.703
[cssd(6679)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 12:09:49.559
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(6628)]CRS-5818:Aborted com
mand 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CR
SAGF00113:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/agent/ohasd/oracs
sdagent_root/oracssdagent_root.log.
2012-09-03 12:09:49.568
[cssd(6679)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 12:09:54.931
[ohasd(6404)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.
2012-09-03 12:09:55.118
[ohasd(6404)]CRS-2757:Command 'Start' timed out waiting for response from the re
source 'ora.cssd'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/
grid_1/log/rac1/ohasd/ohasd.log.
2012-09-03 12:10:03.074
[ohasd(6404)]CRS-2765:Resource 'ora.diskmon' has failed on server 'rac1'.
2012-09-03 12:10:11.846
[cssd(7196)]CRS-1713:CSSD daemon is started in clustered mode
[client(7329)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(7341)]CRS-10001:ACFS-9322: done.
2012-09-03 12:10:19.356
[cssd(7196)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 12:10:19.437
[cssd(7196)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det

ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2012-09-03 12:15:45.670
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor(7130)]CRS-5822:Agent '/u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root' disconnected from ser
ver. Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/ra
c1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-09-03 12:15:45.684
[/u01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent(7172)]CRS-5822:Agent '/u01
/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root' disconnected from server.
Details at (:CRSAGF00117:) in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/a
gent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-09-03 12:32:10.178
[ohasd(4768)]CRS-2112:The OLR service started on node rac1.
2012-09-03 12:32:12.252
[ohasd(4768)]CRS-8017:location: /etc/oracle/lastgasp has 100 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 12:32:30.906
[ohasd(4768)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 12:32:39.206
[ohasd(4768)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-09-03 12:32:48.276
[cssd(5041)]CRS-1713:CSSD daemon is started in exclusive mode
2012-09-03 12:33:23.841
[cssd(5041)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 12:33:23.922
[cssd(5041)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
[cssd(5041)]CRS-1636:The CSS daemon was started in exclusive mode but found an a
ctive CSS daemon on node rac2 and is terminating; details at (:CSSNM00006:) in /
u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log
2012-09-03 12:33:34.637
[cssd(5041)]CRS-1603:CSSD on node rac1 shutdown by user.
2012-09-03 12:33:40.992
[ohasd(4768)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.
2012-09-03 12:35:11.356
[ohasd(5249)]CRS-2112:The OLR service started on node rac1.
2012-09-03 12:35:11.456
[ohasd(5249)]CRS-8017:location: /etc/oracle/lastgasp has 102 reboot advisory log
files, 0 were announced and 0 errors occurred
2012-09-03 12:35:15.617
[ohasd(5249)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2012-09-03 12:35:18.278
[ohasd(5249)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD da
emon is not running).
2012-09-03 12:35:21.388
[cssd(5516)]CRS-1713:CSSD daemon is started in exclusive mode
2012-09-03 12:35:30.818
[cssd(5516)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2012-09-03 12:35:30.852
[cssd(5516)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DATA1; det
ails in /u01/grid/oracle/product/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
[root@rac1 rac1]# PuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPu
TTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPu
TTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPu
TTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPu
TTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPu
TTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPu
TTYPuTTY
-bash: PuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuT

TYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuT
TYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuT
TYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuT
TYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuT
TYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTYPuTTY: c
ommand not found
[root@rac1 rac1]# cd ohasd/
[root@rac1 ohasd]# tail ohasd.log
2012-09-03 12:37:12.750: [UiServer][2764983184] S(0xad408c20): set Properties (
root,0xb445e40)
2012-09-03 12:37:12.751: [UiServer][2764983184] S(0xb0ab0a0): Accepted client co
nnection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=51)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2012-09-03 12:37:12.770: [UiServer][2767084432] processMessage called
2012-09-03 12:37:12.770: [UiServer][2767084432] Sending message to PE. ctx= 0xad
44cb50
2012-09-03 12:37:12.770: [UiServer][2767084432] Sending command to PE: 4
2012-09-03 12:37:12.770: [ CRSPE][2771286928] Processing PE command id=104. De
scription: [Stat Resource : 0xa4706a78]
2012-09-03 12:37:12.776: [ CRSPE][2771286928] PE Command [ Stat Resource : 0xa
4706a78 ] has completed
2012-09-03 12:37:12.776: [ CRSPE][2771286928] UI Command [Stat Resource : 0xa4
706a78] is replying to sender.
2012-09-03 12:37:12.780: [UiServer][2767084432] Done for ctx=0xad44cb50
2012-09-03 12:37:12.790: [UiServer][2764983184] Closed: remote end failed/disc.
[root@rac1 ohasd]# tail -200f ohasd.log
2012-09-03 12:35:19.383: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 451
2012-09-03 12:35:19.383: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 451
2012-09-03 12:35:19.383: [ CRSPE][2771286928] RI [ora.cssdmonitor 1 1] new ext
ernal state [ONLINE] old value: [OFFLINE] on rac1 label = []
2012-09-03 12:35:19.383: [ CRSPE][2771286928] CRS-2676: Start of 'ora.cssdmoni
tor' on 'rac1' succeeded
2012-09-03 12:35:19.385: [
0
2012-09-03 12:35:19.385: [
ra.cssd' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 48

2012-09-03 12:35:19.386: [
2
2012-09-03 12:35:19.386: [
ra.diskmon' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 48

CRSPE][2771286928] CRS-2672: Attempting to start 'o

CRSPE][2771286928] CRS-2679: Attempting to clean 'o

2012-09-03 12:35:19.386: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:35:19.386: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_START[ora.cssd 1 1] ID 4098:480
2012-09-03 12:35:19.386: [
AGFW][2781793168] Creating the resource: ora.cssd
1 1
2012-09-03 12:35:19.387: [
AGFW][2781793168] Initializing the resource ora.cs

sd 1 1 for type ora.cssd.type

2012-09-03 12:35:19.387: [
AGFW][2781793168] SR: acl = owner:root:rw-,pgrp:oi
nstall:rw-,other::r--,user:oracle:r-x
2012-09-03 12:35:19.387: [
AGFW][2781793168] Starting the agent: /u01/grid/or
acle/product/11.2.0/grid_1/bin/cssdagent with user id: root and incarnation:2
2012-09-03 12:35:19.409: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2672: Attempting to start 'ora.cssd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:35:19.411: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2679: Attempting to clean 'ora.diskmon' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:35:19.438: [
AGFW][2781793168] Starting the HB [Interval = 300
00, misscount = 10kill allowed=0] for agent: /u01/grid/oracle/product/11.2.0/gri
d_1/bin/cssdagent_root
2012-09-03 12:35:19.439: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_CLEAN[ora.diskmon 1 1] ID 4100:482
2012-09-03 12:35:19.439: [
AGFW][2781793168] Starting the agent: /u01/grid/or
acle/product/11.2.0/grid_1/bin/orarootagent with user id: root and incarnation:2
2012-09-03 12:35:19.492: [
AGFW][2781793168] Starting the HB [Interval = 300
00, misscount = 6kill allowed=1] for agent: /u01/grid/oracle/product/11.2.0/grid
_1/bin/orarootagent_root
2012-09-03 12:35:19.496: [
AGFW][2781793168] Could not forward message [RESOU
RCE_CLEAN[ora.diskmon 1 1] ID 4100:482] to agent. /u01/grid/oracle/product/11.2.
0/grid_1/bin/orarootagent_root is not running
2012-09-03 12:35:19.496: [
AGFW][2781793168] Starting of the agent: /u01/grid
/oracle/product/11.2.0/grid_1/bin/orarootagent with user id root is already in p
rogress.
2012-09-03 12:35:19.761: [CLSFRAME][2788096912] New IPC Member:{Relative|Node:0|
Process:7|Type:3}:AGENT
2012-09-03 12:35:19.761: [CLSFRAME][2788096912] New process connected to us ID:{
Relative|Node:0|Process:7|Type:3} Info:AGENT
2012-09-03 12:35:19.789: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:35:19.789: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/orarootagent_root with pid:5475 connected to server.
2012-09-03 12:35:19.789: [
AGFW][2781793168] Could not find the resource type
[ app.appvip.type ]
2012-09-03 12:35:19.789: [
AGFW][2781793168] Could not find the resource type
[ ora.acfs.type ]
2012-09-03 12:35:19.789: [
AGFW][2781793168] Could not find the resource type
[ ora.cluster_vip.type ]
2012-09-03 12:35:19.790: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.crs.type] ID 8196:498 to the agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.790: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.ctss.type] ID 8196:500 to the agent /u01/grid/oracle/product/

11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.791: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.diskmon.type] ID 8196:502 to the agent /u01/grid/oracle/produ
ct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.792: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:504 to the agent /u01/grid/oracle/
product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.792: [
AGFW][2781793168] Could not find the resource type
[ ora.gns.type ]
2012-09-03 12:35:19.792: [
AGFW][2781793168] Could not find the resource type
[ ora.network.type ]
2012-09-03 12:35:19.792: [
AGFW][2781793168] Could not find the resource type
[ ora.registry.acfs.type ]
2012-09-03 12:35:19.792: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:506 to the agent /u01/grid/oracle/produ
ct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.792: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_CLEAN[ora.diskmon 1 1] ID 4100:482 to the agent /u01/grid/ora
cle/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.793: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:35:19.835: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.crs.type] ID 8196:498 from the agent /u01/grid/oracle/product
/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.841: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.ctss.type] ID 8196:500 from the agent /u01/grid/oracle/produc
t/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.847: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.diskmon.type] ID 8196:502 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.853: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:504 from the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:19.856: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:506 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:20.119: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_CLEAN[ora.diskmon 1 1] ID 4100:507 from the agent /u01/grid/oracle/p
roduct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:20.119: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_CLEAN[ora.diskmon 1 1] ID 4100:482
2012-09-03 12:35:20.119: [ CRSPE][2771286928] Received reply to action [Clean]
message ID: 482
2012-09-03 12:35:20.121: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_CLEAN[ora.diskmon 1 1] ID 4100:507 from the agent /u01/grid/oracle/p
roduct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:20.121: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_CLEAN[ora.diskmon 1 1] ID 4100:482
2012-09-03 12:35:20.121: [ CRSPE][2771286928] Received reply to action [Clean]
message ID: 482
2012-09-03 12:35:20.121: [ CRSPE][2771286928] Cleared dirty state for [ora.dis
kmon 1 1]
2012-09-03 12:35:20.122: [ CRSPE][2771286928] CRS-2681: Clean of 'ora.diskmon'
on 'rac1' succeeded
2012-09-03 12:35:20.124: [
3
2012-09-03 12:35:20.124: [
ra.diskmon' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 52

CRSPE][2771286928] CRS-2672: Attempting to start 'o

2012-09-03 12:35:20.124: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2681: Clean of 'ora.diskmon' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:35:20.125: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_START[ora.diskmon 1 1] ID 4098:523
2012-09-03 12:35:20.125: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_START[ora.diskmon 1 1] ID 4098:523 to the agent /u01/grid/ora
cle/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:20.126: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2672: Attempting to start 'ora.diskmon' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:35:20.131: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:42
2012-09-03 12:35:20.131: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:20.131: [
AGFW][2781793168] Suicide request rejected for the
agent: /u01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:20.131: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:42
2012-09-03 12:35:20.203: [CLSFRAME][2788096912] New IPC Member:{Relative|Node:0|
Process:8|Type:3}:AGENT
2012-09-03 12:35:20.203: [CLSFRAME][2788096912] New process connected to us ID:{
Relative|Node:0|Process:8|Type:3} Info:AGENT
2012-09-03 12:35:20.213: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:35:20.213: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdagent_root with pid:5473 connected to server.
2012-09-03 12:35:20.214: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.cssd.type] ID 8196:542 to the agent /u01/grid/oracle/product/
11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:35:20.214: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESOURCE_ADD[ora.cssd 1 1] ID 4356:544 to the agent /u01/grid/oracle/product/
11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:35:20.214: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_START[ora.cssd 1 1] ID 4098:480 to the agent /u01/grid/oracle
/product/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:35:20.215: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:35:20.284: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.cssd.type] ID 8196:542 from the agent /u01/grid/oracle/produc
t/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:35:20.304: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_ADD[ora.cssd 1 1] ID 4356:544 from the agent /u01/grid/oracle/produc
t/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:35:26.176: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_START[ora.diskmon 1 1] ID 4098:524 from the agent /u01/grid/oracle/p

roduct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:26.176: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_START[ora.diskmon 1 1] ID 4098:523
2012-09-03 12:35:26.176: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 523
2012-09-03 12:35:26.178: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_START[ora.diskmon 1 1] ID 4098:524 from the agent /u01/grid/oracle/p
roduct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:35:26.178: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_START[ora.diskmon 1 1] ID 4098:523
2012-09-03 12:35:26.178: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 523
2012-09-03 12:35:26.178: [ CRSPE][2771286928] RI [ora.diskmon 1 1] new externa
l state [ONLINE] old value: [OFFLINE] on rac1 label = []
2012-09-03 12:35:26.179: [ CRSPE][2771286928] CRS-2676: Start of 'ora.diskmon'
on 'rac1' succeeded
2012-09-03 12:35:26.179: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:35:38.310: [ CRSCCL][2811444112]Obtained GPnP Profile.
2012-09-03 12:35:38.310: [ CRSCCL][2811444112]USING CLSC ============
2012-09-03 12:35:38.313: [ default][2811444112]clsu_get_private_ip_addr: Calling
clsu_get_private_ip_addresses to get first private ip
2012-09-03 12:35:38.313: [ default][2811444112]Check namebufs
2012-09-03 12:35:38.313: [ default][2811444112]Finished checking namebufs
2012-09-03 12:35:38.314: [
GPnP][2811444112]clsgpnp_Init: [at clsgpnp0.c:404]
gpnp tracelevel 3, component tracelevel 0
2012-09-03 12:35:38.314: [
GPnP][2811444112]clsgpnp_Init: [at clsgpnp0.c:534]
'/u01/grid/oracle/product/11.2.0/grid_1' in effect as GPnP home base.
2012-09-03 12:35:38.322: [
GPnP][2811444112]clsgpnp_InitCKProviders: [at clsg
pnp0.c:3866] Init gpnp local security key providers (2) fatal if both fail
2012-09-03 12:35:38.322: [
GPnP][2811444112]clsgpnp_InitCKProviders: [at clsg
pnp0.c:3869] Init gpnp local security key proveders 1 of 2: file wallet (LSKP-FS
W)
2012-09-03 12:35:38.323: [
GPnP][2811444112]clsgpnpkwf_initwfloc: [at clsgpnp
kwf.c:398] Using FS Wallet Location : /u01/grid/oracle/product/11.2.0/grid_1/gpn
p/rac1/wallets/peer/
2012-09-03 12:35:38.323: [
GPnP][2811444112]clsgpnp_InitCKProviders: [at clsg
pnp0.c:3891] Init gpnp local security key provider 1 of 2: file wallet (LSKP-FSW
) OK
2012-09-03 12:35:38.323: [
GPnP][2811444112]clsgpnp_InitCKProviders: [at clsg
pnp0.c:3897] Init gpnp local security key proveders 2 of 2: OLR wallet (LSKP-CLS
W-OLR)
[ CLWAL][2811444112]clsw_Initialize: OLR initlevel [70000]
2012-09-03 12:35:38.323: [
GPnP][2811444112]clsgpnp_InitCKProviders: [at clsg
pnp0.c:3919] Init gpnp local security key provider 2 of 2: OLR wallet (LSKP-CLSW
-OLR) OK
2012-09-03 12:35:38.323: [
GPnP][2811444112]clsgpnp_getCK: [at clsgpnp0.c:195
0] <Get gpnp security keys (wallet) for id:1,typ;7. (2 providers - fatal if all
fail)
2012-09-03 12:35:38.323: [
GPnP][2811444112]clsgpnpkwf_getWalletPath: [at cls

gpnpkwf.c:498] req_id=1 ck_prov_id=1 wallet path: /u01/grid/oracle/product/11.2.

0/grid_1/gpnp/rac1/wallets/peer/
2012-09-03 12:35:38.334: [
GPnP][2811444112]clsgpnpwu_walletfopen: [at clsgpn
pwu.c:494] Opened SSO wallet: '/u01/grid/oracle/product/11.2.0/grid_1/gpnp/rac1/
wallets/peer/cwallet.sso'
2012-09-03 12:35:38.334: [
GPnP][2811444112]clsgpnp_getCK: [at clsgpnp0.c:196
5] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))
2012-09-03 12:35:38.334: [
GPnP][2811444112]clsgpnp_getCK: [at clsgpnp0.c:198
2] Got gpnp security keys (wallet).>
2012-09-03 12:35:38.337: [
GPnP][2811444112]clsgpnp_getCK: [at clsgpnp0.c:195
0] <Get gpnp security keys (wallet) for id:1,typ;4. (2 providers - fatal if all
fail)
2012-09-03 12:35:38.337: [
GPnP][2811444112]clsgpnpkwf_getWalletPath: [at cls
gpnpkwf.c:498] req_id=1 ck_prov_id=1 wallet path: /u01/grid/oracle/product/11.2.
0/grid_1/gpnp/rac1/wallets/peer/
2012-09-03 12:35:38.346: [
GPnP][2811444112]clsgpnpwu_walletfopen: [at clsgpn
pwu.c:494] Opened SSO wallet: '/u01/grid/oracle/product/11.2.0/grid_1/gpnp/rac1/
wallets/peer/cwallet.sso'
2012-09-03 12:35:38.346: [
GPnP][2811444112]clsgpnp_getCK: [at clsgpnp0.c:196
5] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))
2012-09-03 12:35:38.346: [
GPnP][2811444112]clsgpnp_getCK: [at clsgpnp0.c:198
2] Got gpnp security keys (wallet).>
2012-09-03 12:35:38.346: [
GPnP][2811444112]clsgpnp_Init: [at clsgpnp0.c:837]
GPnP client pid=5249, tl=3, f=3
2012-09-03 12:35:38.355: [GIPCXCPT][2811444112] gipcShutdownF: skipping shutdown
, count 3, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2012-09-03 12:35:38.363: [GIPCXCPT][2811444112] gipcShutdownF: skipping shutdown
, count 2, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
2012-09-03 12:35:38.366: [ CRSCCL][2811444112]Listening endpoint created sucess
fully @ (ADDRESS=(PROTOCOL=tcp)(DEV=47)(HOST=10.10.10.1)(PORT=58864)).con = 0xb1
0eee0
2012-09-03 12:35:38.367: [ CRSCCL][2811444112]clsCclRdGetMembers: using discove
ry gather delay of 1 sec
2012-09-03 12:35:40.504: [ CRSCCL][2811444112][0 services found] [0 bids found]
2012-09-03 12:35:40.504: [ CRSCCL][2811444112]RdRefreshMembers called
2012-09-03 12:35:40.506: [ CRSCCL][2811444112]Trying member ID = 15
2012-09-03 12:35:40.506: [ CRSCCL][2811444112]Bidding: mdns:service:CLSFRAME._t
cp://10.10.10.1:8888/host=10.10.10.1,clUid=1c1eaa13e24ecf72ff651f5a8fef6ace,mNam
e=rac1,mIdBid=15,mIncn=1346655938/15_bid c:1c1eaa13e24ecf72ff651f5a8fef6ace
2012-09-03 12:35:41.353: [ CRSCCL][2811444112]clsCclRegRdCb: state=REGISTER typ
estr= . Url=mdns:service:CLSFRAME._tcp://10.10.10.1:8888/host=10.10.10.1,clUid=1
c1eaa13e24ecf72ff651f5a8fef6ace,mName=rac1,mIdBid=15,mIncn=1346655938/15_bid c:1
c1eaa13e24ecf72ff651f5a8fef6ace
2012-09-03 12:35:50.651: [ CRSCCL][2811444112]clsCclRdGetMembers: using discove
ry gather delay of 2 sec
2012-09-03 12:35:54.863: [ CRSCCL][2811444112][0 services found] [1 bids found]
2012-09-03 12:35:54.863: [ CRSCCL][2811444112]RdRefreshMembers called
2012-09-03 12:35:54.865: [ CRSCCL][2811444112]Local Member ID selected: 15
2012-09-03 12:35:54.865: [ CRSCCL][2811444112]memberdata = 1:OHASD:rac1
2012-09-03 12:35:54.865: [ CRSCCL][2811444112]Registering: mdns:service:CLSFRAM
E._tcp://10.10.10.1:8888/host=10.10.10.1,mName=rac1,clUid=1c1eaa13e24ecf72ff651f
5a8fef6ace,mId=15,mIncn=1346655938,mData=1:OHASD:rac1/CLSFRAME_15_1346655938 c:1
c1eaa13e24ecf72ff651f5a8fef6ace
2012-09-03 12:35:55.921: [ CRSCCL][2811444112]clsCclRegRdCb: state=REGISTER typ
estr= . Url=mdns:service:CLSFRAME._tcp://10.10.10.1:8888/host=10.10.10.1,mName=r
ac1,clUid=1c1eaa13e24ecf72ff651f5a8fef6ace,mId=15,mIncn=1346655938,mData=1:OHASD
:rac1,mPort=58864/CLSFRAME_15_1346655938 c:1c1eaa13e24ecf72ff651f5a8fef6ace
2012-09-03 12:36:04.933: [ CRSCCL][2811444112]Initial membership:
2012-09-03 12:36:04.933: [ CRSCCL][2811444112]Dumping member data -----------------

2012-09-03 12:36:04.933: [ CRSCCL][2811444112]Member (15, 1346655938) on node

port=.
2012-09-03 12:36:04.933: [ CRSCCL][2811444112]Done -----------------2012-09-03 12:36:04.933: [ CRSCCL][2811444112]My info in memdata: 15, 134665593
8
2012-09-03 12:36:04.933: [ CRSCCL][2811444112]clsCclRdWatchPeers called
2012-09-03 12:36:04.934: [ CRSCCL][2731703184]cclCommunicationHandler started.
2012-09-03 12:37:12.750: [UiServer][2764983184] S(0xad408c20): set Properties (
root,0xb445e40)
2012-09-03 12:37:12.751: [UiServer][2764983184] S(0xb0ab0a0): Accepted client co
nnection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=51)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2012-09-03 12:37:12.770: [UiServer][2767084432] processMessage called
2012-09-03 12:37:12.770: [UiServer][2767084432] Sending message to PE. ctx= 0xad
44cb50
2012-09-03 12:37:12.770: [UiServer][2767084432] Sending command to PE: 4
2012-09-03 12:37:12.770: [ CRSPE][2771286928] Processing PE command id=104. De
scription: [Stat Resource : 0xa4706a78]
2012-09-03 12:37:12.776: [ CRSPE][2771286928] PE Command [ Stat Resource : 0xa
4706a78 ] has completed
2012-09-03 12:37:12.776: [ CRSPE][2771286928] UI Command [Stat Resource : 0xa4
706a78] is replying to sender.
2012-09-03 12:37:12.780: [UiServer][2767084432] Done for ctx=0xad44cb50
2012-09-03 12:37:12.790: [UiServer][2764983184] Closed: remote end failed/disc.
2012-09-03 12:45:25.501: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_START[ora.cssd 1 1] ID 4098:545 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:25.501: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:480
2012-09-03 12:45:25.501: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_START[ora.cssd 1 1] ID 4098:545 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:25.501: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:480
2012-09-03 12:45:25.501: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:34
2012-09-03 12:45:25.501: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:25.501: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-09-03 12:45:25.502: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 480
2012-09-03 12:45:25.502: [ CRSPE][2771286928]Created alert : (:CRSPE00111:) :
Start action timed out!
2012-09-03 12:45:25.502: [ CRSPE][2771286928] Start action failed with error c
ode: 3
2012-09-03 12:45:25.502: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 480
2012-09-03 12:45:25.502: [ CRSPE][2771286928] CRS-2674: Start of 'ora.cssd' on
'rac1' failed
2012-09-03 12:45:25.503: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2674: Start of 'ora.cssd' on 'rac1' failed]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]

]
2012-09-03 12:45:25.503: [ CRSPE][2771286928] Sending message to agfw: id = 87
3
2012-09-03 12:45:25.503: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:873
2012-09-03 12:45:25.503: [
AGFW][2781793168] Agent: /u01/grid/oracle/product/
11.2.0/grid_1/bin/cssdagent with user id root is being stopped. Will be restarte
d when agent stops completly
2012-09-03 12:45:25.503: [
AGFW][2781793168] Could not forward message [RESOU
RCE_CLEAN[ora.cssd 1 1] ID 4100:873] to agent. /u01/grid/oracle/product/11.2.0/g
rid_1/bin/cssdagent_root is not running
2012-09-03 12:45:25.503: [
AGFW][2781793168] Can not start Agent: /u01/grid/o
racle/product/11.2.0/grid_1/bin/cssdagent with user id root is waiting to be res
tarted.
2012-09-03 12:45:25.503: [ CRSPE][2771286928] CRS-2679: Attempting to clean 'o
ra.cssd' on 'rac1'
2012-09-03 12:45:25.504: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2679: Attempting to clean 'ora.cssd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:25.509: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STATUS[Proxy] ID 20481:211
2012-09-03 12:45:25.510: [
AGFW][2781793168] Received state change for ora.cs
sdmonitor 1 1 [old state = ONLINE, new state = OFFLINE]
2012-09-03 12:45:25.510: [
AGFW][2781793168] Agfw Proxy Server sending messag
e to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:
{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:879]
2012-09-03 12:45:25.510: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: RESOURCE_STATUS[Proxy] ID 20481:211
2012-09-03 12:45:25.510: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:213
2012-09-03 12:45:25.510: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:25.510: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:213
2012-09-03 12:45:25.510: [ CRSPE][2771286928] State change received from rac1
for ora.cssdmonitor 1 1
2012-09-03 12:45:25.510: [ CRSPE][2771286928] Processing PE command id=105. De
scription: [Resource State Change (ora.cssdmonitor 1 1) : 0xa4706780]
2012-09-03 12:45:25.510: [ CRSPE][2771286928] RI [ora.cssdmonitor 1 1] new ext
ernal state [OFFLINE] old value: [ONLINE] on rac1 label = []
2012-09-03 12:45:25.510: [ CRSPE][2771286928] Resource Resource Instance ID[or
a.cssdmonitor 1 1]. Values:
STATE=OFFLINE
TARGET=ONLINE
LAST_SERVER=rac1
CURRENT_RCOUNT=0
LAST_RESTART=1346655919
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=1
STATE_CHANGE_VERS=1

LAST_FAULT=0
DEGREE_ID=1
ID=ora.cssdmonitor 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:START of [ora.cssd 1 1] on [rac1] : 0xb0b50b8
ReadLocks:|STATE INITED||INITIAL CHECK DONE| has failed!
2012-09-03 12:45:25.511: [ CRSPE][2771286928] Processing unplanned state chang
e for [ora.cssdmonitor 1 1]
2012-09-03 12:45:25.511: [ CRSPE][2771286928] ora.cssdmonitor 1 1: uptime exce
eds uptime threshold , resetting restart count
2012-09-03 12:45:25.511: [ CRSPE][2771286928] Local Recovery picked for [ora.c
ssdmonitor 1 1]
2012-09-03 12:45:25.513: [ CRSPE][2771286928] Sending message to agfw: id = 88
1
2012-09-03 12:45:25.513: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_START[ora.cssdmonitor 1 1] ID 4098:881
2012-09-03 12:45:25.513: [
AGFW][2781793168] Agent: /u01/grid/oracle/product/
11.2.0/grid_1/bin/cssdmonitor with user id root is being stopped. Will be restar
ted when agent stops completly
2012-09-03 12:45:25.513: [
AGFW][2781793168] Could not forward message [RESOU
RCE_START[ora.cssdmonitor 1 1] ID 4098:881] to agent. /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdmonitor_root is not running
2012-09-03 12:45:25.513: [
AGFW][2781793168] Can not start Agent: /u01/grid/o
racle/product/11.2.0/grid_1/bin/cssdmonitor with user id root is waiting to be r
estarted.
2012-09-03 12:45:25.513: [ CRSPE][2771286928] CRS-2672: Attempting to start 'o
ra.cssdmonitor' on 'rac1'
2012-09-03 12:45:35.718: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:35.718: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:35.718: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 6
2012-09-03 12:45:35.718: [ CRSCOMM][2788096912] IPC listener connection to membe
r 6 has been removed
2012-09-03 12:45:35.718: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:6|Type:3}
2012-09-03 12:45:35.718: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:6|Type:3}
2012-09-03 12:45:35.719: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:35.719: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:35.719: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/cssdmonitor_root disconnected.
2012-09-03 12:45:35.719: [
AGFW][2781793168] Restarting the agent /u01/grid/o
racle/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:35.719: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdmonitor_root[5449] stopped!
2012-09-03 12:45:35.719: [ CRSCOMM][2781793168] removeConnection: Member 6 does
not exist.
2012-09-03 12:45:35.719: [
AGFW][2781793168] Starting the agent: /u01/grid/or
acle/product/11.2.0/grid_1/bin/cssdmonitor with user id: root and incarnation:3
2012-09-03 12:45:35.754: [
AGFW][2781793168] Starting the HB [Interval = 300
00, misscount = 10kill allowed=0] for agent: /u01/grid/oracle/product/11.2.0/gri
d_1/bin/cssdmonitor_root
2012-09-03 12:45:36.438: [CLSFRAME][2788096912] New IPC Member:{Relative|Node:0|
Process:9|Type:3}:AGENT
2012-09-03 12:45:36.438: [CLSFRAME][2788096912] New process connected to us ID:{

Relative|Node:0|Process:9|Type:3} Info:AGENT
2012-09-03 12:45:36.764: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:45:36.764: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdmonitor_root with pid:5938 connected to server.
2012-09-03 12:45:36.765: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:911 to the agent /u01/grid/oracle/p
roduct/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.765: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:913 to the agent /u01/grid/oracle/p
roduct/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.765: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_START[ora.cssdmonitor 1 1] ID 4098:881 to the agent /u01/grid
/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.766: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:45:36.790: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:911 from the agent /u01/grid/oracle
/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.798: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:913 from the agent /u01/grid/oracle
/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.898: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_START[ora.cssdmonitor 1 1] ID 4098:914 from the agent /u01/grid/orac
le/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.899: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_START[ora.cssdmonitor 1 1] ID 4098:881
2012-09-03 12:45:36.899: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_START[ora.cssdmonitor 1 1] ID 4098:914 from the agent /u01/grid/orac
le/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:36.899: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_START[ora.cssdmonitor 1 1] ID 4098:881
2012-09-03 12:45:36.899: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 881
2012-09-03 12:45:36.899: [ CRSPE][2771286928] Received reply to action [Start]
message ID: 881
2012-09-03 12:45:36.899: [ CRSPE][2771286928] RI [ora.cssdmonitor 1 1] new ext
ernal state [ONLINE] old value: [OFFLINE] on rac1 label = []
2012-09-03 12:45:36.900: [ CRSPE][2771286928] CRS-2676: Start of 'ora.cssdmoni
tor' on 'rac1' succeeded
2012-09-03 12:45:36.900: [ CRSPE][2771286928] PE Command [ Resource State Chan
ge (ora.cssdmonitor 1 1) : 0xa4706780 ] has completed
2012-09-03 12:45:36.900: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: CMD_COMPLETED[Proxy] ID 20482:924
2012-09-03 12:45:36.900: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: CMD_COMPLETED[Proxy] ID 20482:924
2012-09-03 12:45:36.900: [
AGFW][2781793168] Agfw received reply from PE for
resource state change for ora.cssdmonitor 1 1
2012-09-03 12:45:40.687: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:40.687: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:40.687: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 8
2012-09-03 12:45:40.687: [ CRSCOMM][2788096912] IPC listener connection to membe
r 8 has been removed
2012-09-03 12:45:40.687: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:8|Type:3}
2012-09-03 12:45:40.687: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:8|Type:3}

2012-09-03 12:45:40.688: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:40.688: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:40.688: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/cssdagent_root disconnected.
2012-09-03 12:45:40.688: [
AGFW][2781793168] Restarting the agent /u01/grid/o
racle/product/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:40.688: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdagent_root[5473] stopped!
2012-09-03 12:45:40.688: [ CRSCOMM][2781793168] removeConnection: Member 8 does
not exist.
2012-09-03 12:45:40.688: [
AGFW][2781793168] Starting the agent: /u01/grid/or
acle/product/11.2.0/grid_1/bin/cssdagent with user id: root and incarnation:3
2012-09-03 12:45:40.726: [
AGFW][2781793168] Starting the HB [Interval = 300
00, misscount = 10kill allowed=0] for agent: /u01/grid/oracle/product/11.2.0/gri
d_1/bin/cssdagent_root
2012-09-03 12:45:41.286: [CLSFRAME][2788096912] New IPC Member:{Relative|Node:0|
Process:10|Type:3}:AGENT
2012-09-03 12:45:41.286: [CLSFRAME][2788096912] New process connected to us ID:{
Relative|Node:0|Process:10|Type:3} Info:AGENT
2012-09-03 12:45:41.301: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:45:41.301: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdagent_root with pid:5958 connected to server.
2012-09-03 12:45:41.302: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESTYPE_ADD[ora.cssd.type] ID 8196:954 to the agent /u01/grid/oracle/product/
11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.302: [
AGFW][2781793168] Agfw Proxy Server sending messag
e: RESOURCE_ADD[ora.cssd 1 1] ID 4356:956 to the agent /u01/grid/oracle/product/
11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.303: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:873 to the agent /u01/grid/oracle
/product/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.303: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-09-03 12:45:41.334: [
AGFW][2781793168] Received the reply to the messag
e: RESTYPE_ADD[ora.cssd.type] ID 8196:954 from the agent /u01/grid/oracle/produc
t/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.335: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_ADD[ora.cssd 1 1] ID 4356:956 from the agent /u01/grid/oracle/produc
t/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.342: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:957 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.342: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:873
2012-09-03 12:45:41.342: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:957 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.342: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:873
2012-09-03 12:45:41.342: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:34
2012-09-03 12:45:41.342: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdagent_root
2012-09-03 12:45:41.342: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-09-03 12:45:41.342: [ CRSPE][2771286928] Received reply to action [Clean]
message ID: 873
2012-09-03 12:45:41.342: [ CRSPE][2771286928] Received reply to action [Clean]

message ID: 873

2012-09-03 12:45:41.343: [
'rac1' succeeded

CRSPE][2771286928] CRS-2681: Clean of 'ora.cssd' on

2012-09-03 12:45:41.343: [ CRSPE][2771286928] Sequencer for [ora.cssd 1 1] has

completed with error: CRS-0215: Could not start resource 'ora.cssd'.
2012-09-03 12:45:41.343: [
9
2012-09-03 12:45:41.344: [
a.diskmon' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 96

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:41.344: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2681: Clean of 'ora.cssd' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.344: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:969
2012-09-03 12:45:41.344: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:969 to the agent /u01/grid/orac
le/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:45:41.345: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.diskmon' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.352: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:970 from the agent /u01/grid/oracle/pr
oduct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:45:41.352: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.diskmon 1 1] ID 4099:969
2012-09-03 12:45:41.352: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 969
2012-09-03 12:45:41.353: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:970 from the agent /u01/grid/oracle/pr
oduct/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:45:41.353: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.diskmon 1 1] ID 4099:969
2012-09-03 12:45:41.353: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 969
2012-09-03 12:45:41.353: [ CRSPE][2771286928] RI [ora.diskmon 1 1] new externa
l state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:41.353: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.diskmon'
on 'rac1' succeeded
2012-09-03 12:45:41.353: [
6
2012-09-03 12:45:41.354: [
a.cssdmonitor' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 97

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:41.354: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.diskmon' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.354: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:976
2012-09-03 12:45:41.354: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:976 to the agent /u01/grid/
oracle/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.354: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.356: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:358
2012-09-03 12:45:41.368: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:45:41.368: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:358
2012-09-03 12:45:41.382: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:977 from the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.382: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:976
2012-09-03 12:45:41.383: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:977 from the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.383: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 976
2012-09-03 12:45:41.383: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:976
2012-09-03 12:45:41.383: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 976
2012-09-03 12:45:41.383: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:50
2012-09-03 12:45:41.383: [ CRSPE][2771286928] RI [ora.cssdmonitor 1 1] new ext
ernal state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:41.383: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.383: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:50
2012-09-03 12:45:41.383: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.cssdmonit
or' on 'rac1' succeeded
2012-09-03 12:45:41.383: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded]

MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]

]
2012-09-03 12:45:41.384: [
7
2012-09-03 12:45:41.384: [
a.gpnpd' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 98

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:41.384: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987
2012-09-03 12:45:41.384: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987 to the agent /u01/grid/oracle
/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:41.385: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.399: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:41.400: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:41.400: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 7
2012-09-03 12:45:41.400: [ CRSCOMM][2788096912] IPC listener connection to membe
r 7 has been removed
2012-09-03 12:45:41.400: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:7|Type:3}
2012-09-03 12:45:41.400: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:7|Type:3}
2012-09-03 12:45:41.400: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:41.400: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:41.400: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/orarootagent_root disconnected.
2012-09-03 12:45:41.401: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/orarootagent_root[5475] stopped!
2012-09-03 12:45:41.401: [ CRSCOMM][2781793168] removeConnection: Member 7 does
not exist.
2012-09-03 12:45:42.411: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:988 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:42.411: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987
2012-09-03 12:45:42.411: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 987
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 9
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912] IPC listener connection to membe

r 9 has been removed

2012-09-03 12:45:43.007: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:9|Type:3}
2012-09-03 12:45:43.007: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:9|Type:3}
2012-09-03 12:45:43.008: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:43.008: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:43.008: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/cssdmonitor_root disconnected.
2012-09-03 12:45:43.008: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdmonitor_root[5938] stopped!
2012-09-03 12:45:43.008: [ CRSCOMM][2781793168] removeConnection: Member 9 does
not exist.
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 10
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912] IPC listener connection to membe
r 10 has been removed
2012-09-03 12:45:43.207: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:10|Type:3}
2012-09-03 12:45:43.207: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:10|Type:3}
2012-09-03 12:45:43.207: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:43.207: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:43.207: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/cssdagent_root disconnected.
2012-09-03 12:45:43.207: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdagent_root[5958] stopped!
2012-09-03 12:45:43.207: [ CRSCOMM][2781793168] removeConnection: Member 10 does
not exist.
2012-09-03 12:45:43.209: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:988 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:43.209: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987
2012-09-03 12:45:43.210: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 987
2012-09-03 12:45:43.210: [ CRSPE][2771286928] RI [ora.gpnpd 1 1] new external
state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:43.210: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.gpnpd' on
'rac1' succeeded
2012-09-03 12:45:43.211: [
36
2012-09-03 12:45:43.211: [
a.mdnsd' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 10

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:43.211: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]

]
2012-09-03 12:45:43.211: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036
2012-09-03 12:45:43.211: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036 to the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:43.212: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:43.570: [ CRSCCL][2811444112]Restarting RD find queries.
2012-09-03 12:45:44.240: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1037 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:44.240: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036
2012-09-03 12:45:44.241: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1036
2012-09-03 12:45:45.029: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1037 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:45.029: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036
2012-09-03 12:45:45.030: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1036
2012-09-03 12:45:45.030: [ CRSPE][2771286928] RI [ora.mdnsd 1 1] new external
state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:45.030: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.mdnsd' on
'rac1' succeeded
2012-09-03 12:45:45.030: [
43
2012-09-03 12:45:45.030: [
a.gipcd' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 10

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:45.031: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:45.031: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043
2012-09-03 12:45:45.031: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043 to the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:45.031: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1']
MSGTYPE:

TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]

]
2012-09-03 12:45:46.050: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1044 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:46.050: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043
2012-09-03 12:45:46.051: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1043
2012-09-03 12:45:46.849: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1044 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:46.849: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043
2012-09-03 12:45:46.850: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1043
2012-09-03 12:45:46.850: [ CRSPE][2771286928] RI [ora.gipcd 1 1] new external
state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:46.850: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.gipcd' on
'rac1' succeeded
2012-09-03 12:45:46.850: [ CRSPE][2771286928] PE Command [ Start Resource : 0x
b109ff8 ] has completed
2012-09-03 12:45:46.851: [ CRSPE][2771286928] UI Command [Start Resource : 0xb
109ff8] is replying to sender.
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:644
2012-09-03 12:45:46.851: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:644
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: CMD_COMPLETED[Proxy] ID 20482:1054
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: CMD_COMPLETED[Proxy] ID 20482:1054
2012-09-03 12:45:46.852: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:46.852: [UiServer][2767084432] Container [ Name: UI_DATA
ora.cssd:
TextMessage[215]
]
2012-09-03 12:45:46.852: [UiServer][2767084432] Done for ctx=0xa4700c80
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 5
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912] IPC listener connection to membe

r 5 has been removed

2012-09-03 12:45:46.860: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:5|Type:3}
2012-09-03 12:45:46.860: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:5|Type:3}
2012-09-03 12:45:46.860: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:46.860: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:46.860: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/oraagent_oracle disconnected.
2012-09-03 12:45:46.860: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/oraagent_oracle[5397] stopped!
2012-09-03 12:45:46.860: [ CRSCOMM][2781793168] removeConnection: Member 5 does
not exist.
2012-09-03 12:45:46.878: [UiServer][2764983184] Closed: remote end failed/disc.
2012-09-03 12:49:53.783: [UiServer][2764983184] S(0xad408c00): set Properties (
root,0xb0b66b8)
2012-09-03 12:49:53.783: [UiServer][2764983184] S(0xb0ab0a0): Accepted client co
nnection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=29)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2012-09-03 12:49:53.801: [UiServer][2767084432] processMessage called
2012-09-03 12:49:53.801: [UiServer][2767084432] Sending message to PE. ctx= 0xad
449980
2012-09-03 12:49:53.801: [UiServer][2767084432] Sending command to PE: 5
2012-09-03 12:49:53.802: [ CRSPE][2771286928] Processing PE command id=106. De
scription: [Stat Resource : 0xb109ff8]
2012-09-03 12:49:53.807: [ CRSPE][2771286928] PE Command [ Stat Resource : 0xb
109ff8 ] has completed
2012-09-03 12:49:53.808: [ CRSPE][2771286928] UI Command [Stat Resource : 0xb1
09ff8] is replying to sender.
2012-09-03 12:49:53.809: [UiServer][2767084432] Done for ctx=0xad449980
2012-09-03 12:49:53.821: [UiServer][2764983184] Closed: remote end failed/disc.
[root@rac1 ohasd]# cat alertrac1.log
cat: alertrac1.log: No such file or directory
[root@rac1 ohasd]# tail -200f ohasd.log
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.356: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:358
2012-09-03 12:45:41.368: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/orarootagent_root
2012-09-03 12:45:41.368: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:358
2012-09-03 12:45:41.382: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:977 from the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.382: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:976
2012-09-03 12:45:41.383: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:977 from the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.383: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 976
2012-09-03 12:45:41.383: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.cssdmonitor 1 1] ID 4099:976
2012-09-03 12:45:41.383: [ CRSPE][2771286928] Received reply to action [Stop]

message ID: 976

2012-09-03 12:45:41.383: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:50
2012-09-03 12:45:41.383: [ CRSPE][2771286928] RI [ora.cssdmonitor 1 1] new ext
ernal state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:41.383: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/cssdmonitor_root
2012-09-03 12:45:41.383: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:50
2012-09-03 12:45:41.383: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.cssdmonit
or' on 'rac1' succeeded
2012-09-03 12:45:41.383: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.384: [ CRSPE][2771286928] Sending message to agfw: id = 98
7
2012-09-03 12:45:41.384: [ CRSPE][2771286928] CRS-2673: Attempting to stop 'or
a.gpnpd' on 'rac1'
2012-09-03 12:45:41.384: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987
2012-09-03 12:45:41.384: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987 to the agent /u01/grid/oracle
/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:41.385: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:41.399: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:41.400: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:41.400: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 7
2012-09-03 12:45:41.400: [ CRSCOMM][2788096912] IPC listener connection to membe
r 7 has been removed
2012-09-03 12:45:41.400: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:7|Type:3}
2012-09-03 12:45:41.400: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:7|Type:3}
2012-09-03 12:45:41.400: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:41.400: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:41.400: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/orarootagent_root disconnected.
2012-09-03 12:45:41.401: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/orarootagent_root[5475] stopped!

2012-09-03 12:45:41.401: [ CRSCOMM][2781793168] removeConnection: Member 7 does

not exist.
2012-09-03 12:45:42.411: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:988 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:42.411: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987
2012-09-03 12:45:42.411: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 987
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 9
2012-09-03 12:45:43.007: [ CRSCOMM][2788096912] IPC listener connection to membe
r 9 has been removed
2012-09-03 12:45:43.007: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:9|Type:3}
2012-09-03 12:45:43.007: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:9|Type:3}
2012-09-03 12:45:43.008: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:43.008: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:43.008: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/cssdmonitor_root disconnected.
2012-09-03 12:45:43.008: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdmonitor_root[5938] stopped!
2012-09-03 12:45:43.008: [ CRSCOMM][2781793168] removeConnection: Member 9 does
not exist.
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 10
2012-09-03 12:45:43.206: [ CRSCOMM][2788096912] IPC listener connection to membe
r 10 has been removed
2012-09-03 12:45:43.207: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:10|Type:3}
2012-09-03 12:45:43.207: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:10|Type:3}
2012-09-03 12:45:43.207: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:43.207: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:43.207: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/cssdagent_root disconnected.
2012-09-03 12:45:43.207: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/cssdagent_root[5958] stopped!
2012-09-03 12:45:43.207: [ CRSCOMM][2781793168] removeConnection: Member 10 does
not exist.
2012-09-03 12:45:43.209: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:988 from the agent /u01/grid/oracle/prod
uct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:43.209: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.gpnpd 1 1] ID 4099:987
2012-09-03 12:45:43.210: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 987
2012-09-03 12:45:43.210: [ CRSPE][2771286928] RI [ora.gpnpd 1 1] new external
state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:43.210: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.gpnpd' on
'rac1' succeeded

2012-09-03 12:45:43.211: [
36
2012-09-03 12:45:43.211: [
a.mdnsd' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 10

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:43.211: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:43.211: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036
2012-09-03 12:45:43.211: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036 to the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:43.212: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:43.570: [ CRSCCL][2811444112]Restarting RD find queries.
2012-09-03 12:45:44.240: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1037 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:44.240: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036
2012-09-03 12:45:44.241: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1036
2012-09-03 12:45:45.029: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1037 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:45.029: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.mdnsd 1 1] ID 4099:1036
2012-09-03 12:45:45.030: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1036
2012-09-03 12:45:45.030: [ CRSPE][2771286928] RI [ora.mdnsd 1 1] new external
state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:45.030: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.mdnsd' on
'rac1' succeeded
2012-09-03 12:45:45.030: [
43
2012-09-03 12:45:45.030: [
a.gipcd' on 'rac1'

CRSPE][2771286928] Sending message to agfw: id = 10

CRSPE][2771286928] CRS-2673: Attempting to stop 'or

2012-09-03 12:45:45.031: [UiServer][2767084432] Container [ Name: ORDER

MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded]
MSGTYPE:

TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]

]
2012-09-03 12:45:45.031: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043
2012-09-03 12:45:45.031: [
AGFW][2781793168] Agfw Proxy Server forwarding the
message: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043 to the agent /u01/grid/oracl
e/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:45.031: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
]
2012-09-03 12:45:46.050: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1044 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:46.050: [
AGFW][2781793168] Agfw Proxy Server sending the re
ply to PE for message:RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043
2012-09-03 12:45:46.051: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1043
2012-09-03 12:45:46.849: [
AGFW][2781793168] Received the reply to the messag
e: RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1044 from the agent /u01/grid/oracle/pro
duct/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:46.849: [
AGFW][2781793168] Agfw Proxy Server sending the la
st reply to PE for message:RESOURCE_STOP[ora.gipcd 1 1] ID 4099:1043
2012-09-03 12:45:46.850: [ CRSPE][2771286928] Received reply to action [Stop]
message ID: 1043
2012-09-03 12:45:46.850: [ CRSPE][2771286928] RI [ora.gipcd 1 1] new external
state [OFFLINE] old value: [ONLINE] label = []
2012-09-03 12:45:46.850: [ CRSPE][2771286928] CRS-2677: Stop of 'ora.gipcd' on
'rac1' succeeded
2012-09-03 12:45:46.850: [ CRSPE][2771286928] PE Command [ Start Resource : 0x
b109ff8 ] has completed
2012-09-03 12:45:46.851: [ CRSPE][2771286928] UI Command [Start Resource : 0xb
109ff8] is replying to sender.
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: AGENT_SUICIDE[Proxy] ID 20486:644
2012-09-03 12:45:46.851: [
AGFW][2781793168] Suicide request received from /u
01/grid/oracle/product/11.2.0/grid_1/bin/oraagent_oracle
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: AGENT_SUICIDE[Proxy] ID 20486:644
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server received the m
essage: CMD_COMPLETED[Proxy] ID 20482:1054
2012-09-03 12:45:46.851: [
AGFW][2781793168] Agfw Proxy Server replying to th
e message: CMD_COMPLETED[Proxy] ID 20482:1054
2012-09-03 12:45:46.852: [UiServer][2767084432] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:

TextMessage[ora.cssd]
WAIT:
TextMessage[0]

]
2012-09-03 12:45:46.852: [UiServer][2767084432] Container [ Name: UI_DATA
ora.cssd:
TextMessage[215]
]
2012-09-03 12:45:46.852: [UiServer][2767084432] Done for ctx=0xa4700c80
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912][FFAIL] Couldnt clscreceive messa
ge, no message: 11
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912] Client disconnected.
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912][FFAIL] Listener got clsc error 1
1 for memNum. 5
2012-09-03 12:45:46.859: [ CRSCOMM][2788096912] IPC listener connection to membe
r 5 has been removed
2012-09-03 12:45:46.860: [CLSFRAME][2788096912] Removing IPC Member:{Relative|No
de:0|Process:5|Type:3}
2012-09-03 12:45:46.860: [CLSFRAME][2788096912] Disconnected from AGENT process:
{Relative|Node:0|Process:5|Type:3}
2012-09-03 12:45:46.860: [ CRSPE][2771286928] Disconnected from server:
2012-09-03 12:45:46.860: [
AGFW][2781793168] Agfw Proxy Server received proce
ss disconnected notification, count=1
2012-09-03 12:45:46.860: [
AGFW][2781793168] /u01/grid/oracle/product/11.2.0/
grid_1/bin/oraagent_oracle disconnected.
2012-09-03 12:45:46.860: [
AGFW][2781793168] Agent /u01/grid/oracle/product/1
1.2.0/grid_1/bin/oraagent_oracle[5397] stopped!
2012-09-03 12:45:46.860: [ CRSCOMM][2781793168] removeConnection: Member 5 does
not exist.
2012-09-03 12:45:46.878: [UiServer][2764983184] Closed: remote end failed/disc.
2012-09-03 12:49:53.783: [UiServer][2764983184] S(0xad408c00): set Properties (
root,0xb0b66b8)
2012-09-03 12:49:53.783: [UiServer][2764983184] S(0xb0ab0a0): Accepted client co
nnection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=29)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2012-09-03 12:49:53.801: [UiServer][2767084432] processMessage called
2012-09-03 12:49:53.801: [UiServer][2767084432] Sending message to PE. ctx= 0xad
449980
2012-09-03 12:49:53.801: [UiServer][2767084432] Sending command to PE: 5
2012-09-03 12:49:53.802: [ CRSPE][2771286928] Processing PE command id=106. De
scription: [Stat Resource : 0xb109ff8]
2012-09-03 12:49:53.807: [ CRSPE][2771286928] PE Command [ Stat Resource : 0xb
109ff8 ] has completed
2012-09-03 12:49:53.808: [ CRSPE][2771286928] UI Command [Stat Resource : 0xb1
09ff8] is replying to sender.
2012-09-03 12:49:53.809: [UiServer][2767084432] Done for ctx=0xad449980
2012-09-03 12:49:53.821: [UiServer][2764983184] Closed: remote end failed/disc.
[root@rac1 ohasd]# ps -ef|grep grid_1/bin|wc -l
2
[root@rac1 ohasd]# ps -ef|grep grid_1/bin
root
5249
1 0 12:35 ?
00:00:01 /u01/grid/oracle/product/11.2.0/
grid_1/bin/ohasd.bin exclusive
root
6157 5608 0 12:50 pts/1
00:00:00 grep grid_1/bin
[root@rac1 ohasd]# ps -ef|grep grid_1/bin
root
6182 5608 0 12:50 pts/1
00:00:00 grep grid_1/bin
[root@rac1 ohasd]#