Starting:

Starting Cluster Services on All DB Nodes

Connect to your individual account on one of the DB nodes and execute

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl start cluster -all

Check Status of Cluster resources

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl status resource -t

Starting Cluster Services on specific DB Node

Connect to your individual account on specific DB Node and execute

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl start cluster

Check Status of Cluster resources

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl status resource -t

Starting Database Instances on All DB Nodes

Connect to oracle account on one of the DB nodes and execute

srvctl start database -d <dbname>

srvctl start database -d CVUT1

Check Status of Database Instances

srvctl status database -d <dbname>

srvctl status database -d CVUT1

Instance cvsan1 is running on node ebizoracsb01

Instance cvsan2 is running on node ebizoracsb02
Instance cvsan3 is running on node ebizoracsb03

Check Status of Database services to confirm they started under correct instance
srvctl status service -d <dbname> -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

srvctl status service -d CVUT1 -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

srvctl status service -d <dbname>

srvctl status service -d CVUT1

 Service cp_service1 is running on instance(s) cvsan1
Service cp_service2 is running on instance(s) cvsan2
Service cp_service3 is running on instance(s) cvsan3
Service app_service1 is running on instance(s) cvsan1
Service app_service2 is running on instance(s) cvsan2
Service app_service3 is running on instance(s) cvsan3

Note: It takes few seconds for services to start after Database Instances come up.

Starting specific Database Instance

Connect to oracle account on one of the DB nodes and execute

srvctl start instance -d <dbname> -i <instance name>

srvctl start instance -d cvsan -i cvsan1

srvctl status instance -d <dbname> -i <instance name>

srvctl status instance -d cvsan -i cvsan1

Instance cvsan1 is running on node ebizoracsb01

Check Status of Database services to confirm they relocated under correct instance
srvctl status service -d <dbname> -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Service cp_service1 is running on instance(s) cvsan1

Service cp_service2 is running on instance(s) cvsan2
Service cp_service3 is running on instance(s) cvsan3
Service app_service1 is running on instance(s) cvsan1
Service app_service2 is running on instance(s) cvsan2
Service app_service3 is running on instance(s) cvsan3

Note: It takes few seconds for services to start after Database Instance(s) come up.

Starting Database Services

Database services start automatically using FAN script after instances come up. Check Services status
srvctl status service -d <dbname> -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

srvctl status service -d CVUT1 -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Service cp_service1 is running on instance(s) cvsan1

Service cp_service2 is running on instance(s) cvsan2
 Service cp_service3 is running on instance(s) cvsan3
Service app_service1 is running on instance(s) cvsan1
Service app_service2 is running on instance(s) cvsan2
Service app_service3 is running on instance(s) cvsan3

If the service(s) listed above are not running, then start them
srvctl start service -d <dbname> -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

To relocate services from one instance to another instance if not running under appropriate instance
srvctl relocate service -db <dbname> -service <service_name> -i <from instance> -t <to instance>

Starting Application Services

After database instances are services are up and running, it is time to bring up the application services.

Start APPS listener(adalnctl.sh) on all application nodes so that when adstrtal.sh is run, all
managers startup on their Primary Node

Run $ADMIN_SCRIPTS_HOME/adstrtal.sh on all nodes starting with Primary Node.

Another option to startup applications on All Application Nodes from Primary Node is

$ADMIN_SCRIPTS_HOME/adstrtal.sh -mode=allnodes

Stopping:

Stopping Application Services

Run $ADMIN_SCRIPTS_HOME/adstpall.sh on all nodes starting with Primary Node.

Another option to stop applications on All Application Nodes from Primary Node is

$ADMIN_SCRIPTS_HOME/adstpall.sh -mode=allnodes

Stopping Database Instances on All DB Nodes

Connect to oracle account on one of the DB nodes and execute

srvctl stop database -d <dbname>

srvctl stop database -d CVUT1

Check Status of Database Instances

srvctl status database -d <dbname>

srvctl status database -d CVUT1

[ocvprd@ebizoracpr01 ~]$ srvctl status database -d CVPRD

Instance cvprd1 is running on node ebizoracpr01

Instance cvprd2 is running on node ebizoracpr02

Instance cvprd3 is running on node ebizoracpr03

[ocvprd@ebizoracpr01 ~]$

Instance cvsan1 is not running on node ebizoracsb01

Instance cvsan2 is not running on node ebizoracsb02
Instance cvsan3 is not running on node ebizoracsb03

Check Status of Database services to confirm they are down

srvctl status service -d <dbname> -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

srvctl status service -d CVUT1 -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

srvctl status service -d CVPRD -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

[ocvprd@ebizoracpr01 ~]$ srvctl status service -d CVPRD -s

cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Service cp_service1 is running on instance(s) cvprd1

Service cp_service2 is running on instance(s) cvprd2

Service cp_service3 is running on instance(s) cvprd3

Service app_service1 is running on instance(s) cvprd1

Service app_service2 is running on instance(s) cvprd2

Service app_service3 is running on instance(s) cvprd3

You have new mail in /var/spool/mail/ocvprd

[ocvprd@ebizoracpr01 ~]$

Service cp_service1 is not running.

Service cp_service2 is not running.
Service cp_service3 is not running.
Service app_service1 is not running.
 Service app_service2 is not running.
Service app_service3 is not running.

Stopping specific Database Instance

Connect to oracle account on one of the DB nodes and execute

srvctl stop instance -d <dbname> -i <instance name> -failover

srvctl stop instance -d cvsan -i cvsan1 -failover

“-failover” argument failover the DB services running on stopped instance to Available Instance

srvctl status instance -d <dbname> -i <instance name>

srvctl status instance -d cvsan -i cvsan1

Instance cvsan1 is not running on node ebizoracsb01

Check Status of Database services to confirm they relocated to available instance

srvctl status service -d <dbname> -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Service cp_service1 is running on instance(s) cvsan2

Service cp_service2 is running on instance(s) cvsan2
Service cp_service3 is running on instance(s) cvsan3
Service app_service1 is running on instance(s) cvsan2
Service app_service2 is running on instance(s) cvsan2
Service app_service3 is running on instance(s) cvsan3

Stopping Cluster Services on All DB Nodes

Connect to your individual account on one of the DB nodes and execute

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl stop cluster -all -f

Check Status of Cluster resources

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl status resource -t

Stopping Cluster Services on specific DB Node

Connect to your individual account on specific DB Node and execute

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl stop cluster -f

Check Status of Cluster resources

sudo /opt/app/grid/product/12.1.0.2/bin/crsctl status resource -t

Starting DB Services
srvctl start service -d cvsan -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Stopping DB Services
srvctl stop service -d cvsan -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Checking status of DB Services

srvctl status service -d cvsan -s cp_service1,cp_service2,cp_service3,app_service1,app_service2,app_service3

Relocate DB Services
srvctl relocate service -db <dbname> -service <service_name> -i <from instance> -t <to instance>
 Stopping / Starting Active Data Guard

Syntax to Stop a running Oracle Standby Database::

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

SHUTDOWN IMMEDIATE;

Syntax to Start a Oracle Standby Database::

STARTUP MOUNT;

ALTER DATABASE OPEN READ ONLY;

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Validate log sync is working between primary and ADG in v$archived_log table

Overall outage procedure

Stop –

1) JOBS ON HOLD

2) Crontab comment out on all servers.

3) Stop all three application node 3 > 2 > 1

ebizapplpr01.corp.coopervision.com

ebizapplpr02.corp.coopervision.com

ebizapplpr03.corp.coopervision.com

4) Stop weblogic JMS server

Login to http://weblapplpr03.corp.coopervision.com:7065/console weblogic/webgod_2020

Stop below mention admin and jms_server as mentioned below (force shutdown now)

5) Stop weblogic ADF server

Login to http://weblapplpr03.corp.coopervision.com:7001/console weblogic/webgod_2020

Stop below mention admin and adf_server as mentioned below (force shutdown now)

6) Kill weblogic process, login to below servers

weblapplpr03.corp.coopervision.com
 weblapplpr04.corp.coopervision.com

$ sudo su – weblogic

Select cvprd & 1

$ kill -9 -1

7) Stop standby database

ebizdbmspr03.corp.coopervision.com

$ sudo -u ocvprd -i

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

SQL > SHUTDOWN IMMEDIATE;

8) Stop CVPRD database

Stop Database / Instance (based on the requirement) using srvctl command as mentioned in
earlier section of document

-------------------------à Shutdown compelted,

OS patching , no need to stop database/instanceOnly apps shutdown required

Startup

1) Start database / if not already using using srvctl command as mentioned in earlier section of
document

Check the listener status its running

1) Start application

Start application services as mentioned in

earlier section either one by one after
 starting all three app listener or using
allnode option

2) Start weblogic JMS server

Login to weblapplpr03.corp.coopervision.com

$ sudo su - weblogic

Response cvprd and then 1

$ cd $DOMAIN_HOME/bin

$ . ./setDomainEnv.sh

$ cd $DOMAIN_HOME/bin

$ ./startWebLogic.sh &

Check webloigic server start RUNNING

$ ./startNodeManager.sh &

Login to weblapplpr04.corp.coopervision.com

$ sudo su - weblogic

Response cvprd and then 1

$ cd $DOMAIN_HOME/bin

$ . ./setDomainEnv.sh

$ cd $DOMAIN_HOME/bin

$ ./startNodeManager.sh &

Login to http://weblapplpr03.corp.coopervision.com:7065/console

Start all three jms_servers and ensure they are in runing startus (may take 5 mins to start)

After 5 mins check, all 306 bridges are active and forwarding messages
 3) Start weblogic ADF server

Login to weblapplpr03.corp.coopervision.com

$ sudo su - weblogic

Response cvprd and then 2

$ cd $DOMAIN_HOME/bin

$ . ./setDomainEnv.sh

$ cd $DOMAIN_HOME/bin

$ ./startWebLogic.sh &

Check webloigic server start RUNNING

$ ./startNodeManager.sh &

Login to weblapplpr04.corp.coopervision.com
$ sudo su - weblogic

Response cvprd and then 2

$ cd $DOMAIN_HOME/bin

$ . ./setDomainEnv.sh

$ cd $DOMAIN_HOME/bin

$ ./startNodeManager.sh &

Login to http://weblapplpr03.corp.coopervision.com:7001/console

Start all two adf_servers and ensure they are in runing startus (may take 5 mins to start)

Check below URL should open in crome/mozilla browser, it will ask for CVPRD eBS userid or password
and the should open

http://weblapplpr03.corp.coopervision.com:7003/ManageError/faces/
 SearchConsole.jsf

http://weblapplpr04.corp.coopervision.com:7003/ManageError

4) Start stand by database

Login to ebizdbmspr03.corp.coopervision.com

SQL > STARTUP MOUNT;

SQL > ALTER DATABASE OPEN READ ONLY;

SQl > ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

5) Application sanity check including mailer

6) RELESE JOBS FROM HOLD

7) Crontab activation on all servers.