Document 1054902.1
Document 1054902.1
Dashboard Knowledge Service Requests Patches & Updates Community Powered by AI. Do not input sensitive data
Give Feedback...
Copyright (c) 2025, Oracle. All rights reserved. Oracle Confidential.
How to Validate Network and Name Resolution Setup for the Clusterware and RAC (Doc ID 1054902.1) To Bottom
Purpose Yes
No
Scope
Details
Document Details
A. Requirement
B. Example of what we expect
Type:
C. Syntax reference Status: BULLETIN
Last Major PUBLISHED
D. Multicast 28-Mar-2025
Update:
E. Runtime network issues Last 28-Mar-2025
Update: English
F. Symptoms of network issues
Language:
G. Basics of Subnet
References
Related Products
Gen 1 Exadata Cloud at
APPLIES TO: Customer (Oracle Exadata
Database Cloud Machine)
Oracle Cloud Infrastructure -
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later Database Service
Oracle Cloud Infrastructure - Database Service - Version N/A and later Oracle Database Backup
Oracle Database Backup Service - Version N/A and later Service
Oracle Database - Enterprise
Oracle Database - Enterprise Edition - Version 10.1.0.2 and later Edition
Oracle Database Cloud Exadata Service - Version N/A and later Oracle Database Cloud
Generic UNIX Exadata Service
Generic Linux Show More
o Network ping with package size of Network Adapter (NIC) MTU should work on all public and private network and the time of
ping should be small (sub-second). Document References
Troubleshoot Grid
o IP address 127.0.0.1 should only map to localhost and/or localhost.localdomain, not anything else. Infrastructure/RAC Database
installer/runInstaller Issues
o 127.*.*.* should not be used by any network interface. [1056322.1]
o Public NIC name must be same on all nodes. Grid Infrastructure Startup
During Patching, Install or
Upgrade May Fail Due to
o Private NIC name should be same in 11gR2 and must be same for pre-11gR2 on all nodes Multicasting Requirement
[1212703.1]
o Public and private network must not be in link local subnet (169.254.*.*), should be in non-related separate subnet.
OSWatcher [301137.1]
o MTU should be the same for corresponding network on all nodes. Grid Infrastructure Redundant
Interconnect and
o Network size should be same for corresponding network on all nodes. ora.cluster_interconnect.haip
[1210883.1]
o As the private network needs to be directly attached, traceroute should work with a packet size of NIC MTU without Oracle Clusterware Cannot
fragmentation or going through the routing table on all private networks in 1 hop. Start on all Nodes: Network
communication with node
<NAME> missing for 90% of
o Firewall needs to be turned off on the private network. timeout interval [1507482.1]
o For 10.1 to 11.1, name resolution should work for the public, private and virtual names. Show More
Recently Viewed
o For 11.2 without Grid Naming Service (aka GNS), name resolution should work for all public, virtual, and SCAN names; and if
SCAN is configured in DNS, it should not be in local hosts file. Top 5 Issues That Cause
Node Reboots or Evictions or
o For 11.2.0.2 and above, multicast group 230.0.1.0 should work on private network; with patch 9974223, both group 230.0.1.0 Unexpected Recycle of CRS
and 224.0.0.251 are supported. With patch 10411721 (fixed in 11.2.0.3), broadcast will be supported. See Multicast/Broadcast [1367153.1]
section to verify. ORA-29740: RAC Instance
Crash / Eviction [1951228.1]
o For 11.2.0.1-11.2.0.3, Installer may report a warning if reverse lookup is not setup correctly for pubic IP, node VIP, and SCAN Troubleshooting gc block lost
VIP, with bug 9574976 fix in 11.2.0.4, the warning shouldn't be there any more. and Poor Network
Performance in a RAC
o OS level bonding is recommended for the private network for pre-11.2.0.2. Depending on the platform, you may implement Environment [563566.1]
bonding, teaming, Etherchannel, IPMP, MultiPrivNIC etc, please consult with your OS vendor for details. Started from 11.2.0.2, Recommendation for the
Real Application Cluster
Redundant Interconnect and HAIP is introduced to provide native support for multiple private network, refer to note 1210883.1
Interconnect and Jumbo
for details. Frames [341788.1]
Release Schedule of Current
o The commands verifies jumbo frames if it's configured. To know more about jumbo frames, refer to note 341788.1 Database Releases
[742060.1]
Show More
B. Example of what we expect
Example below shows what we expect while validating the network and name resolution setup. As the network setup is slightly
different for 11gR2 and 11gR1 or below, we have both case in the below example. The difference between 11gR1 or below and
11gR2 is for 11gR1, we need a public name, VIP name, private hostname, and we rely on the private name to find out the
private IP for cluster communication. For 11gR2, we do not rely on the private name anymore, rather the private network is
selected based on the GPnP profile while the clusterware comes up. Assuming a 3-node cluster with the following node
information:
Nodename |Public IP |VIP name |VIP |Private |private IP1 |private IP2
---------|----------|---------|-----------|--------|----------------------
|eth0/1500 | | | |eth1/1500 |
---------|----------|---------|-----------|--------|----------------------
|eth0/1500 | | | |eth1/1500 |
---------|----------|---------|-----------|--------|----------------------
|eth0/1500 | | | |eth1/1500 |
---------|----------|---------|-----------|--------|----------------------
11gR2 cluster
|NIC/MTU | | |NIC/MTU |
---------|----------|---------|-----------|------------|----------
|eth0/1500 | | |eth1/1500 |
---------|----------|---------|-----------|------------|----------
|eth0/1500 | | |eth1/1500 |
---------|----------|---------|-----------|------------|----------
|eth0/1500 | | |eth1/1500 |
---------|----------|---------|-----------|------------|----------
----------|-----------|-----------|--------------------
----------|-----------|-----------|--------------------
Below is what is needed to be verify on each node - please note the example is from a Linux platform:
/bin/netstat -in
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 203273 0 0 0 2727 0 0 0 BMRU
2. To find out the IP address and subnet, compare Broadcast and Netmask on all nodes
/sbin/ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3E:11:11:11
inet addr:120.X.X.1 Bcast:120.xxx.xxx.127 Mask:255.xxx.xxx.128
inet6 addr: fe80::216:3eff:fe11:1111/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:203245 errors:0 dropped:0 overruns:0 frame:0
TX packets:2681 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:63889908 (60.9 MiB) TX bytes:319837 (312.3 KiB)
..
In the above example, the IP address for eth0 is 120.X.X.1, broadcast is 120.X.X.127, and net mask is 255.xxx.xxx.128, which is
subnet of 120.xxx.xxx.0 with a maximum of 126 IP addresses. Refer to Section "Basics of Subnet" for more details.
Note: An active NIC must have both "UP" and "RUNNING" flag; on Solaris, "PHYSRUNNING" will indicate whether the physical
interface is running
Below is an example ping output from node1 public IP to node2 public hostname:
Please pay attention to the packet loss and time. If it is not 0% packet loss, or if it is not sub-second time, then it indicates there
is a problem in the network. Please engage network administrator to check further.
3.1 Ping all public nodenames from the local public IP with packet size of MTU
3.2.1 Ping all private IP(s) from all local private IP(s) with packet size of MTU
applies to 11gR2 example, private name is optional
3.2.2 Ping all private nodename from local private IP with packet size of MTU
applies to 11gR1 and earlier example
Example below shows traceroute from node1 private IP to node2 private hostname
# Packet size of MTU - on Linux packet length needs to be MTU - 28 bytes otherwise error send: Message too
long is reported.
# For example with MTU value of 1500 we would use 1472 :
MTU size packet traceroute complete in 1 hop without going through the routing table. Output other than above indicates
issue, i.e. when "*" or "!H" presents.
Note: traceroute option "-F" may not work on RHEL3/4 OEL4 due to OS bug, refer to note: 752844.1 for details.
4.1 Traceroute all private IP(s) from all local private IP(s) with :
applies to 11gR2 onwards
If "-F" option does not work, then traceroute without the "-F" parameter but with packet that's triple the MTU size, i.e.:
4.2 Traceroute all private nodename from local private IP with packet size of MTU
applies to 11gR1 and earlier example
If "-F" option does not work, then run traceroute without the "-F" parameter but with packet that's triple MTU size, i.e.:
/bin/ping -c 2 <nodename1>v
/bin/ping -c 2 <nodename1>v
/bin/ping -c 2 <nodename2>v
/bin/ping -c 2 <nodename2>v
/bin/ping -c 2 <nodename3>v
/bin/ping -c 2 <nodename3>v
/usr/bin/nslookup <nodename1>v
/usr/bin/nslookup <nodename2>v
/usr/bin/nslookup <nodename3>v
/usr/bin/nslookup <scanname>
# SCAN example
# If SCAN name is setup in DNS, it should not be in local hosts file
/bin/grep <scanname> /etc/hosts
/bin/grep 120.X.X.21 /etc/hosts
/bin/grep 120.X.X.22 /etc/hosts
/bin/grep 120.X.X.23 /etc/hosts
C. Syntax reference
Linux:
/bin/netstat -in
/sbin/ifconfig
/bin/ping -s <MTU> -c 2 -I source_IP nodename
/bin/traceroute -s source_IP -r -F nodename-priv <MTU-28>
/usr/bin/nslookup
Solaris:
/bin/netstat -in
/usr/sbin/ifconfig -a
/usr/sbin/ping -i source_IP -s nodename <MTU> 2
/usr/sbin/traceroute -s source_IP -r -F nodename-priv <MTU>
/usr/sbin/nslookup
HP-UX:
/usr/bin/netstat -in
/usr/sbin/ifconfig NIC
/usr/sbin/ping -i source_IP nodename <MTU> -n 2
/usr/contrib/bin/traceroute -s source_IP -r -F nodename-priv <MTU>
/bin/nslookup
AIX:
/bin/netstat -in
/usr/sbin/ifconfig -a
/usr/sbin/ping -S source_IP -s <MTU> -c 2 nodename
/bin/traceroute -s source_IP -r nodename-priv <MTU>
/bin/nslookup
Windows:
MTU:
Windows XP: netsh interface ip show interface
Windows Vista/7: netsh interface ipv4 show subinterfaces
ipconfig /all
ping -n 2 -l <MTU-28> -f nodename
tracert
nslookup
D. Multicast
Started with 11.2.0.2, multicast group 230.0.1.0 should work on private network for bootstrapping. patch 9974223 introduces
support for another group 224.0.0.251
As fix for bug 10411721 is included in 11.2.0.3, broadcast is supported for bootstrapping as well as multicast. When 11.2.0.3
Grid Infrastructure starts up, it will try broadcast, multicast group 230.0.1.0 and 224.0.0.251 simultaneously, if anyone succeeds,
it will be able to start.
On hp-ux, if 10 Gigabit Ethernet is used as private network adapter, without driver revision B.11.31.1011 or later of the
10GigEthr-02 software bundle, multicast may not work. Run "swlist 10GigEthr-02" command to identify the current version on
your HP server.
OSWatcher or Cluster Health Monitor(IPD/OS) can be deployed to capture runtime network issues.
o ping doesn't work, ping packet loss or ping time is too long (not sub-second)
o gipcd.log shows:
or
o ocssd.log shows:
or
o crsd.log shows:
or
or
o octssd.log shows:
G. Basics of Subnet
REFERENCES
NOTE:1212703.1 - Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement
NOTE:301137.1 - OSWatcher
Related
Products
Oracle Cloud > Oracle Infrastructure Cloud > Oracle Cloud at Customer > Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine)
Oracle Cloud > Oracle Platform Cloud > Oracle Cloud Infrastructure - Database Service > Oracle Cloud Infrastructure - Database Service
Oracle Cloud > Oracle Platform Cloud > Oracle Database Backup Service > Oracle Database Backup Service
Oracle Database Products > Oracle Database Suite > Oracle Database > Oracle Database - Enterprise Edition > Clusterware
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Exadata Service > Oracle Database Cloud Exadata Service
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Service > Oracle Database Exadata Express Cloud Service
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Service > Oracle Database Cloud Service
Oracle Database Products > Oracle Database Suite > Oracle Database > Oracle Database - Standard Edition > Real Application Cluster
Oracle Database Products > Oracle Database Suite > Oracle Database > Oracle Database - Standard Edition > Clusterware > Clusterware Install
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Service > Oracle Database Cloud Schema Service
Keywords
ADDRESS; CLUVFY; CRS; FRAGMENTATION; GRID INFRASTRUCTURE; INFRASTRUCTURE; IP ADDRESS; RAC; VERIFICATION
Errors
ORA-12514
Translations
English Source Chinese 简体中文 Japanese 日本語 Korean 한국어
Back to Top
Copyright (c) 2025, Oracle. All rights reserved. Legal Notices and Terms of Use Privacy Statement