Oracle E-Business Suite Network

Utilities: Best Practices

An Oracle White Paper
March 2008






Oracle E-Business Suite Network
Utilities: Best Practices
Executive Summary .......................................................................................... 3
Introduction ....................................................................................................... 3
The Oracle E-Business Suite Network Test Form....................................... 4
When is a Network Test not a Network Test?................................. 4
What Do the Latency Figures Represent?................................................. 4
What Variation Can I Expect?.................................................................... 6
What Do the Bandwidth Figures Represent? ........................................... 7
When and How to Use the Network Test Form..................................... 8
Using Ping to Establish Network Latency and Link Characteristics......... 8
Comparing Small and Large Packet Behavior .......................................... 9
Comparing Network Test Form Latency to Actual Network Latency10
Client System Analyzer ................................................................................... 10
Conclusion........................................................................................................ 12
Appendix A Additional Network Test Form Results............................. 13
Appendix B Ping, Traceroute, and the MTU.......................................... 14
Comparing Large and Small Packet Response Times ........................... 14
Example of Using Ping with a Small Packet ...................................... 14
Example of Using Ping with a Large Packet...................................... 14
Example Using Ping to Check Network Variance over Time......... 15
Using Shunra Cloud Catcher..................................................................... 15
Checking For Packet Fragmentation ....................................................... 16
Example Where Ping Matches the MTU............................................ 16
Example: Ping Exceeds the MTU....................................................... 17
Checking Network Routing....................................................................... 17
Utilities that can be used when ICMP is Blocked.................................. 17

Oracle E-Business Suite Network Utilities: Best Practices Page 2

EXECUTIVE SUMMARY
This white paper describes how to use the Oracle E-Business Suite Network Test
form (available in all versions) and Client Analyzer (introduced in 11.5.10) to help
identify the source of performance issues. Typically, these tools and associated
analytical techniques will form part of a holistic approach to troubleshooting. The
techniques can help isolate and identify problems on the network and middle tier.
Understanding the purpose of the tools and how to interpret their output enables
end-users to collect the requisite diagnostic information and perform rudimentary
diagnosis when performance problems occur. Consequently this saves the time and
expense of deploying specialized personnel in the early stages of a performance
investigation, or for lengthy periods waiting for intermittent problems to occur.
In addition to showing example usage of the tools, the paper presents sample
output for users to compare with their own measurements, and thereby draw
conclusions more quickly and effectively.
INTRODUCTION
Performance issues can be very frustrating for end-users, as well as complex to
diagnose and resolve. This is particularly true for problems experienced at remote
locations where help may not be readily available. Even more difficult is the
diagnosis of intermittent issues, especially when there does not appear to be a direct
correlation to a time or system load pattern. It can be costly to involve technology
teams spanning all the interacting technologies, and even more costly when
engaging external parties to assist.
The effects of network latency may become significant and inescapable for users in
remote locations working over a WAN link. This may lead to occasions where end-
user expectations and the throughput necessary to sustain the business will need to
be balanced against the practical and economic limits of the available network
capacity.
While there is a plethora of tools available to monitor the system, database, and
application tiers, there are very few available to diagnose performance at remote
locations. There are even fewer options available to an end-user.
There are two diagnostic tools available in the Oracle E-Business Suite:
The Network test Form
The Client Analyzer
It is important to understand the purpose of the tools, their limitations, and how to
interleave and compare the Oracle E-Business Suite network results with the
output from simple operating system commands. The techniques do not require
specialized skills and therefore much progress can be made towards identifying the
root cause of the problem by junior system administration staff and suitably
experienced end-users.
Oracle E-Business Suite Network Utilities: Best Practices Page 3

THE ORACLE E-BUSINESS SUITE NETWORK TEST FORM
The Network Test Form, as shown in Figure 1, is available under the System
Administrator responsibility.
The Network Test Form can be added to a
users menu so that they can run tests and
collate diagnostic information when
performance problems occur.


Figure 1: Network Test form
When is a Network Test not a Network Test ?
The name of the form is slightly misleading and perhaps should have been the
"Oracle Forms Network Test", as it actually shows the time taken for Oracle Forms
round trips. To measure client latency, the form simply updates a Form item and
then issues the synchronize built-in Oracle Forms command, which generates a
network round trip between the client and the middle tier. It is important to note
that the network latency test timings shown in this form do not correlate with the
results of standard network tests such as ping.
What Do the Latency Figures Represent?
Figure 1 shows the Network Test form with the results shown for a LAN
connection. Looking first at the form server to database connection. As the
database and application tiers should all be connected on the same isolated virtual
private network; the network latency should always be low and bandwidth should be
high. Latencies higher than 2-3ms tend to result in multiple performance
complaints from end-users. Regardless of the test, the sample data can be ignored
as you can build up your own sample data for specific locations.
The Network Test Form measures the time
for a Form round trip, which is not the
same as the time for a network round trip
as it includes middle tier processing time.
Oracle E-Business Suite Network Utilities: Best Practices Page 4

Notice that the client to middle tier time (in Figure 1) for this series of tests on a
LAN shows an average of 34.9ms, which is substantially higher than the expected
network latency of 0-3ms. This is because the Network Test form measurement
provides a composite value based on the network latency plus the time needed to
process the synchronize command on the middle tier. Therefore, in addition to
variations in the network, the time is also affected by the load on the middle tier.
This set of results fields would be better titled as Forms Latency Results to
differentiate it from the commonly accepted meaning of the term.
The Network Test form measurement
provides a composite value based on the
network latency plus the time needed to
process the synchronize command on the
middle tier.
To create a set of baseline figures that can be used for comparison, a WAN
simulator was used to replicate various network conditions. Figure 2 shows the test
environment.


Figure 2: WAN simulation
Independent testing using a Vocality WAN/Satellite Simulator provided identical
results. Figure 3 shows results for a controlled environment with an unloaded
middle tier.


Figure 3: Forms Latency compared to Simulated Network Latency
(controlled load test)
Oracle E-Business Suite Network Utilities: Best Practices Page 5

These charts show a very similar profile for all network connections and all adhere
to the same linear function (y = mx +c) - the equation of a straight line where m is
the gradient and c is a constant. In all cases the gradient m is 2. The constant c is
derived by the y-axis value when the latency is 0. This is referred to as the latency
offset and depends on the available network link bandwidth:
Form Ping time =
2 x Network Latency + Latency Offset.
Form Ping time = 2 x Network Latency + Latency Offset
The latency offsets for the available bandwidth for this test scenario are shown in
Table 1.
Available
Bandwidth
Latency
Offset
2048Kbps 36
1024Kbps 45
512Kbps 64
256Kbps 108
128Kbps 188
Table 1: Offsets for Available Link Bandwidth
What Variation Can I Expect?
The Network Test Screen results in Figure 3 were derived using a simulator in a
controlled environment; you may find that your results are more variable as a result
of fluctuations in network capacity and load on the middle tier.
Figure 4 shows an example of results that you might expect with a variable load
profile on a middle tier that was also very low on available memory. For clarity,
only the following two sets of measurements are shown:
Measurements are affected by network
latency and load on the middle tier.
No bandwidth restriction (effectively Unlimited on a gigabit switch)
256 Kbps


Figure 4: Forms Latency compared to Simulated Latency (variable load test)
Oracle E-Business Suite Network Utilities: Best Practices Page 6

Example results for all latencies in this particular scenario are shown in Appendix
A. If you experience a lot of variation caused by middle tier loading, compare
Network Test form results from the main site with those from remote users to help
identify which system component is causing the problem.
If you experience large variations in
middle tier loading, run the Network Test
form on a local LAN and compare
results with remote users.
What Do the Bandwidth Figures Represent?
The Network Test forms bandwidth measurement is more complex and the
evidence suggests that the results are less useful. Behind the scenes, the form
measures the time it takes to send a series of packets, which is affected not only by
the available bandwidth, but also by the network latency. This measurement can be
considered as a specialized throughput metric that may be of use when comparing
test runs at different times of the day, or from different locations.
Figure 5 shows the Bytes per Second result provided by the form, which has
been converted to Kbps for clarity and consistency.



Figure 5: Network Test Form Bandwidth compared to actual Network
latency and bandwidth
Oracle E-Business Suite Network Utilities: Best Practices Page 7

When and How to Use the Network Test Form
There are two main ways to use the form, which are as follows:
Compare results across locations with
similar links, and secondly, throughout the
day to see if there is a correlation between
the performance problem and poor results.
Comparisons between locations The form should show
approximately the same latency between the client and server for similar
network topologies in terms of latency, bandwidth, and saturation. A
substantial difference between round trip average times for users on
similar links at separate locations identifies a case for further network
analysis.
Comparisons throughout the day Run the test several times during
the day. If, when the performance problem occurs, the latency times are
about the same as at other times, you can conclude that the problem is not
in the network, and probably not in the middle tier.
If the results change dramatically for either of these scenarios, the next step is to
establish whether the problem is with the network or middle tier load. This can be
achieved by comparing the results with the network ping time, described next.
In essence, use the Network Test form to make comparisons do not be too
concerned about the actual values. Simply record and compare the latency figure,
which is more consistent and predictable than the Forms bandwidth measurement.
USING PING TO ESTABLISH NETWORK LATENCY AND LINK
CHARACTERISTICS
Oracle E-Business Suite traffic tends to have either large or small packets, with few
of an intermediate size. Provided that a network administrator has not blocked or
reduced the priority of ICMP (Internet Control Message Protocol) traffic (which
you should confirm before proceeding further), use ping to test the network from
the client to the server. The ping command and other utilities that can be used
when ICMP is blocked are described in Appendix B.
If ICMP is blocked, consider deploying a
Linux client at remote locations to enable
the use of utility programs such as tcpping
and tcptraceroute that can tunnel through
many common firewall filters.
Using both small (32 byte) and full size packets (1472 bytes for normal IP traffic)
mimics normal Oracle E-Business Suite network traffic. Using this approach, ping
can indicate whether there may be issues on the network that need further
investigation.
Comparing the ping round trip times for
small and large packets can indicate a
range of network problems.
Even though ping is a relatively simple test, you can make the following deductions:
The 32 byte ping command fails ICMP traffic may have been blocked
for security reasons. In this case, you will still be able to use the Network
Test form as it uses a different method to generate traffic, but you will not
be able to make comparisons between the two sets of results.
The 32 byte packet is very slow Higher than expected round trip times
for this small packet indicate either that ICMP priority has been reduced,
there are routing issues, or there are other network problems.
The 32 byte and 1472 byte timings are similar Similar results indicate
that there are no bandwidth or link saturation problems.
Oracle E-Business Suite Network Utilities: Best Practices Page 8

The 32 byte and 1472 byte timings are very different Significantly
higher times for the large size packet indicate a network problem such as
hitting bandwidth limitations or restrictions, link saturation, or packet
prioritization issues. Comparing results with users at other locations will
highlight specific network segments that may need to be investigated.
The round trip time varies over time The timings for the ping
command will occasionally vary when one or more of the network links is
saturated. A narrow range shows a consistent network profile, whereas a
wide range indicates network saturation or other issues that require further
investigation. Work with your networking team to determine whether, for
example, packet shaping, routing, or simply the amount of traffic is
creating a bottleneck for Oracle E-Business Suite traffic.
Comparing Small and Large Packet Behavior
Most networks do not usually have issues transferring small packets unless
saturated. However, large packets can be affected by a host of network issues and
can take significantly longer to transfer. The difference in the round trip times can
provide important information about the dynamics of a network.
Figure 6 compares the response times of 32-byte and 1472-byte packets.
Comparing the difference between the times to transfer small and large packets can
indicate network problems; this rudimentary comparison is frequently overlooked.


Figure 6: Comparing Network Latency for Different Packet Sizes
Oracle E-Business Suite Network Utilities: Best Practices Page 9

Comparing Network Test Form Latency to Actual Network Latency
Before looking more closely at methods to measure the actual network latency, it is
useful to note that information extrapolated from the Network Test Screen can be
enhanced by comparing the form latency measurement to the actual network latency.
When performance degrades, one of the following scenarios will help identify the
problem:
Comparing the form latency with the ping
times can help narrow down the location
of problems to the network, middle tier, or
other system components.
Form latency and network latency stay the same The network has
not changed in terms of latency (routing issues) or available bandwidth.
The responsiveness of the middle tier has not changed. It would appear
that the problem exists in one of the other system components.
Form latency increases but network latency stays the same The
network figures have not changed, but the middle tier is less responsive.
This indicates a problem in the middle tier (not the database server as it is
not involved in the synchronize command).
Form latency increases and network latency increases This could
indicate either a routing issue or a problem with a network device. It can
also point to problems with packet prioritization or possibly network
saturation issues. Recall that the Network Test form latency is the sum of
the network latency plus the time to process the synchronize command. The
difference between the form and network results will show whether the
latency shown on the form is a consequence of the increase in network
latency, or whether there is an additional element of increased load on the
middle tier that needs to be investigated.
The difference between the form and
network results can be used to isolate the
problem to the network or middle tier.
Form latency and network latency both decrease In this case, the
network congestion has improved, as has the responsiveness of the middle
tier. This implies that the problem must be caused by one of the other
system components.
CLIENT SYSTEM ANALYZER
The second tool available in Oracle E-Business Suite 11.5.10 and later is the Client
System Analyzer. It is intended to provide an inventory of the client PC and the
browser configuration, but also includes details about the network. The Client
System Analyzer collects client configuration data and writes it to the Web server,
where it is made available to both Oracle Applications Manager and Oracle
Enterprise Manager 10g Grid Control.
It is available to all users that have access to the Diagnostics menu, which is
controlled by the Hide Diagnostics Menu Entry profile option. The Client System
Analyzer function can also be attached to custom menus and responsibilities via
Oracle Application Object Library function security. Further information is
available in Oracle MetaLink Note 277904.1: Client System Analyzer Tasks for
Oracle E-Business Suite Users.
Oracle E-Business Suite Network Utilities: Best Practices Page 10

Figure 7 shows the analyzer being accessed from the Help:Diagnostics menu. A
sample screen showing the network information is shown in Figure 8.


Figure 7: Navigating to the Client System Analyzer


Figure 8: Client System Analyzer results
Oracle E-Business Suite Network Utilities: Best Practices Page 11

The techniques used by the Client System Analyzer to calculate latency and
bandwidth are different from those used by the Network Test Form. The latency
results obtained by running the Client System Analyzer in a simulated WAN
environment are shown in Figure 9. As with the Network Test form, the bandwidth
calculations used by the Client System Analyzer are also problematic.
The Client System Analyzer is readily
available, but you need to know the
available bandwidth on the link to be able
to interpret the results.


Figure 9: Client System Analyzer Latency results
The Client System Analyzer network test fails to give meaningful results with less
than 256Kbps available bandwidth, or for network links with greater than 400ms
latency. If either of these conditions exists, the client system analyzer will indicate
that the network test has failed prior to producing the results. Failed tests show a
bandwidth result of "-1".
CONCLUSION
The Network Test form and Client System Analyzer can provide information
essential to a holistic performance investigation. They can be easily understood and
employed by end-users to collect performance information when problems occur
therefore eliminating the need to deploy extensive monitoring tools at remote sites.
Furthermore, having users collect the information alleviates the requirement for
lengthy involvement by IT staff while they wait for transient problems to occur.
Comparing results from Oracle E-Business Suite utilities with those from standard
network utilities such as ping and traceroute can provide the basis for a powerful
technique that not only help create a composite picture, but can also help isolate a
problem to the network, middle tier, or other system component. These techniques
have been used to quickly establish the root cause of network problems during
many holistic performance investigations.
Comparing results from Oracle E-Business
Suite with standard network utilities is a
useful diagnostic technique.
Oracle E-Business Suite Network Utilities: Best Practices Page 12

APPENDIX A ADDITIONAL NETWORK TEST FORM RESULTS
Figure A1 expands the Network Test form latency results shown in Figure 4.
Unlike the results for a controlled environment (as shown in Figure 3), this chart
shows the results for a system that has insufficient memory on the middle tier,
which this accounts for the non-linearity of the results.



Figure A1: Forms Latency compared to Simulated Latency (variable load
test)




Oracle E-Business Suite Network Utilities: Best Practices Page 13

APPENDIX B PING, TRACEROUTE, AND THE MTU
This appendix describes how to mimic Oracle E-Business Suite traffic by using
standard operating system commands. The ping command is well known and
documented it sends packets to a destination server and provides information on
the time it takes for a network round trip. Provided a network administrator has
not blocked or reduced the priority of ICMP traffic (which you should confirm
before proceeding further), you can use ping to test the network from the client to
the server. Use both small and full size packets in order to mimic the Oracle E-
Business Suite network traffic.
Comparing Large and Small Packet Response Times To eliminate the need to have users type in
the requisite ping command each time, use
a shortcut to run a batch file. Alternatively,
use a more sophisticated graphical tool
such as Shunra Cloud Catcher (shown in
the next section).
The size of data packets is set by the Maximum Transmission Unit (MTU); the vast
majority of networks are set to 1500 bytes for IP traffic though Ethernet frame
headers can vary. Some networks use jumbo packets, which are generally 9,000
bytes. You can check the value with your network administrator, or use the
approach described in the MTU section of this appendix.
The following examples show how a user would use ping with differing packet sizes
from a DOS command prompt.
Example of Using Ping with a Small Packet
C:>ping 10.10.10.1
Pinging 10.10.10.1 with 32 bytes of data:
Reply from 10.10.10.1: bytes=32 time=227ms TTL=243
Reply from 10.10.10.1: bytes=32 time=192ms TTL=243
Reply from 10.10.10.1: bytes=32 time=193ms TTL=243
Reply from 10.10.10.1: bytes=32 time=191ms TTL=243
Ping statistics for 10.10.10.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 191ms, Maximum = 227ms, Average = 200ms
Example of Using Ping with a Large Packet
C:>ping -l 1472 10.10.10.1
Pinging 10.10.10.1 with 1472 bytes of data:
Reply from 10.10.10.1: bytes=1472 time=360ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=323ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=260ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=265ms TTL=243
Ping statistics for 10.10.10.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 260ms, Maximum = 360ms, Average = 302ms
Oracle E-Business Suite Network Utilities: Best Practices Page 14


Example Using Ping to Check Network Variance over Time
On Windows, the command ping -n 1000 <ip address> will send 1000 packets.
C:> ping -l 1472 -n 1000 10.10.10.1
Pinging 10.10.10.1 with 1472 bytes of data:
Reply from 10.10.10.1: bytes=1472 time=360ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=323ms TTL=243


Ping statistics for 10.10.10.1:
Packets: : Sent = 1000, Received = 1000, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 230ms, Maximum = 360ms, Average = 298ms
Using Shunra Cloud Catcher
Shunra Cloud Catcher (http://www.shunra.com/download) is a freeware
application that graphically displays the results of a series of ping tests. Figure B1
shows the Cloud Catcher in use; variations can be seen immediately obviating the
need to have user or network staff trawl through text files.


Figure B1: Shunra Cloud Catcher

Oracle E-Business Suite Network Utilities: Best Practices Page 15

Checking For Packet Fragmentation
Figure B2 shows an example where the MTU varies between hops. This is
important because all packets larger than the smallest MTU (on the path between
the client and middle tier server) will be fragmented into multiple packets. Having
to disassemble and reassemble packets reduces performance as well as increasing
the byte count that needs to be transferred because of the packet framing for the
additional packets.
Any difference between the client and middle tier will be negotiated prior to data
transfer; however, this does not always apply when, for example, the routing across
WAN or satellite links changes. This is easily overcome by predefining the packet
size and tuning TCP parameters at either end of the connection (client and middle
tier).

Figure B2: Understanding the MTU (used with permission)
You can determine the path MTU using the ping command, or use a program such
as mturoute (http://www.elifulkerson.com/projects/mturoute.php). You can also
test for this type of issue by using the ping f command (the exact syntax depends
on the platform) as this prevents packets from being sent if they have to be
fragmented to reach their destination.
It is important to note that when you use the ping command, the packets are
framed and 28 bytes are added by the network card before they are sent over the
network. Using ping with a packet size of 1472 bytes thereby creates a 1500 byte
packet on the wire, which is the standard MTU for IP networks.
Always check the physical location of each
hop to ensure optimal routing.
The following examples show how to use ping to establish the MTU size.
On Linux use:
ping -s 1472 -M do <server>. Other utilities
such as tracepath will also show the MTU.
Example Where Ping Matches the MTU
C:>ping -l 1472 -f 10.10.10.1
Pinging 10.10.10.1 with 1472 bytes of data:
Reply from 10.10.10.1: bytes=1472 time=330ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=268ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=252ms TTL=243
Reply from 10.10.10.1: bytes=1472 time=311ms TTL=243
Ping statistics for 10.10.10.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 252ms, Maximum = 330ms, Average = 290ms
Oracle E-Business Suite Network Utilities: Best Practices Page 16

Oracle E-Business Suite Network Utilities: Best Practices Page 17
Example: Ping Exceeds the MTU
C:>ping -l 1473 -f 10.10.10.1
Pinging 10.10.10.1 with 1473 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 10.10.10.1:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)
In this example DF means the Dont fragment bit is set on the ping packet.
Checking Network Routing
The traceroute tool (tracert in Windows) shows which routers and network devices
are traversed between the client and middle tier, and as such can be used to help
identify routing problems. As with ping, tracert may be blocked on your system for
security reasons, typically to prevent mapping tools from being used to build a
complete picture of your network.
If traceroute is available, users can collate the routing information from the client to
the middle tier and possibly even check for variations between tests. If the route
changes at the same time that they see poor performance, this indicates issues with
the load balancer, or more fundamental issues with the network routing.
One step, which is commonly missed, is to identify the physical location of each
hop. At one site there were about 4 hops, which looked reasonable until it was
found that one of the routers was located on the other side of the continent.
Always check the physical location of
each hop to ensure optimal routing.
Utilities that can be used when ICMP is Blocked
There are two main tools that can be used when ICMP traffic is blocked. Both
tcptraceroute (http://freshmeat.net/projects/tcptraceroute/) and tcpping
(http://www.kehlet.cx/articles/77.html) use TCP SYN instead of ICMP packets
and time how long it takes to receive the subsequent SYN/ACKs or RSTs. This
approach means that they can tunnel through many common firewall filters.
Unfortunately both of the following tools are UNIX utilities, which would generally
need to be run by system administrators or possibly deployed on a remote Linux
client







Oracle E-Business Suite Network Utilities: Best Practices
March 2008
Authors: Andy Tremayne and Robert Azzopardi
Editorial Review: Robert Farrington

Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.

Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com

Copyright 2008, Oracle. All rights reserved.
This document is provided for information purposes only and the
contents hereof are subject to change without notice.
This document is not warranted to be error-free, nor subject to any
other warranties or conditions, whether expressed orally or implied
in law, including implied warranties and conditions of merchantability
or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document and no contractual obligations
are formed either directly or indirectly by this document. This document
may not be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without our prior written permission.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.