JBoss Enterprise Application

Platform: Troubleshooting Guide

JBoss Enterprise Application Platform includes one of the leading Java EE
certified application server which is used by many fortune companies to run
their mission critical applications. This guide introduces some common tools
and commands which can be used to obtain diagnosis information and covers
some common troubleshooting tips related to applications running on JBoss
EAP.This Troubleshooting Guide is intended for people who are responsible for
maintaining,administering, or supporting JBoss Enterprise Application Platform
(EAP) environments.

Contents
1. Common Diagnostic Tools for Troubleshooting ..................2
1.1 Shell Commands for Linux/Unix Users.............................2
1.2 Tools in Java Standard Edition..............................................3
1.3 Tools available with JBoss EAP.............................................5
1.3.1 JMX-Console.........................................................................5
1.3.2 JBoss Operations Network.............................................5
2. Troubleshooting Systems bottleneck........................................6
2.1 CPU....................................................................................................6
2.1.1 Under utilization.................................................................6
2.1.2 High utilization....................................................................6
2.2 Disk IO.............................................................................................7
2.3 OutOfMemory Errors.................................................................8
3. Troubleshooting Application Server...........................................9
3.1 Troubleshooting Classloading..............................................9
3.2 JBoss AS Thread Pool................................................................9
3.3 Database Connection Pooling.............................................11
3.4 Clustering....................................................................................12
3.5 JBoss Messaging.......................................................................13
4. References...........................................................................................15
5. Feedback .............................................................................................15

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 1

1. Common Diagnostic Tools for Troubleshooting
In this section of the document we will cover some commands and tools which
are available as part of OS, JVM and Red Hat Subscription that can be very
handy in troubleshooting different problems.

1.1. Shell Commands for Linux/Unix users

There are many management and monitoring tools available for Unix/Linux
users as executable shell commands. top and vmstat are the two most useful
commands that can be used for troubleshooting some system level resource
contention and can provide us information that can help us in pinning down the
issue.

top provides a dynamic real-time view of a running system. It can provide

listing of most CPU intensive tasks and provides interactive interface for
manipulating processes.For example below is the output of top command

Figure 1: top Command output

The upper section describes the statistics about the machine, including Tasks,
CPU, Memory and Swaps, in real time. The lower section provides information
on per-process basis. For example you can see that there is one java process
with PID 6707 run by the user somil which is using 13.6 percent of the memory
and 0.3 percent of the CPU cycles

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 2

Another useful command is vmstat which provides virtual memory stats. It can
provide details about the paging activity, block I/O, memory etc which can be
very useful in troubleshooting some performance related issues. It is
recommended to use vmstat with delays of 5 to 10 seconds. For example
below is the output of vmstat command with 10 second delay:

Figure 2: vmstat Command output with 10 seconds delay

We will see later in the document how we can use these shell commands in
troubleshooting issues that we see with Java EE application deployed in JBoss
Enterprise Application Platform.

1.2. Tools in Java Standard Edition

The Java SE provides comprehensive monitoring and management support. It
not only defines the management interfaces for the Java virtual machine, but
also provides out-of-the-box remote monitoring and management on the Java
platform and of applications that run on it. Much of this functionality is also
available in JBoss Operations Network

JConsole: The JConsole graphical user interface is a monitoring tool that

complies to the Java Management Extensions (JMX) specification. JConsole uses
the extensive instrumentation of the Java Virtual Machine (Java VM) to provide
information about the performance and resource consumption of applications
running on the Java platform

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 3

 Figure 3: JConsole

JConsole can be used to observe information about an application running on the Java
platform.Some of the core monitoring and management functionalities provided by the
JConsole includes
• Detect low memory
• Enable or disable GC and class loading verbose tracing
• Deadlocks detection
• Control the log level in an application

Java VisualVM: It is an intuitive graphical user interface that provides detailed

information about Java technology-based applications (Java applications) while
they are running on a given Java Virtual Machine . It combines several
monitoring, troubleshooting, and profiling utilities into a single tool

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 4

 Figure 4: Java VisualVM

Java VisualVM is useful to Java application developers to troubleshoot

applications and to monitor and improve the applications' performance. Java
VisualVM can allow developers to generate and analyze heap dumps, track
down memory leaks, perform and monitor garbage collection, and perform
lightweight memory and CPU profiling.

1.3. Tools available with JBoss EAP

1.3.1. JMX-Console
JBoss EAP ships with jmx-console which can provide a raw view into the
microkernel of the JBoss application server. It lists all registered services
(MBeans) that are active in the application server and that can be accessed
either through the JMX console itself or programmatically from Java code.It can
be used to get a snapshot into how the app server is behaving in terms of
services deployed, resource utilization etc.

1.3.2. JBoss Operations Network

JBoss Operations Tool is a single integrated management and monitoring tool
for all JBoss Middleware. From the JBoss Operations Network (JBoss ON) console
you can inventory resources from the operating system to applications,Control
and audit your application configurations to standardize deployments and
Manage, monitor and tune your applications for improved visibility,
performance and availability.

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 5

2. Troubleshooting System Bottleneck
In this section we will see how we can use some of the tools and commands
from previous section to pin down the issues at the System level which may be
causing issues with applications deployed in JBoss EAP.
2.1.CPU
2.1.1. Under Utilization
If your CPU is idle for too long it may not be a good thing. You should be
suspicious if you have high CPU idle time along with
• No unusual input /output activity
• Independent of increased load
Response time degrades too rapidly with increased load
If you see any of these cases it may be possible that the application server is
waiting for some resources to be released by other processes. You can obtain
more information by getting Application Server ThreadDump by either using
jmx-console that comes with JBoss EAP or by using jvisulavm that ships with
oracle jdk

2.1.2. High Utilization

Generally its believed that if CPU utilization is high it may be due to some
system bottleneck. While in some cases it may be true but its not always the
case. For example below is the output from vmstat command

bash-4.0$ vmstat
procs -----------memory---------- ----swap--- ---io----system-----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 1700600 239956 753072 0 0 6 4 70 85 35 55 5 0 0

Figure 5: vmstat Command output showing high CPU Utilization

In this case CPU is busy (90% ) but it does not necessarily mean that there is a
system bottleneck. It may rather indicate that its running in optimal state.
One should be concerned if the run queue (r value under procs column) is more
that the number of CPUs on the machine. For example below is the out from
vmstat command

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 6

 bash-4.0$ vmstat
procs -----------memory---------- ----swap--- ---io----system-----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
50 0 1410500 244556 873170 0 0 7 3 60 75 35 60 1 4 0

Figure 6: vmstat Command output showing CPU constraint

In this case the CPU utilization is high (which may or may not be an issue), but
at the same time the value of r under the procs column is 5 which exceeds the
number of CPUs on the machine which confirms that there is CPU constraint .
You can further examine the cpu column to determine where its spending most
of its cycles. In this example CPU is spending 60% of its cycles in performing
system calls as reflected by sy value under cpu column. This may be because
the application may be executing lots of input/output , sockets or timestamp
creation. For example there may be a class thats opening socket for each
request, using a pooled approach may solve the problem, another example can
be a class performing lots of input/output , using buffered approach can solve
the problem.

In multi CPU machine there might be a case where one of the CPU is
experiencing high utilization. It may be caused by if single thread is used to
manage some resources. You can check the garbage collection, make sure its
configured properly and then you should verify if there is any contention for
some resources.

2.2. Disk IO
Generally reading and writing to a disk can be slow and may result in frequent
bottleneck for enterprise application. In Linux (RHEL/Fedora) you can use iostat
command to get input/output statistics. For example below is the output of
iostat command

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 7

 bash-4.0$ iostat -xd sda
Linux 2.6.30.10-105.2.23.fc11.i586 (localhost.sjain) 03/28/2011

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.13 0.86 0.46 0.29 11.10 9.21 26.97 0.03 39.58 2.20 0.17

Figure 7: iostat Command output

From the Disk IO troubleshooting perspective we will be interested in last two

column (for details on what other parameters mean in the output please refer to
man page of iostat). svctm provides the average service time ( in millisecond)
for I/O requests and %util provides the percentage of CPU time during which I/
O requests were issued to the device (bandwidth utilization for the device).
You should pay attention if you find excessive values for these parameters in
the output:
• High service time (svctm)
• High %util
A high value for these two indicates consistently high reads/writes operations.
Some possible causes for a Java Enterprise application are:
• Excessive logging
• Stateful Session Bean Passivation
• Poorly configured database cache.
If the bottleneck is not caused by the application, then you should consider
spreading the file system of the disk on to two or more disks. As an alternative,
move the file system to another faster disk/controller or replace the existing
disk/controller with a faster one.

2.3. OutOfMemory Errors

This section attempts to cover some of the most common reasons why you may
see an OutOfMemoryError from your JBoss application server. Interestingly,
there are several cases where the JVM may report an OutOfMemoryError even if
it is not really out of all of its available memory.

• Most modern Java virtual machines segments the memory heap into
generations. Your virtual machine may complain about lack of memory
when it has only exhausted one segment (a specific generation) from its
total maximum heap size. Also under some conditions on Linux/Unix
systems running out of some Operating System resources may yield an
OutOfMemoryError for example the inability for the OS to create more new
threads for the JVM.

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 8

 • It is possible to get an OutOfMemoryError when your Java virtual machine
really does run out of its maximum heap. There could be few reasons for
this, for example you may have a cache configuration that allows more
instances to be kept in memory than the JVM really can fit into its heap. Or
it may be simply that your JVM has been configured with a maximum heap
size too small to run all your application server services.
• An OOME is also thrown when the permanent memory is exhausted and
that is not part of the heap per se. That is a JVM specific area of memory
where information on loaded classes is maintained. If you have a mountain
of classes (e.g, a lot of EJBs and JSP pages) you can easily exhaust this area.
Oftentimes an application will fail to deploy or fail to redeploy. Is that case
increase your permanent memory space as follows to avoid OOME. The
default with the -server switch is 64 megabytes: 

set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx512m -XX:MaxPermSize=128m

You can set this in the run.bat (Windows) or run.sh (Linux) or run.conf

You can use VisualVM or JConsole to get in depth information on classes being
loaded and can narrow it to the classes causing memory leaks.

3. Troubleshooting JBoss Application Server

3.1. Troubleshooting AS Classloading
By default JBoss Application Server uses a flat class loading model that avoids
the need to redundantly include classes in different layers. While deploying your
application if you encounter errors like ClassCastException, IllegalAccessErrors,
VerifyErrors, it may be because you have applications that cannot share classes
because of version conflicts. In that case you need to isolate the classes from
other deployments.   There are two levels of scoping, isolation from other
deployments, and isolation that overrides the loading of JBoss server classes.
For more details on how to configure scoped class loading refer to the
“Classloading and You “ whitepaper in the Red Hat customer portal

3.2. JBoss AS Thread Model

The Java platform is designed to support concurrent programming. Application
servers like JBoss uses a pool of worker threads to improve the performance
( as thread creation and destruction can be expensive) and also provides a
control on the number of threads in the application. If you run into a situation
where server has a poor response time or does not responds or just starts
aborting the connection with RuntimeException it may be quite possible that

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 9

all your threads in the pool are busy. The error or exception you get is
determined by Thread Pool configuration in jboss-service.xml.

<mbean code="org.jboss.util.threadpool.BasicThreadPool"
name="jboss.system:service=ThreadPool">

<attribute name="Name">JBoss System Threads</attribute>

<attribute name="ThreadGroupName">System Threads</attribute>

<attribute name="KeepAliveTime">60000</attribute>

<attribute name="MaximumPoolSize">10</attribute>

<attribute name="MaximumQueueSize">1000</attribute>

<attribute name="BlockingMode">run</attribute>
</mbean>
Figure 8: Example of AS Thread Pool configuration

When all the thread in the pool are busy, all the new request are added to the
queue and queue size is determined by the QueueSize attribute. When the
queue has reached its max, any subsequent request behavior would depend on
the value of Blocking Mode attribute.

Figure 9:JMX-Console with Thread Pool Screen

Using JMX console you can check the size of the queue. If you have a steady (or
worse, an increasing) QueueSize on your server, then you should consider
raising the MaximumPoolSize pool size attribute.However, simply incrementing
the queue size might not be enough to solve your problems—analyze at first
where your application is slowing down. For example, a very common scenario
is that your threads are busy because they are handling slow or stuck JDBC

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 10

connections. Here, merely increasing the MaximumPoolSize would only move
the bottleneck into another area. In this scenario, you should first tune your
queries or increase the JDBC connection pool.

3.3. Database Connection Pooling

Connection Pooling: Database connection are quite expensive to create and
maintain because of the overhead of establishing a network connection and
initializing a database connection session in the back end database. The
database's ongoing management of all of its connection session can impose a
major limiting factor on the scalability of the application. Valuable database
resources such as locks, memory cursor, transaction logs, statement handles
and temporary tables all tend to increase based on the number of concurrent
connection sessions. Therefore it is very useful to use connection pool when
connecting to database. Database trace file can be used for troubleshooting any
issues related to database connection. It contains detailed information on the
interaction with the database and can help understand what might be going
wrong with your application. Refer to respective vendors drivers documentation
for examples on trace file configuration. Most drivers use the traceFile property
as follows:

/config/db/myappdb={
“traceFile” : “${myapp}/logs/db.txt”
}

Connection Verifiers: Depending on the nature of the application sometimes it

may be useful to check validity of the connection before its checked out of
connection pool by application ( if connection in the pool is invalid then any
database operation performed using that connection will result in error). JBoss
Connection Pool configuration allows few options that you can use to check the
validity of the connection. For example you can invoke SQL on connection
creation and / or pool access. Some database vendors provides exception
sorters. Oracle provides an API to ping Database. For more information on
various connection pooling parameters available in JBoss refer to DataSource
Parameters

Connection Leak: While Connection pooling has many advantages if not used
properly can result in many issues which may be difficult to troubleshoot. If
connections are not returned to the connection pool they will be kept open and
the available connection will leak away. At some point threads will either block
waiting for connection to be available or will generate exceptions perhaps after
few retries. There can be number of reason. For example:
• Developer forgets to return the connection

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 11

 • Possible deadlocks- there is a resource contention
• Exception being thrown in the code and connection close code being
skipped
These conditions can all be hard to find. One solution is to trace who is
acquiring and release locks. You can use JBoss Operations Network or log files
to get the info. There is an open-source JDBC profiler called P6Spy. It acts as a
proxy for your vendors JDBC driver, so it can log SQL execution times for
analyzing connection issues. If you are using Hibernate, JPA or EJB3 you can
enable the “show_sql” setting to view actual SQL generated.

3.4. Clustering
Clustering allows you to run an application on several parallel servers (a.k.a
cluster nodes) while providing a single view to application clients. Load is
distributed across different servers, and even if one or more of the servers fails,
the application is still accessible via the surviving cluster nodes. Clustering is
crucial for scalable enterprise applications, as you can improve performance by
adding more nodes to the cluster. Clustering is crucial for highly available
enterprise applications, as it is the clustering infrastructure that supports the
redundancy needed for high availability. JBoss Application Server uses JGroups
and JBoss Cache for providing Clustering capabilities. For more details refer to
the “Clustering considerations” whitepaper in the Red Hat customer portal.

Nodes do not form a cluster:

Make sure your machine is set up correctly for IP multicast. There are 2 test
programs that can be used to detect this: McastReceiverTest and
McastSenderTest. Go to the $JBOSS_HOME/server/all/lib directory and start
McastReceiverTest, for example:

$ java -cp jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10

-port 5555

Then in another window start McastSenderTest:

$ java -cp jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -

port 5555

If you want to bind to a specific network interface card (NIC), use -bind_addr
192.168.0.2, where 192.168.0.2 is the IP address of the NIC to which you want
to bind. Use this parameter in both the sender and the receiver.

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 12

You should be able to type in the McastSenderTest window and see the output
in the McastReceiverTest window. If not, try to use -ttl 32 in the sender. If this
still fails, consult a system administrator to help you setup IP multicast
correctly, and ask the admin to make sure that multicast will work on the
interface you have chosen or, if the machines have multiple interfaces, ask to
be told the correct interface. Once you know multicast is working properly on
each machine in your cluster, you can repeat the above test to test the network,
putting the sender on one machine and the receiver on another.

Causes of missing heartbeats in Failure Detection

Sometimes a member is suspected by Failure Detection Protocol because a
heartbeat ack has not been received for some time T (defined by timeout and
max_tries). This can have multiple reasons, e.g. in a cluster of A,B,C,D; C can be
suspected if (note that A pings B, B pings C, C pings D and D pings A):
B or C are running at 100% CPU for more than T seconds. So even if C sends a
heartbeat ack to B, B may not be able to process it because it is at 100%
B or C are garbage collecting, same as above. A combination of the 2 cases
above.
The network loses packets. This usually happens when there is a lot of traffic
on the network, and the switch starts dropping packets (usually broadcasts
first, then IP multicasts, TCP packets last).
B or C are processing a callback. Let's say C received a remote method call over
its channel and takes T+1 seconds to process it. During this time, C will not
process any other messages, including heartbeats, and therefore B will not
receive the heartbeat ack and will suspect C.

3.5. JBoss Messaging

Message on logs: AIO wasn't located on this platform. Possible causes are:
• Linux is not your platform
Just ignore the message as NIO will be selected automatically or change the
journal type to NIO on jbm-configuration.xml
• The JBoss Messaging JNI wrapper is not on library.path
Solution: Make sure you have libJBMLibAIO32.so or libJBMLibAIO64.so as
provided on the download package.
• libaio is not installed
Make sure you have libaio installed at your Linux distribution. This could
be done as yum install libaio1 on Fedora or apt-get install libaio1 on
Debian. (Refer to your manual to how to upgrade or install packages)
• libaio is very old
All the testing has been done using libaio 0.3.106 and Kernel 2.6. If you
have older versions you might need to upgrade or change the journal type

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 13

 to NIO on jbm-configuration.xml

Low response time even though the CPU and IO seems ok

Possible causes are:
• You are using short transactions in your system and you don't have
TCPNoDelay set
Make sure you have remoting-tcp-nodelay set to true on jbm-
configuration.xml
• The journal directory is a NFS
You shouldn't use the journal over a NFS. Make sure you have direct access
to the disk device.
• You are using Asynchronous IO on Linux and you don't have an ext3 or
ext2 file system

If using AIO, make sure the journal folder is on an ext2 or ext3 file system.

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 14

4. References
• top, vmstat, iostat : RHEL man pages

• JConsole: http://download.oracle.com/javase/6/docs/technotes/guides/
management/jconsole.html

• VisualVM: http://download.oracle.com/javase/6/docs/technotes/guides/
visualvm/index.html

• JMX-Console: http://community.jboss.org/wiki/JMXConsole

• JBoss Operations Network: http://docs.redhat.com/docs/en-US/

JBoss_Operations_Network/2.4/html-single/Basic_Admin_Guide/index.html

• Out of Memory Exceptions: http://community.jboss.org/wiki/

OutOfMemoryExceptions

• JBoss ClassLoading: http://docs.redhat.com/docs/en-US/

JBoss_Enterprise_Application_Platform/5/html/Installation_Guide/
migration.apps.html#migration.app.loader

• http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/
5/html-single/Administration_And_Configuration_Guide/
index.html#id3276717

• http://docs.jboss.org/jbossmessaging/docs/userguide-2.0.0.alpha1/html/
troubleshooting.html

5. Questions/Comments/Issues
If you have questions or comments about this whitepaper, please enter them in the Red
Hat customer portal for this specific whitepaper:

https://access.redhat.com/knowledge/techbriefs .

If you have a technical issue following this whitepaper please open a support case:

https://access.redhat.com/support/cases/new

JBoss Enterprise Application Platform : Troubleshooting Guide |Somil Jain| 15