Note: Net.Data logging and tracing support is available through the latest Net.

Data or
HTTP group PTFs. See the iSeries Net.Data Web site for updated information:
http://www.iseries.ibm.com/netdata

You should also refer to 7.3, “Net.Data: A ready-made scripting tool” on page 161.

13.2.5 HTTP server trace

An HTTP server trace provides additional information about server operations, from process
management to URI interpretation. Server traces can be activated from the GUI Manage
HTTP Servers display by starting the server with the lowercase -ve, -vi, and -vv startup
parameters. The Start TCP/IP Server (STRTCPSVR) CL command also supports these
startup options. You can collect the same data when the server is already active using the
Trace TCP/IP Application (TRCTCPAPP) and Dump User Trace (DMPUSRTRC) commands.
Be advised that this tracing facility does not support concurrent tracing of multiple HTTP
servers.

Note: The HTTP Server (powered by Apache) does not support the -vi, -ve, and -vv
switches on server restart. If you are unable to end all server jobs, use the Trace TCP/IP
Application (TRCTCPAPP) command instead (see Figure 13-17).

Trace TCP/IP Application (TRCTCPAPP)

Type choices, press Enter.

TCP/IP application . . . . . . . > *HTTP *FTP, *SMTPSVR, *SMTPCLT...

Trace option setting . . . . . . *ON *ON, *OFF, *END, *CHK
Maximum storage for trace . . . *APP 1-16000, *APP
Trace full action . . . . . . . *WRAP *WRAP, *STOPTRC
HTTP server instance . . . . . . > MYSERVER Character value
Trace level . . . . . . . . . . *ERROR *ERROR, *INFO, *VERBOSE

Bottom
F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display
F24=More keys

Figure 13-17 Trace TCP/IP Application (TRCTCPAPP): Turning on trace

The HTTP server trace can be set to operate at three different levels: error, information, and
verbose. User trace data for both parent and child helper jobs is automatically dumped as
soon as a failure condition is detected. The Dump User Trace (DMPUSRTRC) command is
otherwise used to direct trace output to the same database file while server jobs are still
active. Job name, number, and user profile for each one of your HTTP server jobs are
required. Trace output is dumped to file QAP0ZDMP in QTEMP in a member called
QP0Znnnnnn (where nnnnnn is the HTTP server job number you gave to the DMPUSRTRC
command).

Chapter 13. Problem determination: When things do not go as planned 341

 Table 13-4 shows the usage and purpose of this tracing facility.
Table 13-4 Service trace activation
Startup switch TRCTCPAPP Trace output

-ve *ERROR Server startup only. Nothing else is recorded unless an error
occurs.

-vi *INFO Server startup and initialization, including directive processing,

character conversion, and client request handling.

-vv *VERBOSE All the above plus application programming interface (API) and
module invocation, HTTP headers, error messages.

Here are three examples of how an HTTP server trace can be collected in different
environments:
򐂰 A test environment, an ideal situation in which you have complete control over the server
򐂰 A business-critical application, where the HTTP server is the core component of your On
Demand Business infrastructure and must be available at any time
򐂰 Somewhere in between, a third scenario that fits in between the two extremes

A test environment
You are testing a stand-alone HTTP Server (powered by Apache) configuration that is not
working as you expected. The server stopped and you are ready to collect an HTTP trace.
1. Start the HTTP server with the -ve, -vi, or -vv option. Look for completion message
CPCA984 (see Figure 13-18) in the server job log for confirmation that the trace option
you specified was accepted.

Additional Message Information

Message ID . . . . . . : CPCA984 Severity . . . . . . . : 00

Message type . . . . . : Completion
Date sent . . . . . . : 10/09/01 Time sent . . . . . . : 09:09:09

Message . . . . : User Trace option changed for job

032335/QTMHHTTP/ITSOSRV1.

Bottom
Press Enter to continue.

F3=Exit F6=Print F9=Display message details F12=Cancel

F21=Select assistance level

Figure 13-18 Additional Message Information: CPCA984 - A successful activation

2. Reproduce the failure. Skip the next step if the server jobs already ended.
3. Stop the HTTP server.

342 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
4. Use the Work with Spooled Files (WRKSPLF) CL command or iSeries Navigator
(Operations Navigator for V5R1) to retrieve QZSRHTTPTR spooled files for the
QTMHHTTP user profile (see Figure 13-19).

Work with All Spooled Files

Type options, press Enter.

1=Send 2=Change 3=Hold 4=Delete 5=Display 6=Release 7=Messages
8=Attributes 9=Work with printing status

Device or Total Cur

Opt File User Queue User Data Sts Pages Page Copy
QZSRHTTPTR QTMHHTTP PRT01 QSRV035027 HLD 28 1
QZSRHTTPTR QTMHHTTP PRT01 QSRV035029 HLD 29 1
QZSRHTTPTR QTMHHTTP PRT01 QSRV035028 HLD 44 1

Bottom
Parameters for options 1, 2, 3 or command
===> WRKSPLF SELECT(QTMHHTTP)
F17=Top F18=Bottom F21=Select assistance level F24=More keys

Figure 13-19 Work with All Spooled Files: Startup switch output

A business-critical application
The HTTP Server (powered by Apache) is a business-critical application running full time. It
cannot be stopped. You are experiencing occasional problems and need a server trace to
identify the source of your troubles.
1. Start the trace using the command:
TRCTCPAPP APP(*HTTP) SET(*ON) HTTPSVR(SERVERNAME) TRCLVL(*VERBOSE)
Select an appropriate level of information. Message CPC1129 (see Figure 13-21) is
posted to the server job log.
2. Reproduce the failure. Skip the next step if server jobs already ended.
3. Stop the trace with the command:
TRCTCPAPP APP(*HTTP) SET(*OFF)

Chapter 13. Problem determination: When things do not go as planned 343

 4. Use the Work with Spooled Files (WRKSPLF) CL command or iSeries Navigator to
retrieve the QZSRHTTPTR spooled files for the user profile you used for signon (see
Figure 13-20).

Work with All Spooled Files

Type options, press Enter.

1=Send 2=Change 3=Hold 4=Delete 5=Display 6=Release 7=Messages
8=Attributes 9=Work with printing status

Device or Total Cur

Opt File User Queue User Data Sts Pages Page Copy
QZSRHTTPTR GBANCHELLI PRT01 QSRV035077 HLD 1 1
QZSRHTTPTR GBANCHELLI PRT01 QSRV035078 HLD 1 1
QZSRHTTPTR GBANCHELLI PRT01 QSRV035076 HLD 1 1

Bottom
Parameters for options 1, 2, 3 or command
===>
F3=Exit F10=View 4 F11=View 2 F12=Cancel F22=Printers F24=More keys

Figure 13-20 TRCTCPAPP output

Somewhere in between
You are developing an intranet application, but you constantly run into an error. The server
cannot be stopped, but you are free to choose your debugging options.
1. Start the trace using the command:
TRCTCPAPP APP(*HTTP) SET(*ON) HTTPSVR(SERVERNAME) TRCLVL(*VERBOSE)
Select an appropriate level of information. Message CPC1129 (see Figure 13-21) is
posted to the server job logs.

Additional Message Information

Message ID . . . . . . : CPC1129 Severity . . . . . . . : 00

Message type . . . . . : Completion
Date sent . . . . . . : 10/11/01 Time sent . . . . . . : 17:30:11

Message . . . . : Job 035076/QTMHHTTP/GERONIMO changed by GBANCHELLI.

Bottom
Press Enter to continue.

F3=Exit F6=Print F9=Display message details F12=Cancel

F21=Select assistance level

Figure 13-21 CPC1129: The trace is active

2. Reproduce the error.

344 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 3. At this point, you want to examine the data collected so far, but you still need to keep
tracing server activity. Retrieve job numbers for your HTTP server jobs and feed them to
the DMPUSRTRC command. Message CPCA986 is posted to your job log (see
Figure 13-22).

Additional Message Information

Message ID . . . . . . : CPCA986 Severity . . . . . . . : 00

Message type . . . . . : Completion
Date sent . . . . . . : 10/11/01 Time sent . . . . . . : 17:31:43

Message . . . . : User Trace data for job 035076/QTMHHTTP/GERONIMO dumped to

member QP0Z035076 in file QAP0ZDMP in library QTEMP.
Cause . . . . . : The User Trace records associated with job
035076/QTMHHTTP/GERONIMO were successfully dumped to member QP0Z035076 in
file QAP0ZDMP in library QTEMP.

Bottom
Press Enter to continue.

F3=Exit F6=Print F9=Display message details F12=Cancel

F21=Select assistance level

Figure 13-22 CPCA986: Trace data has been dumped

4. Look for the file QAP0ZDMP in QTEMP. This file contains a QP0Znnnnnn member for
each one of the server job numbers (nnnnnn) you used in the DMPUSRTRC command.

Tips: Never forget to stop the trace with TRCTCPAPP APP(*HTTP) SET(*OFF) when it
is no longer needed. Also remember that you can access only the content of the
QTEMP library from your current session. It is discarded as soon as you sign off.

13.2.6 Collection Services performance data

Web-based transaction processing and Web-serving environments continue to grow in
importance and complexity. Your HTTP Server (powered by Apache) is the focal point of
many different kinds of On Demand Business environments including SSL, cache
accelerators such as FRCA, and WebSphere Application Server.

Tip: The iSeries HTTP Server (powered by Apache) does not support mod_status. This
simple module allows a Web administrator to take a picture of an Apache server and see
performance-related statistics that drill down to the work performed by each individual
thread.

mod_status was adjusted to work with the Apache 2.0 threaded server by the Apache
Software Foundation (ASF). However the fact that iSeries implements asynchronous
input/output (I/O) (see 10.2.1, “Threads and asynchronous I/O” on page 228) provides
complex challenges for implementing mod_status on the iSeries server.

Unique to the iSeries, HTTP server statistics are saved into collection services in V5R2.
The advantage on the iSeries server is that these reports can provide a more holistic view
of system performance. For example, it helps in situations where you may say, “Ah, I see
the reason that the HTTP Server (powered by Apache) is running so slow. That
programmer recompiled the entire LOB application again on the production server!”

Chapter 13. Problem determination: When things do not go as planned 345

 Beginning in V5R2, you can define your own performance collection categories with iSeries
Collection Services. The HTTP Server (powered by Apache) uses this new feature to
integrate performance data into Collection Services.

As shown Figure 13-23, the Standard plus protocol profile (the default used by Collection
Services) automatically collects HTTP Server (powered by Apache) if Collection Services
detects this application server is active on the system. As with all Collection Services
“collection object data”, the new statistics are placed into the following files via the currently
available iSeries Navigator “Create performance database files” function or the OS/400
Create Performance Data (CRTPFRDTA) command:
򐂰 QAPMHTTPB: Contains the basic data for HTTP Server (powered by Apache)
򐂰 QAPMHTTPD: Contains detailed data for HTTP Server (powered by Apache)

HTTP data collection category to contain HTTP performance data for Collection Services.
The HTTP performance data can then be queried to analyze HTTP server activity and better
understand what types of HTTP transactions are being processed by the iSeries (for
example, static files, CGI, or Java Servlets).

In addition, V5R2 Performance Tools for iSeries, 5722-PT1, has new sections in the System
and Component Reports for HTTP statistics.

Figure 13-23 Start Collection Services: Data to Collect page

The types of data collected are broken down into the following ways. Note that this is per
server job and the statistics are shown for each interval and request type within the interval.

346 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
򐂰 SR: Requests handled internally by the server itself. No program processing is necessary.
򐂰 SL: Requests of all types received via SSL. Reports activity that occurred over an SSL
connection even though that activity is also reported with other applicable request types.
򐂰 PX: Proxy requests.
򐂰 CG: CGI requests.
򐂰 WS: WebSphere requests.
򐂰 JV: IBM Java Servlet Engine requests.
򐂰 UM: Requests handled by user modules.
򐂰 FS: Static requests handled by FRCA.
򐂰 FX: Requests proxied by FRCA.

Starting Collection Services for the HTTP Server (powered by Apache)

We use iSeries Navigator to start and work with Collection Services. To start Collection
Services, as shown in Figure 13-23, follow these steps:
1. Log on to your iSeries server using with iSeries Navigator.
2. Expand your server →Configuration and Service.
3. Right-click Collection Services and select Start Collecting...
4. You see the Start Collection Services window (Figure 13-24). For the most part, you may
accept all the defaults.
Take note of the library to store collections since you need to know this later to find the
files. Also, we chose to set the Default collection interval for detailed data to 5 minutes.
5. Click OK to start Collection Services.

Chapter 13. Problem determination: When things do not go as planned 347

 Figure 13-24 Start Collection Services: General page

At this point, your HTTP Server (powered by Apache) server and related applications should
be up and running in a “steady state”. Collect the data for as long as you need.

When you are done collecting the data, stop Collection Services:
1. Right-click Collection Services and select Stop Collecting...
2. In the Stop Collection Services panel, click OK.

Performance Tools reports

Performance Tools for iSeries were enhanced in V5R2 to generate reports based on the
HTTP performance data. The reports contain information about the transactions processed
by HTTP server jobs. Follow these steps to see the System and Component reports for the
HTTP server:
1. From a 5250 session, enter the Start Performance Tools (STRPFRT) command.
2. Select option 3 (Print performance report).
3. Change the Library to QMPGDATA. Or, specify the library in which you saved the collection
data. Press Enter to refresh the list of collections.

348 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 4. In the Print Performance Report - Sample data display (Figure 13-25), type option 1
(System report) to the left of the collection data member. Use the Date and Time columns
to make sure you select the correct one.

Print Performance Report - Sample data

Library . . . . . . QMPGDATA

Type option, press Enter.

1=System report 2=Component report 3=Job report 4=Pool report
5=Resource report

Option Member Text Date Time

1 Q168151618 06/17/03 15:16:18
Q168125025 06/17/03 12:50:25
Q143095558 05/23/03 09:55:58

Figure 13-25 Performance Tools: Option 1 (System report)

5. In the Select Sections for Report display, press F6 to print the entire report.
6. In the Select Categories for Report display, press F6 to print the entire report.
7. In the Specify Report Options display, enter a meaningful report title.

Tip: We like to copy this System Report title so we can paste it to the Component
Report title later. This allows us to match these pairs of reports in the future.

8. Press Enter to submit this work to the batch queue.

9. In the Print Performance Report - Sample data display (Figure 13-25), type option 2
(Component report).
10.Repeat steps 5 through 8 for the Component report.

When the batch jobs finish, you should have two new spool files in OUTQ QPFROUTQ.
Figure 13-26 shows the output of the System Report.

Display Spooled File

File . . . . . : QPPTSYSR Page/Line 9/1
Control . . . . . +5 Columns 1 - 130
Find . . . . . .
*...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+....3
System Report 052303 10:22:0
HTTP Server Summary Page 000
Batch jobs
Member . . . : Q143095558 Model/Serial . : 270/10-4RT9M Main storage . . : 8000.0 MB Started . . . . : 05/23/03 09:55:5
Library . . : QMPGDATA System name . . : ASM27 Version/Release : 5/ 2.0 Stopped . . . . : 05/23/03 10:15:0
Partition ID : 000 Feature Code . : 22AB-2253-1520
Server Server job Server job Server start ------- Threads ------- -- Inbound Connections -- Requests Responses
name user number date/time Active Idle Non-SSL SSL received sent
---------- ---------- ---------- -------------- ---------- ---------- ------------ ------------ ----------- -----------
ADMIN QTMHHTTP 050851 05/22/03 11:04 0 40 0 0 0 0
HATSLEHTTP QTMHHTTP 051068 05/22/03 11:16 0 40 0 0 0 0
IAXHTTPEXA QTMHHTTP 051477 05/23/03 08:47 0 40 0 0 0 0
IAXHTTPEXB QTMHHTTP 051045 05/22/03 11:15 1 39 39 0 39 39
IAXHTTPSSL QTMHHTTP 051352 05/23/03 08:45 0 40 0 321 321 321
IWA QTMHHTTP 051022 05/22/03 11:13 0 40 0 0 0 0

Figure 13-26 Performance Tools: System Report - HTTP Server Summary

Chapter 13. Problem determination: When things do not go as planned 349

 Figure 13-27 shows the output of the Component Report for the HTTP server activity. This
example is for the same HTTP server - IAXHTTPSSL included in the System Report HTTP
statistics example. Here you can see that, of the 321 requests shown to be processed by this
server in the System Report example, 308 were processed during the 5 minute interval ended
at 10:00. Thirteen were processed during the 5 minute interval ended at 10:05. You can also
see the average “K bytes” (1024 bytes) sent each second was 5 KB and received was 1 KB
during the 10:00 interval.

Looking closely, you can tell via the SL Request Type (all requests handled under an SSL
connection) that all such requests were handled by some application running under a
WebSphere Application Server (WS Request Type). As defined on the next page, SL counts
also appear under another request type. You have to adjust to this accounting method to
know that actually 321 requests were received from browsers, not the 642 (2 x 321) shown as
total requests received and responses sent.

This example shows that no response was rejected or considered “in error”. High error or
reject values need to be investigated.

Note also the algorithm used when computing average K bytes per second. The Performance
Tools code knows that SL and WS values represent duplicates of each other. Using our 2
intervals example, we have 5K Bytes and 0K bytes per second transmitted averaged as 2K
bytes per second and 1K Bytes and 0K bytes per second received averaged as 0 K bytes per
second.

Component Report 5/23/03 10:22:0

HTTP Server Activity Page 2
Batch jobs
Member : Q143095558 Model/Serial . : 270/10-4RT9M Main storage . . : 8000.0 MB Started . . . . : 05/23/03 09:55:5
Library . . : QMPGDATA System name . . :ASM27 Version/Release : 5/ 2.0 Stopped . . . . : 05/23/03 10:15:0
Partition ID : 000 Feature Code . :22AB-2253-1520
Server : 051352/QTMHHTTP/IAXHTTPSSL
----------- Requests ---------- -------- Responses --------- KB KB
Itv Req Pct Pct Transmitted Received
End type Received Rejected Rejected Sent Error Error /Second /Second
----- ---- ----------- -------- -------- ----------- -------- ----- ----------- -----------
10:00 SL 308 0 .00 308 0 .00 5 1
10:00 WS 308 0 .00 308 0 .00 5 1
10:05 SL 13 0 .00 13 0 .00 0 0
10:05 WS 13 0 .00 13 0 .00 0 0
Column Total Average
------------------------------ ---------------- ----------------
Requests Received 642
Requests Rejected 0
Pct Requests Rejected .000
Responses Sent 642
Responses in error 0
Pct Responses in error .000
KB Transmitted/Second 2
KB Received/Second 0
Itv End -- Interval end time (hour and minute)
Req type -- The type of request being reported
Requests Received -- The number of requests received by the server
Requests Rejected -- The number of requests received that were not valid
Pct Requests Rejected -- Percentage of requests received that were not valid
Responses Sent -- The number of responses sent
Error Responses -- The number of responses in error
Pct Error Responses -- Percentage of responses in error
KB Transmitted/Second -- Number of kilobytes (1024 bytes) transmitted per second
KB Received/Second -- Number of kilobytes (1024 bytes) received per second

Figure 13-27 Performance Tools: Component Report - HTTP Server Activity

350 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 For more information
For further study of this topic, refer to the following resources:
򐂰 The iSeries Information Center has a starting point for performance-related topics that
include the logging of information with iSeries Collection Services:
http://publib.boulder.ibm.com/iseries/v5r2/ic2924/info/rzahx/rzahxebushttp.htm
򐂰 Two Redbooks deal with collection services at V5R1. They do not have specific
information about the HTTP Server (powered by Apache) information that is collected at
V5R2.
– Managing OS/400 with Operations Navigator V5R1 Volume 1: Overview and More,
SG24-6226
– Managing OS/400 with Operations Navigator V5R1 Volume 5: Performance
Management, SG24-6565

13.2.7 Other startup parameters

Server startup parameters can also aid in problem determination and in testing your server
configuration. In addition to the more debug-oriented -ve, -vi, -vv startup parameters
discussed in 13.2.5, “HTTP server trace” on page 341, the following options are available:

Tip: These parameters are case sensitive.

򐂰 -netccsid [nnn]: Overrides the DefaultNetCCSID directive.

򐂰 -fsccsid [nnn]: Overrides the default DefaultFsCCSID directive.
򐂰 -d [serverroot]: Sets the initial value for the ServerRoot variable to serverroot. This can
be overridden by the ServerRoot directive.
򐂰 -f [configuration]: Uses the values in the configuration variable on startup. If the
configuration does not begin with a /, then it is treated as a path relative to the ServerRoot.
For example, the following command advises your server to use the configuration file from
the fully qualified path that is specified:
STRTCPSVR SERVER(*HTTP) HTTPSVR(ITSONEW '-f /www/apachedft/conf/httpd.conf')
The following command loads the file from the [serverroot]/conf directory:
STRTCPSVR SERVER(*HTTP) HTTPSVR(ITSONEW '-f conf/httpd.conf')
򐂰 -C [directive]: Processes the given directive as though it was part of the configuration.
򐂰 -c [directive]: Processes the given directive after reading all the regular configuration
files.
򐂰 -V: Displays the base version of the server, build date, and a list of compile time settings to
your 5250 session. You may use the Page Up and Page Down keys to review the
information. Press Enter to quit the terminal session. The server is not started. Here is an
example printout:
Server version: Apache/2.0.43
Server built: Nov 26 2002 15:57:01
Server's Module Magic Number: 20020903:0
Architecture: 128-bit
Server compiled with....
-D APR_HAS_SENDFILE
-D NO_LINGCLOSE
-D APR_USE_FCNTL_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D APR_PROCESS_LOCK_IS_GLOBAL

Chapter 13. Problem determination: When things do not go as planned 351

 -D APR_HAS_OTHER_CHILD
-D APR_CHARSET_EBCDIC
-D APACHE_XLATE
-D HTTPD_ROOT="/QIBM/UserData/HTTPA"
-D AS400
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Press ENTER to end terminal session.
򐂰 -l: Displays a list of all modules compiled.
򐂰 -t: Tests the configuration file syntax but does not start the server. This command checks
to see if all DocumentRoot entries exist and are directories. The command STRTCPSVR
SERVER(*HTTP) HTTPSVR(ITSONEW '-t’) results in:
Syntax OK
Press ENTER to end terminal session.

13.2.8 HTTP status codes

In some cases, your HTTP server responds with standard three-digit codes, such as 404,
500, and so on to a client request. These are known as HTTP status codes. They often
provide additional information about the cause of a failure. Table 13-5 offers a quick guide on
HTTP status codes.
Table 13-5 HTTP status codes
Status Meaning Example
code group

1xx Informational. Contains a 101 Switching Protocols

provisional response that Sent when an updated version of the HTTP
influences the client’s behavior. protocol has been negotiated.

2xx Successful. Also returns 200 OK

information about the status of The client request is fulfilled. Can contain
negotiations and transactions. additional information in the reply to GET,
POST, HEAD, TRACE methods.

3xx Redirection. Requests an action 304 Not Modified

from the user agent. Often used in an answer to conditional
requests, usually when caching is involved.

4xx Client error. They are sent to your 404 Not Found
browser window when the page The server is unable to find anything matching
you requested cannot be served. the client request. Also used for security
reasons when the requesting user should not
have access to a particular resource.

5xx Server error. Sent to your browser 503 Service Unavailable

window when the server is unable The server has encountered a temporary
to fulfill your request because of condition that is preventing the fulfillment of a
an internal problem. client request.

For more information about HTTP status codes and their meaning, refer to Request for
Comments (RFC) 2616 on the HTTP 1.1 protocol standard.

352 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
13.2.9 Communications trace
A communications trace is also a powerful tool for problem determination. It is useful for
gathering information about the connection status and response time, and if you are
experiencing troubles with the encoding of your files.

Here is a quick example of using a communications trace to all the IP datagrams received and
sent from the iSeries point of view. The 5250 communication trace commands used are:
򐂰 Start Communications Trace (STRCMNTRC)
򐂰 End Communications Trace (ENDCMNTRC)
򐂰 Dump Communications Trace (DMPCMNTRC)
򐂰 Print Communications Trace (PRTCMNTRC)
򐂰 Delete Communications Trace (DLTCMNTRC)

This assumes that your configuration line object’s name is ETHLINE. Enter the commands
and perform the following actions in the order shown:
1. Enter the following command:
STRCMNTRC CFGOBJ(ETHLINE) CFGTYPE(*LIN) MAXSTG(256K) USRDTA(*MAX)
2. Start your HTTP Server (powered by Apache) and Web client and test your Web
application. Keep your work at a minimum to lessen the amount of data collected.
3. Enter the following command:
ENDCMNTRC CFGOBJ(ETHLINE) CFGTYPE(*LIN)
4. The next step is to output the trace information. You have two options:
– Print the trace in to a spooled file using the following command:
PRTCMNTRC CFGOBJ(ETHLINE) CFGTYPE(*LIN) CODE(*ASCII) FMTBCD(*NO)
This option prints the trace in MAC layer format. If you want to print, for example, a
trace in IP formatting for server port 80, you could enter the following command:
PRTCMNTRC CFGOBJ(ETHLINE) CFGTYPE(*LIN) CODE(*ASCII) FMTTCP(*YES) SLTPORT(80)
FMTBCD(*NO)

Note: The spooled file of the communications trace output contains on the right side
a so called eye-catcher. This is an area where all text in a trace is formatted in a
readable format. However, when using the PRTCMNTRC command with the options
as previously shown, all text, whether uppercase or lowercase, will be displayed in
the spooled file in uppercase. In certain situations, it is necessary to consider the
case. The following command flow creates a spooled file where case-sensitivity is
considered.

– The following commands create a spooled file where uppercase and lowercase
characters are displayed correctly in the eye-catcher area.
DMPCMNTRC CFGOBJ(ETHLINE) CFGTYPE(*LIN) TOSTMF('/barlen/trace01Oct04')
PRTCMNTRC FROMSTMF('/barlen/trace01Oct04') CODE(*ASCII) SLTPORT(80)
This example also formats the trace for IP and contains only data for port 80.
5. Enter the Work with Job (WRKJOB) command and select option 4 (Work with spooled files).
This is a fast way to find the spool file created by the PRTCMNTRC command.
6. Enter the following command:
DLTCMNTRC CFGOBJ(ETHLINE) CFGTYPE(*LIN)

For more information about how to take a communications trace, refer to IBM iSeries Support
Line Knowledge Base document TCP/IP Communications Trace Instructions, 23825849.

Chapter 13. Problem determination: When things do not go as planned 353

13.2.10 Additional resources
For more information, consult the following sources:
򐂰 RFC Index
http://www.rfc-editor.org/rfc.html
򐂰 iSeries Support Line Knowledge Base
http://www-912.ibm.com/s_dir/slkbase.nsf/slkbase

354 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 14

Chapter 14. High availability

If Web serving is a critical aspect of your business, you may want a highly available Web
server environment. Highly available HTTP servers take advantage of iSeries clustering
technology and make it possible to build a highly available Web site. This improves the
availability of business-critical Web applications built with static Hypertext Markup Language
(HTML) pages or Common Gateway Interface (CGI) programs. This enhancement is available
with both the HTTP Server (powered by Apache) in addition to the HTTP Server (original).

Highly available HTTP servers are not supported on versions prior to V5R1.

© Copyright IBM Corp. 2002, 2003, 2005. All rights reserved. 355
14.1 Highly available Web server cluster on the HTTP server
The Web server cluster solution can provide:
򐂰 Planned downtime: If a Web server requires planned maintenance, it is possible to transfer
the work to another node without visible service interruptions to the client.
򐂰 No unplanned downtime: If a machine fails, the work is transferred to another node with no
human involvement and without visible service interruptions to the client.
򐂰 Scalability: When employing multiple nodes, it is possible to distribute the Web site
workload over the cluster nodes.

Clusters are a collection of complete systems that work together to provide a single, unified
computing capability.

A liveness monitor checks the state of the Web server. It interacts with the Web server and
the clustering resource services in the event that a Web server fails (failover), or a manual
switchover takes place (ensures no interruption of Web server services).

You can use the clustered hash table (part of the state replication mechanism) to replicate
highly available CGI program state data across the cluster nodes. That way the state data is
available to all nodes in the event that a Web server fails (failover) or is switched-over
manually (switchover). To take advantage of this capability, an existing CGI program must be
enabled in a highly available Web server environment. CGI programs write to the CGI
application programming interfaces (APIs) to indicate what data is replicated. See HTTP
Server for iSeries Programming, GC41-5435.

Three Web server cluster models are supported:

򐂰 Primary or backup with takeover Internet Protocol (IP) model
򐂰 Primary or backup with a network dispatcher model
򐂰 Peer model

14.1.1 Primary or backup with takeover IP model

In this model, the Web server runs on the primary and all backup nodes. The backup node or
nodes are in an idle state, ready to become the primary Web server should the primary Web
server fail (failover) or a switchover takes place. All client requests are always served by the
primary node. Figure 14-1 illustrates a primary or backup with takeover IP model.

When the primary node fails (failover), or is brought down by the administrator, the
failover/switchover process begins. The following steps are performed during
failover/switchover:
1. One of the backup servers becomes the primary (the first backup in the switchover order).
2. Client requests are redirected to the new primary node. Assuming this client was not in the
process of running a persistent CGI application, the failover is completely transparent.

Tip: You can provide a high availability (HA) environment with two iSeries servers, each
with one instance of the HTTP Server (powered by Apache). Each of these instances
serves the identical (a copy) HTML and images and from identical (a copy) httpd.conf
configuration files. You can do this with a simple configuration. You do not need
third-party software. This is assuming, however, that you are not providing a HA
environment for your CGI application.

356 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
3. If the new primary receives a user request that belongs to a long-running-session (a CGI
program that is updated to be a highly available CGI program), the server restores the
request's state. The new primary retrieves that highly available CGI program's state
information from the clustered hash table. The clustered hash table is part of the state
replication mechanism.
Most non-HA CGI applications behave in the following manner:
a. The client clicks the Submit button to send a new request to the Web server and your
CGI application.
b. Your persistent CGI application “wakes up”. Its state information is saved in static
variables to determine what happened in the past with this client and what to do now.
Parameters found in the URL are parsed and actions are taken. New state information
is saved in the static variables for the next time this client returns to this iSeries server.
HTML code is generated and sent to standard out for presentation to the remote client.
c. The above “cycle” continues until the entire transaction is complete.
Most HA CGI applications behave in the following manner:
a. The client clicks the Submit button, which sends a new request to the Web server and
your CGI application.
b. Your persistent CGI application “wakes up”. Its state information is saved in the
clustered hash table. It reads from the clustered hash table and updates local variables
to determine what happened in the past with this client and what it must do now.
Parameters found in the URL are parsed and actions are taken. New state information
is written to the clustered hash table for the next time this same client returns to this (or
the backup) iSeries server. HA support ensures that the information written to the
clustered hash table on one iSeries server is replicated to the backup server. HTML
code is generated and sent to standard out for presentation to the remote client.
c. The above “cycle” continues until the entire transaction is complete.
4. After the failed node recovers, you can restart the highly available Web server instance,
which then becomes the backup system. If the system administrator wants the failed node
to become primary again, they must perform a manual switchover. They can accomplish
this with the IBM Simple Cluster Management interface available through iSeries
Navigator (Operations Navigator in V5R1), a 5250 CL interface, or a business partner tool.

Network

Client
iSeries Web
Server Cluster

Liveness Clustered Hash Liveness

Monitor Table Monitor

State Replication
Mechanism

Backup Primary

Figure 14-1 High availability: Primary or backup with takeover IP model

Chapter 14. High availability 357

 For an example, see 14.2, “A working primary or backup with takeover IP model” on
page 359.

14.1.2 Primary or backup with a network dispatcher model

In this model, as with the primary or backup with takeover IP model, the Web server runs on
the primary and all backup nodes. The backup nodes are in an idle state and all client
requests are served by the primary node. A network dispatcher, such as the IBM WebSphere
Edge Server, sends client requests to the Web server.

When the primary node fails (failover), or a switchover takes place, the failover/switchover
process begins. The following steps are performed during failover/switchover:
1. One of the backup servers becomes the primary (the first backup in the switchover order).
2. The client requests are sent to the new primary node by the network dispatcher.
3. If the new primary receives a user request that belongs to a long-running-session, the
server needs to restore the request's state. The new primary searches for the state either
locally or in the clustered hash table. The clustered hash table is part of the state
replication mechanism.
4. After the failed node recovers, the system administrator can restart the Web server
instance and it becomes a backup Web server. If the system administrator wants the failed
node to become primary again, they must perform a manual switchover.

Note: A node can join a recovery domain as a primary only if the Cluster Resource Group
(CRG) is in inactive mode.

Figure 14-2 illustrates a primary or backup with a network dispatcher model.

Network

Network Client
Dispatcher

Liveness Clustered Hash Liveness

Monitor Table Monitor

State Replication
Mechanism

Backup Primary

Figure 14-2 High availability: Primary or backup with a network dispatcher model

358 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
14.1.3 Peer model
In this model, there is no declared primary node. All nodes are in an active state and serve
client requests. A network dispatcher, such as IBM WebSphere Edge Server, evenly
distributes requests to different cluster nodes. This guarantees distribution of resources in
case of a heavy load. Linear scalability is not guaranteed beyond a small number of nodes.
After some nodes are added, scalability can disappear, and the cluster performance can
deteriorate.

Figure 14-3 illustrates the peer model.

Network

Network Client
Dispatcher

Liveness Clustered Hash Liveness

Monitor Table Monitor

State Replication
Mechanism

Server 1 Server 2
Figure 14-3 High availability: Peer model

In the event that one node fails (failover), the failed Web server traffic is routed to one of the
other operational Web servers according to the configuration of the network dispatcher.

14.2 A working primary or backup with takeover IP model

This scenario provides the steps that are necessary to get a simple HA environment up and
running between two iSeries servers. On each server is an identical, yet separate, instance of
the HTTP Server (powered by Apache).

14.2.1 Problem definition

For this scenario, let’s suppose that a customer has a network as presented in Figure 14-4.
They have two iSeries servers that are available for serving a standard Web application
across either a public or private network. This Web application involves the serving of
standard HTML and graphics only. The primary iSeries as23 may be down for any reason:
򐂰 Planned outages such as hardware or software maintenance
򐂰 Unplanned outages such as power loss, fire, and so on

They want the ability to seamlessly move all active clients to the backup server as24. And,
when the primary server as23 is brought back online, they want to move back seamlessly all
the clients.

Chapter 14. High availability 359

 Network

Client
iSeries Web
Server Cluster

Backup: Primary:
as24.itsoroch.ibm.com as23.itsoroch.ibm.com

Figure 14-4 Primary or backup with IP takeover: Problem definition

14.2.2 Solution definition

The solution is to take advantage of the Highly Available HTTP Server first introduced at
V5R1 for OS/400 Web applications. As shown in Figure 14-5, it takes advantage of the two
separate iSeries servers as23 and as24 to provide a primary or backup with IP takeover HA
solution.

Network

Cluster Information:
Cluster: ITSOTEST Client
Nodes: AS23 and AS24 iSeries Web
Cluster Resouce Group: PBABASIC00 Server Cluster

Liveness Clustered Hash Liveness

Monitor Table Monitor

State Replication
Mechanism

Backup: Primary:
iSeries host and domain: as24.itsoroch.ibm.com iSeries host and domain: as23.itsoroch.ibm.com
as24 'primary IP address': 10.5.92.38 as23 'primary IP address': 10.5.92.30
Clustered IP address: 10.5.92.51 Clustered IP address: 10.5.92.51

Figure 14-5 Primary or backup with IP takeover: Solution definition

14.2.3 Assumptions
The software and hardware used in this scenario has the following characteristics:
򐂰 We are using V5R2 of OS/400. This scenario can also be created for V5R1, but some of
the interfaces to iSeries clustering have changed.
򐂰 If you plan to use iSeries Navigator and the Simple Cluster Management interface to
configure the cluster, you must install 5722-SS1 OS/400 Option 41 (HA Switchable
Resources). OS/400 Option 41 is not necessary if you plan on using the 5250 commands
to configure your cluster.

360 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 򐂰 Both iSeries servers must have a routable IP address that is accessible from each other
server. This is shown in Figure 14-5 that as23 can PING 10.5.92.38 and as24 can PING
10.5.92.30.

Tip: Do not confuse these IP addresses on both iSeries servers with the clustered IP
address of 10.5.92.51. This clustered IP address is routable (that is, if a client on the
same subnet wants to communicate to this IP address, it uses ARP to resolve the MAC
address on the iSeries) but cannot be active on both systems at the same time.

14.2.4 How to
To configure the primary and backup servers in this scenario, perform the following tasks as
explained in the sections that follow:
򐂰 Step 1: Validate system and TCP/IP settings on both iSeries servers
򐂰 Step 2: Creating the HA cluster for ITSOTEST for nodes AS23 and AS24
򐂰 Step 3: Adding HA clustering directives to both httpd.conf configurations
򐂰 Step 4: Testing the primary and backup servers

Note: iSeries Navigator provides a simple graphical user interface (GUI) tool to manage a
cluster of iSeries servers. We chose to use a 5250 command entry in our configuration of
HA clustering on the iSeries at V5R2 because the CL commands that are provided offer
more features and choices not provided by the GUI. Be sure to make your own choice.

Step 1: Validate system and TCP/IP settings on both iSeries servers

Tip: In addition the steps outlined in this section, you must review the Information Center’s
resources for clustering:
http://publib.boulder.ibm.com/html/as400/infocenter.html

On this site, select your version and language, and click GO! Search for “clusters”. The
document that helps you the most in this step is entitled “Cluster configuration checklist”.

Before you start, you need to validate and possibly modify some of the system and TCP/IP
configuration settings on both iSeries servers. These instructions demonstrate what is
needed on both systems by showing you what we did on as23. In your own environment, you
must perform these steps for both the primary and backup servers.
1. Ensure that clustering is enabled for both iSeries servers:
a. Enter the Display Network Attributes (DSPNETA) command. On this Display Network
Attributes display, page down until you see the value set for the Allow add to cluster as
highlighted in Figure 14-6. This value should be set to *ANY on both iSeries servers.

Display Network Attributes

System: AS23
Allow add to cluster . . . . . . . . . . . . . . : *ANY
Modem country or region ID . . . . . . . . . . . :
Bottom
Press Enter to continue.

F3=Exit F12=Cancel

Figure 14-6 Primary or backup with IP takeover: Allow add to cluster should be *ANY

Chapter 14. High availability 361

 b. If necessary, you can change this value using the Change Network Attributes
(CHGNETA) command:
CHGNETA ALWADDCLU(*ANY)
2. Create the same routable IP address on both iSeries servers to be used as the clustered
IP address. This IP address is the “well-known” IP address at which your Web application
is bound to via the Listen directive. That is, a Domain Name System (DNS) server must
have an A record that maps the primary host’s name of as23.itsoroch.ibm.com to
10.5.92.51.

Tip: It may feel strange to create the same routable IP address on both iSeries. But, as
long as both are not active at the same time this is OK. It is the IP address takeover
feature of OS/400’s HA clustering that automatically allows only one of the iSeries
servers to have 10.5.92.51 active at one time. To be clear, you never manually make
10.5.92.51 active on either iSeries server. HA clustering’s IP takeover does this for you.

Available with V5R2 of OS/400 is a new feature called Virtual IP Address with proxy ARP
(VIPA with proxy Apache Portable Runtime (ARP)). This VIPA is routable in the same
subnet as the other IP addresses associated with the physical Network Interface Cards
(NIC) on your iSeries servers. VIPAs can provide extremely valuable fault tolerance for
situations where you have two or more NICs configured to all be in the same subnet on the
same iSeries. This scenario proceeds to create a clustered IP address on both iSeries as
VIPA with proxy ARP.
The added feature of fault tolerance (for a failure in a NIC) using the VIPA addresses is
detailed in iSeries IP Networks: Dynamic!, SG24-6718.
This scenario works equally well with new local area network (LAN) interfaces (rather than
the virtual IP addresses that we use) of 10.5.92.51 on both iSeries. These can be created
using 5250 commands such as Add TCP/IP Interface (ADDTCPIFC).
But, if you want HA for your Web server, you should follow up and implement VIPA with
proxy ARP as part of a total solution that includes fault tolerance for a failure in one of your
NICs.
a. Using iSeries Navigator create a VIPA with proxy ARP that will be the clustered IP
address on both iSeries servers. Start iSeries Navigator and connect to as23. Expand
Network →TCP/IP Configuration →IPv4.

Note: 5250 command entry cannot be used to create VIPA with proxy ARP in V5R2.
You must use iSeries Navigator.

b. Right-click Interfaces and select New Interface →Virtual IP.

c. Follow the New IPv4 Interface wizard to create the new VIPA interface. When
prompted, specify the following values:
• IP address: 10.5.92.51
• Description: VIPA with proxy ARP
• Subnet mask: 255.255.255.255
• Do you want to start this TCP/IP interface every time TCP/IP is started?: No
• Do you want to start this TCP/IP interface now?: No
d. Change the properties of this VIPA address to “turn on” the proxy ARP feature:
i. Right-click the IP address 10.5.92.51 and select Properties.
ii. Select the Advanced tab and click Enable proxy ARP.
iii. Click OK to save your changes.

362 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
Figure 14-7 Primary or backup with IP takeover: Changing the VIPA address to enable proxy ARP

3. Make sure that the loopback IP address 127.0.0.1 is configured and started on both
iSeries servers.
4. Make sure that the Internet Daemon (INETD) server is started on both iSeries servers:
a. Using iSeries Navigator, expand Network →Servers and then click TCP/IP. In the
panel on the right, a list of TCP/IP servers and their status (started or stopped) is
updated. Make sure the INETD server has the status of started.
b. If the INETD server has the status of stopped, right-click INETD and select Start.

Tip: Because INETD is needed for the proper operation of HA clustering on your
iSeries servers, we recommend that you change the properties of the INETD to
Start when TCP/IP is started.

5. Make sure that Liveness Monitor can run unimpeded in the QBATCH subsystem on both
iSeries servers. OS/400’s HA clustering uses a batch job to slow poll the primary server to
determine if this system is still available to the network. By default, this batch job is started
in QBATCH. You must ensure that the job queue for QBATCH is large enough to always
allow this job to run.
a. To check the QBATCH subsystem, use the 5250 command Display Subsystem
Description (DSPSBSD):
DSPSBSD SBSD(QBATCH)
Select option 6 (Job queue entries). As highlighted in Figure 14-8 on iSeries as23, the
maximum active jobs allowed in QBATCH is 4.

Display Job Queue Entries

System: AS23
Subsystem description: QBATCH Status: ACTIVE

Seq Job Max ---------Max by Priority----------

Nbr Queue Library Active 1 2 3 4 5 6 7 8 9
10 QBATCH QGPL 4 * * * * * * * * *
20 QS36EVOKE QGPL *NOMAX * * * * * * * * *
50 QTXTSRCH QGPL *NOMAX * * * * * * * * *

Figure 14-8 Primary or backup with IP takeover: Max Active for QBATCH showing 4

Chapter 14. High availability 363

 b. To ensure the testing will succeed, use the Change Job Queue Entry (CHGJOBQE)
command to change the maximum active to no maximum:
CHGJOBQE SBSD(QBATCH) JOBQ(QBATCH) MAXACT(*NOMAX)
6. Verify that the basic TCP/IP configuration is correct.
Test the interconnectivity of all the servers and clients to ensure that the Step 2 has the
best chance of succeeding. Refer to the network diagram in Figure 14-5 on page 360.
a. Verify that primary as23 can verify TCP/IP Connection (PING) backup as24 at IP
address 10.5.92.38.
b. Verify that backup as24 can PING primary as23 at IP address 10.5.92.30.
c. Conversely, make sure that a PING to clustered IP address 10.5.92.51 times out. That
is, this IP address should not be active on any host in the 10.5.92.0 subnetwork.
d. Verify that when the client PINGs the primary as23.itsoroch.ibm.com, the name is
resolved to 10.5.92.51. However, this PING should time out.

Step 2: Creating the HA cluster for ITSOTEST for nodes AS23 and AS24
You can perform the following steps on either iSeries server as23 or as24. For the purposes
of demonstration, we create the HA cluster ITSOTEST from as23.

Tip: The 5250 command GO CMDCLU provides a list of HA clustering commands that are
available.

1. On iSeries server as23, use the Create Cluster (CRTCLU) command to create the cluster
ITSOTEST and node AS23.
CRTCLU CLUSTER(ITSOTEST) NODE((AS23 ('10.5.92.30')))
The CRTCLU command should complete without errors.
2. To see the status of your newly created cluster ITSOTEST, enter the Display Cluster
Information (DSPCLUINF) command:
DSPCLUINF CLUSTER(ITSOTEST)
As shown in Figure 14-9, your cluster is created. Currently the cluster has one node. In our
case, this node is named AS23 and it has the active interface of 10.5.92.30.

Display Cluster Information

Cluster . . . . . . . . . . . . . : ITSOTEST
Consistent information in cluster : *YES
Current cluster version . . . . . : 3
Current cluster modification level : 0
Configuration tuning level . . . . : *NORMAL
Number of cluster nodes . . . . . : 1
Detail . . . . . . . . . . . . . . : *BASIC

Cluster Membership List

Potential
Node Mod Device
Node Status Vers Level Domain ------Interface Addresses
AS23 Active 3 0 *NONE 10.5.92.30

Figure 14-9 Primary or backup with IP takeover: Display cluster information showing one node AS23

364 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
3. Create the second cluster node AS24 by entering the Add Cluster Node Entry
(ADDCLUNODE) command (on iSeries server as23):
ADDCLUNODE CLUSTER(ITSOTEST) NODE(AS24 ('10.5.92.38'))
The ADDCLUNODE command should complete without errors.
4. To see the status of your newly created node AS24 in cluster ITSOTEST, enter the
DSPCLUINF command:
DSPCLUINF CLUSTER(ITSOTEST)
As shown in Figure 14-10, node AS24 is added to your cluster ITSOTEST.

Display Cluster Information

Cluster . . . . . . . . . . . . . : ITSOTEST
Consistent information in cluster : *YES
Current cluster version . . . . . : 3
Current cluster modification level : 0
Configuration tuning level . . . . : *NORMAL
Number of cluster nodes . . . . . : 2
Detail . . . . . . . . . . . . . . : *BASIC

Cluster Membership List

Potential
Node Mod Device
Node Status Vers Level Domain ------Interface Addresses
AS23 Active 3 0 *NONE 10.5.92.30
AS24 Active 3 0 *NONE 10.5.92.38

Figure 14-10 Primary or backup with IP takeover: Display cluster information two nodes: AS23, AS24

Step 3: Adding HA clustering directives to both httpd.conf

configurations
Now that you established a cluster between the two iSeries servers as23 and as24, it is time
to add to this cluster a CRG. You do this by creating a basic HTTP Server (powered by
Apache) configuration on iSeries server as23 and then adding the necessary HA server
directives. Then you make an identical copy of this HTTP Server (powered by Apache)
configuration and Web site on iSeries server as24.
1. Using the administration GUI create on iSeries server as23 an HTTP Server (powered by
Apache) server with the attributes shown in Table 14-1.

Tip: Make a special note of the server name and IP address.

Chapter 14. High availability 365

 Table 14-1 Primary or backup with IP takeover: Create New HTTP Server wizard required parameters
Create HTTP Server wizard parameter Value

Server name PBABASIC00

Note: This server name also becomes the
name of the CRG.

Server root /tcp52d00/basicConfig

Document root /tcp52d00/basicConfig/ITSOco

On which IP address and TCP/IP port do you want IP address: 10.5.92.51

your server to listen? Note: This is the address of the clustered IP
address shown in Figure 14-5 on page 360.
Port: 8000

Do you want your new server to use an access log? Yes

2. Add the HA server directives to the PBABASIC00 server on as23:

a. Select the Manage tab.
b. For Server, select PBABASIC00. For Server area, select Global configuration.
c. In the left pane, click System Resources.
d. Select the Highly Available Server tab.
e. Select Enable HTTP server to be highly available. This expands your options for this
tab as shown in Figure 14-11. Specify the following options:
i. Select Primary/backup with IP takeover.
ii. Deselect Enable highly available CGI programs.
iii. In the Liveness monitor settings area, enter the Liveness check URL:
http://10.5.92.51:8000/index.html
This URL is slow polled (as defined by the Time between liveness checks field
which is next) by the backup server to determine if the primary has failed. We
recommend that this page not be a complex dynamic page but rather one that is
simply HTML. Avoid HTML pages that include server-side includes (SSIs) too. Our
simple Web application’s index.html (home page) has static content only and makes
a good liveness test.
iv. For the remaining parameters, keep the defaults.
f. Click OK.

366 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
Figure 14-11 Primary or backup with IP takeover: Enabling HA for the PBABASIC00 server on as23

3. Display the httpd.conf configuration file to view three new directives that are added as
shown in Figure 14-12.

2 LoadModule ha_module /QSYS.LIB/QHTTPSVR.LIB/QZSRCORE.SRVPGM

...
19 HAModel PrimaryBackupWithIPTakeover
20 LmURLCheck http://10.5.92.51:8000/index.html
...

Figure 14-12 Primary or backup with IP takeover: Three new directives to support HA in PBABASIC00

4. Create an identical HTTP Server (powered by Apache) configuration and Web application
on your backup iSeries server as24. There are many ways to accomplish this. As a
high-level overview, we recommend this method:
a. Copy and paste the entire Web application’s HTML, GIFs, and other collateral from the
IFS on as23 to as24. Place these files in a similar directory structure.
b. Use the administrative GUI to create another PBABASIC00 HTTP Server (powered by
Apache) configuration on as24 using the same information found in Table 14-1.
c. Copy and paste the new HA directives from the PBABASIC00 httpd.conf configuration
file on as23 to as24.

Chapter 14. High availability 367

 Tip: In the end, your objective is to have the same Web application and configuration
file on both as23 and as24.

5. Since this is a test, we took the liberty to make some minor modifications to the index.html
(home page) on both the primary as23 and backup as24 iSeries servers. We added the
text “Welcome to Primary” on server as23 and “Welcome to Backup” on server as24. We
did this so the client Web browser could see the difference when it is automatically and
seamlessly switched from the primary to the backup iSeries server. Otherwise, it is difficult
to tell. You can see the differences in the index.html file on both servers in Figure 14-13
and Figure 14-16 on page 370.

Step 4: Testing the primary and backup servers

To test the HA server environment, start the PBABASIC00 server on as23 and then start
PBABASIC00 on as24.
1. Start the primary HTTP server PBABASIC00 on as23. Using the administrative GUI, make
sure select server PBABASIC00 on iSeries server as23 and then click Start. This starts
the HTTP Server (powered by Apache) PBABASIC00. This also automatically starts the
IP address 10.5.92.51 on iSeries server as23. The primary server is started first and the
backup server is started second.

Tip: The CRG is not started because the backup server is still inactive. However, your
Web site is up and running on the primary, you see in the next step.

2. Verify that your primary server is up and running. From a Web browser, enter the URL:
http://10.5.92.51:8000
You should see your home page as shown in Figure 14-13. The modification we made to
the index.html file to say, “Welcome to Primary” to distinguish this page from the similar
one on the backup server.

Figure 14-13 Primary or backup with IP takeover: Primary server on as23 up and running

368 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
3. Start the backup HTTP server PBABASIC00 on as24. Using the administrative GUI, make
sure you select server PBABASIC00 on iSeries server as24 and then click Start. This
starts the HTTP Server (powered by Apache) PBABASIC00.

Tip: This does not start the IP address 10.5.92.51 on iSeries server as24. Only during
a failure of the primary iSeries server as23 does as24 take over and make active this IP
address.

4. Now that the backup server is started, this automatically creates and starts the CRG
PBABASIC00. To list all the CRGs in cluster ITSOTEST, enter the Display CRG
Information (DSPCRGINF) command:
DSPCRGINF CLUSTER(ITSOTEST) CRG(*LIST)
As highlighted in Figure 14-14, our CRG PBABASIC00 is active on primary node AS23.

Tip: Use the following 5250 command for even greater detail about the CRG:
DSPCRGINF CLUSTER(ITSOTEST) CRG(PBABASIC00)

Display CRG Information

Cluster . . . . . . . . . . . . : ITSOTEST
Cluster Resource Group . . . . . : *LIST
Consistent Information in Cluster: *YES
Number of Cluster Resource Groups: 2

Cluster Resource Group List

Cluster Resource Group CRG Type Status Primary Node

PBABASIC00 Application Active AS23
PBABFL0ARB Data Inactive AS23

Figure 14-14 Primary or backup with IP takeover: CRG PBABASIC00 is active on primary node AS23

5. Use the Change CRG Primary (CHGCRGPRI) command to switch the clustered
application PBABASIC00 from the primary node AS23 to the backup node AS24:
CHGCRGPRI CLUSTER(ITSOTEST) CRG(PBABASIC00)
The CHGCRGPRI command should complete without errors.

Tip: One of the major effects of this command is that iSeries server as24 does an IP
takeover of the IP address 10.5.92.51. That is, after the CHGCRGPRI command has
completed, the IP address 10.5.92.51 is inactive on server as23 and active on server
as24.

Chapter 14. High availability 369

 6. Now node AS24 is the primary server for the CRG PBABASIC00. Enter the Display CRG
Information (DSPCRGINF) command:
DSPCRGINF CLUSTER(ITSOTEST) CRG(*LIST)
As highlighted in Figure 14-15, CRG PBABASIC00 is now active on (the new) primary
node AS24.

Display CRG Information

Cluster . . . . . . . . . . . . : ITSOTEST
Cluster Resource Group . . . . . : *LIST
Consistent Information in Cluster: *YES
Number of Cluster Resource Groups: 2

Cluster Resource Group List

Cluster Resource Group CRG Type Status Primary Node

PBABASIC00 Application Active AS24
PBABFL0ARB Data Inactive AS23

Figure 14-15 Primary or backup with IP takeover: CRG PBABASIC00 is active on primary node AS24

7. As a final confirmation, simply refresh the Web browser which still has
http://10.5.92.51:8000 for the URL. As shown in Figure 14-16, the Web client was
seamlessly moved to the backup HTTP server on iSeries as24. The only way you can
really tell from the client’s point of view is from the “Welcome to Backup” text we modified
in the index.html home page.

Figure 14-16 Primary or backup with IP takeover: Backup server on as24 ‘up and running’

The ability of the HTTP Server (powered by Apache) to take advantage of the iSeries HA
APIs is a powerful enhancement to the Apache server. This is another demonstration of the
integration that the Rochester lab has done to bring the Apache server to the iSeries server.

370 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
14.3 For more information
Highly available HTTP servers and iSeries clustering resources include:
򐂰 HTTP Server: What’s new, which includes information about the 17 December 2001
Highly Available HTTP Server announcement
http://www.ibm.com/servers/eserver/iseries/software/http/news/sitenews.html
򐂰 Documentation Center information about highly available Web servers
http://www-1.ibm.com/servers/eserver/iseries/software/http/docs/doc.htm
On this site, search for “high availability”.
򐂰 High Availability and Clusters home page
http://www.ibm.com/servers/eserver/iseries/ha/
򐂰 Clustering documentation in the iSeries Information Center
http://publib.boulder.ibm.com/html/as400/infocenter.html
On this site, select your version and language, and click GO! Then search for “clusters”.
򐂰 The IBM Redbook Clustering and IASPs for Higher Availability on the IBM Eserver
iSeries Server, SG24-5194

Chapter 14. High availability 371

372 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 15

Chapter 15. National language

considerations
This chapter discusses national language considerations on the iSeries server as they relate
to the HTTP Server (powered by Apache). It also discusses many associated applications
such as the Digital Certificate Manager (DCM) and the Cryptographic Service Provider, for
example. It provides information for you to view these applications in many of the world’s
languages.

Tip: It is interesting to note that some of the applications found on the iSeries Tasks page
have their graphical user interface (GUI) driven by Net.Data (for example the DCM). Others
have their GUI driven by servlet (for example the administration GUI for HTTP Server
(powered by Apache)). Because of this underlying difference, each handles national
language support (NLS) in a different manner!

© Copyright IBM Corp. 2002, 2003, 2005. All rights reserved. 373
15.1 Installing secondary languages
The GUI used to configure the HTTP Server (powered by Apache) can be installed in many
different languages. To use different languages, you must install the licensed product:
򐂰 5722-DG1: If you need different languages for the administration GUI
򐂰 5722-SS1, option *BASE and 3: For the initial iSeries TASK page
򐂰 5722-SS1, option 34: For the Digital Certificate Manager (DCM)
򐂰 5722-AC3: For the Cryptographic Service Provider
򐂰 5722-JV1 Developer Kit for Java

Before we can work with different languages, we have to check if the licensed program is
installed and if it is installed in the needed language. To check this, follow these steps:
1. Enter the OS/400 command GO LICPGM and press Page Down.
2. You should now see the Work with Licensed Programs display (Figure 15-1). From here
choose option 20 (Display installed secondary languages) to check if any secondary
languages are installed.

LICPGM Work with Licensed Programs

System: AS20
Select one of the following:

Secondary Languages
20. Display installed secondary languages
21. Install secondary languages
22. Delete secondary languages

Redistribution
40. Create distribution media
41. Work with installation profiles

Completion Status
50. Display log for messages

More...
Selection or command
===>

F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant

F16=AS/400 Main menu

Figure 15-1 Work with Licensed Programs: Page down

374 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers
 3. You now see the Display Installed Secondary Languages display (Figure 15-2). In our
case, 2924 (English) is the primary, and 2929 (German) is the secondary language. Enter
5 (Display installed Licensed Program) to see the installed licensed programs associated
with that secondary language.

Tip: You can find a national language feature code matching table on the Web at:
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/ic2924/index.htm?info/rzahc/rza
hcnlvfeaturecodes.htm

Display Installed Secondary Languages

System: AS20
Primary language . . . . . . : 2924
Description . . . . . . . . : English

Type options, press Enter.

5=Display installed Licensed Program

Option Language Description

5 2929 German

Figure 15-2 Display Installed Secondary Languages

4. In the Display Installed Secondary Language Licensed Programs display (Figure 15-3),
verify the Installed Status of the products. If any of these is in status *ERROR, try to
re-install the product or contact your local service representative.

Display Installed Secondary Language Licensed Programs

System: AS20
Secondary language . . . . . : 2929
Description . . . . . . . . : German

Licensed Installed
Program Status Description
5722SS1 *COMPATIBLE OS/400 - Digital Certificate Manager
5722DG1 *COMPATIBLE IBM HTTP-Server
5722DG1 *COMPATIBLE Triggered Cache Manager

Figure 15-3 Display Installed Secondary Languages: Option 5 (LPPs)

If no secondary language is installed on your system, refer to iSeries Information Center or to

Chapter 10 in Software Installation V5R2, SC41-5120.

15.2 Net.Data based: iSeries Tasks page and DCM

Both, the iSeries Tasks page and the DCM are implemented as a Net.Data application.
Net.Data based GUIs use iSeries host settings to determine which language they should
present to the client. Therefore, you must check the language settings in the user profile that
you used to sign on to the iSeries Tasks page, to display the correct language. As shown in
Figure 15-4, use the OS/400 command Display User Profile (DSPUSRPRF)
USRPRF(WOLFGANGP) to check the settings for your OS/400 user profile.

Chapter 15. National language considerations 375

 Display User Profile - Basic

User profile . . . . . . . . . . . . . . . : WOLFGANGP

Previous sign-on . . . . . . . . . . . . . : 05/07/03 10:11:16

Sign-on attempts not valid . . . . . . . . : 0
Status . . . . . . . . . . . . . . . . . . : *ENABLED
Date password last changed . . . . . . . . : 04/29/03
Password expiration interval . . . . . . . : *SYSVAL
Date password expires . . . . . . . . . : 11/01/03
Set password to expired . . . . . . . . . : *NO
.
.
.
Language identifier. . . . . . . . . . . . : *SYSVAL

Figure 15-4 Display User Profile: Language identifier

If the language ID is set to *SYSVAL (which is the default), the OS/400 system value
QLANGID is used to identify the desired language. To display the system value, use the
OS/400 command Display System Value (DSPSYSVAL SYSVAL(QLANGID)). You should
see the Display System Value display as shown in Figure 15-5.

Display System Value

System value . . . . . : QLANGID

Description . . . . . : Language identifier

Language identifier . : ENU Language abbreviation

Figure 15-5 Display System Value: QLANGID

In our case, this display shows ENU which means US - English. If you have some users that
need to work with other languages (for example German), you can change the Language
identifier parameter to DEU. The next time the user connects to any site pages served by
Net.Data on this iSeries server, they see them in German.

Tip: It is important to learn that if the Web pages are served by Net.Data, the iSeries
server examines the language settings of your browser. It only uses the information found
in either your OS/400 user profile or the default system value. For more information about
using the browser’s language settings to influence the national language chosen by the
iSeries, see the following section.

15.3 Servlet based: Administration GUI

The administrative GUI can be served in any language that is installed on your iSeries server.

Tip: You have to restart the Admin instance after installing the secondary languages to
take advantage of the change.

376 IBM HTTP Server (powered by Apache): An Integrated Solution for IBM Eserver iSeries Servers