Application Guide Volume VII AG2017-01

Maximizing RTAC Performance

Rick Bryson

INTRODUCTION
The SEL Real-Time Automation Controller (RTAC) family of products combines a powerful
IEC 61131 logic engine with industry-standard protocols to perform tasks such as data
concentration, front-end processing, custom automation schemes, human-machine visualization,
and remote control applications. The RTAC family includes a variety of form-factor devices
which provide the flexibility to automate almost any application, from recloser or automated
switch controls in poll top applications, to a complete data acquisition, automation, and control
system in a substation or processing plant.
This application guide provides SEL RTAC application ideas that you can use to greatly enhance
the performance of any system. By understanding some basic concepts in greater depth, you can
make more informed decisions on how to configure your RTAC project and maximize RTAC
performance. We divide our discussion into the following five areas:
1. Project Execution: Overall operation of the RTAC system, including project execution
time, process execution order, and common issues that affect overall system performance.
2. Intelligent Electronic Device (IED) Communication: Client and server
communication, including general protocol details that can affect performance, and
settings you should consider during configuration and system planning.
3. Background Process Execution: Background processes, as used in this document, refer
to those processes that either run in the background or are part of a project but not
directly configurable by the user. These processes can have a significant effect on web
responsiveness and CPU usage on the RTAC.
4. Analysis Tools: Tools included in the ACSELERATOR RTAC® SEL-5033 Software that
provide you with details regarding communication and RTAC project performance.
5. The RTAC Platform: There are many form factors in the SEL RTAC family and various
amounts of memory and processing power. Choosing the right RTAC product for your
particular project ensures you achieve the best performance.

PROJECT EXECUTION

Task Cycle
An RTAC is an open automation platform with no preset or purpose-built processes pre-
programmed in the system. You can define and configure tasks and processes as a group (referred
to as a “project”), such as custom user logic, IED polling, and reporting to a supervisory control
and data acquisition (SCADA) system. A task manages the execution of each of these processes,
also called Program Organizational Units (POUs). All POUs run within a task. For comparison,

Date Code 20170105 SEL Application Guide 2017-01

2

you can think of a task as a main program and each POU as a subtask called by the main
program. Within a project there are two default tasks:
1. Main Task: By default, it contains all user-configured tag lists and POUs, such as
protocols, logic, etc.
2. Automation Task: By default, it remains blank, but you are permitted to assign POUs
and tag lists to this task.
Each task executes on a configurable cycle, referred to as the “task cycle time.” By default, the
Main Task is configured to execute every 100 ms. Unless you are using the Automation Task
cycle, you should leave the configured Automation Task cycle time at the default value of
1000 ms. The Automation Task is a higher priority task that is designed for a separate, faster
processing cycle, if needed.
For example, let us assume we have a requirement to process received MIRRORED BITS®
communications in IEC 61131 user logic every 8 ms, and we also have a requirement to process
IED and SCADA communications every several seconds. To meet the 8 ms requirement, we can
speed up the Main Task cycle time, but that would create an unnecessary burden on the RTAC
system. A more efficient solution would be to move the user logic into the Automation Task
cycle, configure this task to execute every 8 ms, and leave the SCADA communication in the
Main Task cycle at a configured task cycle time of 1000 ms.
Configure task cycle times in the Main Controller Options window, as shown in Figure 1. The
configurable task cycle time allows you to establish deterministic operation of processes within
the RTAC.

Figure 1 Main Controller Options Window

As Figure 1 shows, all configured processes (or POUs) in this RTAC project are placed in the
Main Task by default. Also by default, the RTAC is set to execute the Main Task every 100 ms.
When the task runs, it executes each process assigned to that task, such as the SEL_351S_1_DNP
client in Figure 1. The task executes the processes in the order in which they appear in the Main
Controller Options window under the selected task tab (Main or Automation, in this example).

SEL Application Guide 2017-01 Date Code 20170105

 3

Task Cycle Overruns

If the task executes all processes assigned to it in less time than the configured cycle time, the
task effectively goes dormant until the cycle time expires, at which time it again executes all
processes assigned to it. On the other hand, if the task does not execute all processes within the
configured cycle time, the task will continue to execute all assigned processes until completion.
The task waits until the next scheduled task cycle time to begin execution again.
For example, consider a scenario where all POUs assigned to a task require 150 ms to complete,
but the user-configured task cycle time for that task is 100 ms. The task executes all assigned
processes in 150 ms, then the task goes dormant for 50 ms as it waits for the beginning of the next
processing cycle. In this scenario, the task takes 1.5 task cycles to complete all the POUs assigned
to it, but it still ensures it executes all processes. The effective result is the same as if the RTAC
were configured for a 200 ms processing cycle. As discussed later in Task Timers on page 15,
this is not an efficient way to configure a project because there will be little dormant time for
other background actions. Furthermore, the RTAC project is not taking advantage of the
deterministic nature of running tasks at a configure cycle. In this example, we recommend
configuring the task cycle time to at least 210 ms (150 ms • 140 percent).

Process Execution Order

As mentioned, each process assigned to a task executes in the order listed in the Controller
Options window, as shown in Figure 2. The order can be very important when executing a group
of processes within a task.
ACSELERATOR RTAC places each POU in the process list as you add POUs to the RTAC project.
By default, the Tag Processor process is last in the process list. Because the Tag Processor is
generally used to map input values to output tags, such as those going to an HMI or SCADA, it is
preferable for all tags to be gathered and processed before they are mapped.
A situation where you may need to change the default order of processes is if you wanted to
obtain all inputs from IEDs before executing user logic. To do this, clear the Use default order
check box, click and highlight the process you want to move, and click the Move Up or Move
Down button to shift the order of the process.

Figure 2 Example Process Execution Order

As another example, assume we have a project with the following intended control scheme logic
flow:
1. Check for Commands From SCADA: The RTAC receives set-point or digital
commands from SCADA.
2. Read IED Data: The RTAC polls local IEDs for operational data such as voltage,
current, or binary indications.
3. Execute User Logic: The user logic evaluates input from SCADA and from IEDs to
determine the required output operations.

Date Code 20170105 SEL Application Guide 2017-01

4

4. Tag Processor Mapping: The RTAC maps output values to output IED tags in the Tag
Processor.
5. Send Outputs: A DNP3 client sends the output values to connected IEDs.
See Figure 3 for a diagram of a possible process order. Note that the actual order of the processes
does not match the desired logic flow in the previous numbered list. Based on this less-than-
optimal process order, we predict that the user logic will not reach the correct output conclusions
until several task-cycle executions to collect, evaluate, and move all the required data. In this
example, you should reorder the processes to align with the intended control scheme logic flow,
listed in the previous numbered list (1 through 5), to achieve optimum processing.

Execute User Logic

Send Outputs

Task
Read IED Data
Cycle

Check for Commands From SCADA

Tag Processor Mapping

Figure 3 Less-Than-Optimal Process Order

User Logic Execution Order

As with process execution order, the order within user logic can be very important as well. User
logic is contained in programs, functions, and function blocks. Programs are automatically
executed every task cycle in the order they appear in the Main Controller Options tab, so you do
not need to (and generally should not) make a call to a program from another program. Calling a
program from within another program will execute the called program once because it was called
and again as part of the process list within the task. Functions and function blocks, however, only
run when called from within a program, function, or function block. The order of function and
function block calls and other code execution is easy to discern in Ladder Diagram (LD) and
Structured Text (ST) because it follows the physical layout of the code. Continuous Function
Chart (CFC), however, is by nature a graphical language with no special boundaries or order. The
RTAC executes CFC code in the order indicated by the small green number on each block, shown
in Figure 4.

SEL Application Guide 2017-01 Date Code 20170105

 5

Figure 4 CFC Execution Order

Note that this code will not provide the results you want in a single processing cycle. The
Boolean system_armed is actually the last to be set based on the AND of main_on and
backup_on, as the execution order numbering in the small green boxes shows in Figure 4. The
logical flow of your CFC program may be acceptable if you enter the program and do not make
any changes, however, as Figure 4 shows, if you copy and paste, add, or delete code, the flow
order can be less-than-optimal.
Right-click on any of the green boxes and select Set Execution Order to manually modify the
execution order number of each block. Typically, selecting Order by Data Flow or Order by
Topology resolves most ordering issues introduced when you entered the code in the editor. We
recommend that you always verify the correct execution order of your logic by examining the
small green execution order blocks and running validation testing before system commissioning.

Tag and Variable Assignment Evaluation

An assignment is a user-configured statement, whether occurring in the Tag Processor or in
IEC 61131 user logic, which moves a value from one tag or variable to another. Assignments in a
project execute in the order in which they appear within IEC 61131 logic and within the Tag
Processor.
When the logic processor executes an assignment statement in user logic or in the Tag Processor,
it immediately places the assigned value into the target tag or variable. From that point on, any
reference in the project to that assigned tag uses the newly assigned value. However, processes
such as protocols and physical output drivers do not immediately detect each change that occurs
in the logic engine. The important implication of this is that multiple assignments to a tag do not
necessarily result in multiple immediate actions. The logic engine first executes all POUs within a
task and then the protocols and output processes use the resulting values. For example, Lines 4
through 10 in Figure 5 illustrate a user attempting to send a CLOSE command via DNP3 to a
regulator controller to raise or lower the tap position, depending on the present value of Phase A
volts.

Date Code 20170105 SEL Application Guide 2017-01

6

Figure 5 Incorrect Tag Assignment

Note that in Lines 13 and 14, the user clears the Raise and Lower operClose tags in an attempt to
prevent resending the command in the next processing cycle. The user anticipates that the DNP3
driver will immediately send the correct Raise or Lower operClose command after the
assignments are executed in Lines 4 through10, and then the logic clears the Raise and Lower
operClose commands directly following the operation. However, because the RTAC logic engine
delivers the tag values to the DNP3 driver at the end of each processing cycle, and because
Lines 13 and 14 clear the Raise and Lower operClose commands, the DNP3 driver never sends a
Raise or Lower operClose command to the SEL-751 Relay.
It is also important to be aware that processes in the RTAC evaluate changes based on a rising-
edge trigger and that, upon detection, they typically perform their operations for one processing
cycle. For example, setting a Boolean variable (or similar tag value) to TRUE multiple times
within a processing cycle, or in successive processing cycles, does not necessarily produce
multiple triggers unless the variable or tag changes to FALSE in between the assignments to
TRUE. Because the logic engine uses a rising-edge trigger on the set and on the clear attributes of
those tags that contain an operSet, you should assign TRUE to operSet, FALSE to operClear,
and vice versa (see Figure 6).

Figure 6 Correct Setting and Clearing of Remote Bits

IED COMMUNICATION
IED communication is the configured protocol interaction between IEDs and SCADA; it includes
event collection, polling, and unsolicited data reporting. Although most protocols have unique
characteristics, there are common configurable behaviors that can greatly affect communication
and processing performance.

SEL Application Guide 2017-01 Date Code 20170105

 7

Event Collection
When a relay detects an event, such as an overcurrent condition indicating a fault, the relay can
create a record with fault information and waveform data that provides a record of power system
conditions before, during, and after the fault. Event records are quite large compared to normal
SCADA data and there are often multiple event records to report when an event occurs.
Collecting event records is desirable and a necessary part of operation, but collecting event
records may place a significant burden on the communications channel. When designing a
communications scheme on a channel that includes event collection, plan for potential delays in
other ASCII data collection when large numbers of event records are collected.

Polling and Data Reporting

The communications configuration between RTAC and IED is as important to system
performance as the number of communicating IEDs. The following are some key factors for IED
data communication in an RTAC:
• Polling rate
• Tag quantity
• Change detection and reporting
• Processing cycle

Polling Rate
Polling is the process of an RTAC requesting data from a connected IED and the IED reporting
the data to the RTAC through a supported protocol. Polling is also the process of a SCADA
master requesting data from an RTAC and the RTAC reporting that information. Polling rate is
the frequency with which the client (the device asking for information) requests the data.
Most protocols request data at a configurable rate. It may seem that the highest possible polling
frequency is the best communications configuration, but you must consider several factors when
configuring polling to ensure the best performance of the overall system.

Factors
• IED Response Time. Every device, no matter how powerful, is limited by processing
power, system burden, and the physics of how quickly it receives, processes, and replies
to a request. In addition, most IEDs are not strictly communications devices. For
example, meters, relays, device monitors, and other dedicated electronic devices, though
they have the ability to communicate, have a primary, higher-priority task other than
communication. Polling at a rate faster than the IED can reliably reply causes non-
response conditions which usually trigger poll retries. Retries make this condition worse
by causing more non-response conditions, lost data, and offline indications on the RTAC.
• Data Transmission Speed. Every data channel transfers data at a limited rate. When
configuring poll rates and poll timeout periods, you must consider how long it takes to
physically transmit the maximum number of bits in a message. Polling for data faster
than that data can physically move across a wire can result in polling errors, data delays,
and overall slower updates.

Date Code 20170105 SEL Application Guide 2017-01

8

• IED Data Update Rate. If an IED updates analog values once per second, then polling at
a rate faster than once per second wastes bandwidth and increases the burden on the
RTAC, the IED, and potentially on other equipment on the same communications
channel.
• Data Processing Rate. Similar to IED data update rate, the polling device (SCADA,
RTAC, etc.) has a specified processing rate. If a client device polls an IED faster than the
client device can process the data, then not only is there no benefit, but the extraneous
polls further burden the communications network, the polled IEDs, and the client.
In summary, configuring a device such as an RTAC to poll IEDs too quickly causes sporadic or
continuous offline conditions on the communications channel, delayed data updates, and delays in
control operations. You should configure polling at a rate that meets your required data update
rate, but no faster.

Example
Configure an RTAC with a processing cycle of 150 ms, with 10 clients, all multi-dropped on an
EIA-485 loop, and all communicating using the Modbus® protocol. Configure each Modbus IED
client instance in the RTAC project to poll 200 registers from each IED every 100 ms at 9600 bps
with a 100 ms timeout. Consider the following issues in this example:
• Issue 1. During each task cycle, the RTAC evaluates the timers in each protocol POU to
determine if enough time has elapsed to warrant queuing a poll. Polls are scheduled every
100 ms, but the timer is only evaluated at each task interval, which in this case is every
150 ms. As a result, the most frequent that polls can be sent is every 150 ms.
• Issue 2. Every 100 ms, every poll for every configured IED in this project is supposed to
be queued to transmit. Issue 1 shows that polling will actually occur every 150 ms.
However, because the IEDs are multi-dropped, only one IED can be polled at a time. The
RTAC polls one IED, waits for a reply, and then polls the next one. This implies that, if
every poll were successful, and if we do not consider reply times or other delays, then
each IED would reply, at most, once every 1.5 seconds (150 ms • 10 IEDs = 1500 ms). In
any case, the value of the POU pin labeled Buffer Overflow will increase on each IED
instance in the RTAC project because the POU attempts to queue polls faster than they
can be sent out and processed.
• Issue 3. Even if only one IED were configured in this manner, it would be impossible for
the client to send the poll to the IED, the IED to receive and process the poll, the IED to
reply to the poll, and then for the IED to transmit more than 3200 bits (200 registers •
16 bits/register + 16 bits CRC + header) back to the client every 150 ms. When the
RTAC does not detect a reply within the configured 100 ms timeout, it marks the poll as
failed and will retry a configured number of times. As the retries are performed at the
same speed as the original poll, they too will fail, and the RTAC will mark the channel as
offline. This causes the channel to go into Slow Poll mode, which means the Slow Poll
Multiplier is multiplied by the poll rate. The slower poll rate may result in some
successful polls, and the status of the channel may momentarily change to online.
Immediately, however, the process will repeat itself.

SEL Application Guide 2017-01 Date Code 20170105

 9

Tag Quantity
Polling for more data than is needed wastes bandwidth and processing capacity on the client and
on the server, and it can introduce delays in control operations. For example, if an IED has 500
analog values and you only need to see 35 of them, you can configure the RTAC to only poll for
the needed 35 points, or you can configure the IED to only deliver the 35 points, depending on
the protocol you are using. Conversely, if you are using a protocol such as Modbus that polls for
contiguous lists of registers, it may actually be more efficient to configure a single poll that does
include some unwanted data rather than configure multiple polls to retrieve a few registers each.
However, if there are large gaps in the register ranges that you want to poll, it is better to
configure one poll to access one contiguous group of registers and another poll to access a second
contiguous group of registers to avoid an excessively large single poll that may exceed protocol
or timing restrictions. Figure 7 gives an example of using multiple polls for polling three sublists
of a Modbus map rather than configuring one very large, monolithic poll.

Figure 7 Polling Sublists

Change Detection and Reporting

One of the most common sources of unexpected communications issues is change data reporting,
also referred to as event or class reporting. In protocols such as Modbus, the client always polls
for and receives the same amount of data for each polling cycle. In protocols such as DNP3,
which support change-detect data, the client can be configured to poll for change data on a fast,
cyclic period, such as every second. On a more extended poll period, usually configured in
minutes or greater, the client sends another poll for change data, as well as sending a poll for the
present values or static data. The former (fast) polling method is referred to as event polling (or,
in DNP3, a Class 1, 2, 3 poll); the latter (extended) polling method is referred to as integrity
polling (or, in DNP3, a Class 1, 2, 3, 0 poll). Other protocols that support a similar polling
scheme have different poll types, but the concept is the same.
During commissioning and initial testing, the system may be very stable and the change poll
replies quite small. The user may configure polling in a manner that achieves maximum
performance based on the small amount of data sent on a stable system. However, when there are
extensive system changes, such as those caused by fault conditions, then change poll replies can
increase to a degree that results in overall system communications delays and degradation. The
greatest contributor to these wide swings in the size of change poll replies is the value of analog
deadbands.

Date Code 20170105 SEL Application Guide 2017-01

10

Deadbanding
Under real-world conditions, change data can cause the change poll response to grow so large that
it will not fit in one response message. Binary change data can cause varying response message
lengths, though typically binary changes are not so great as to cause communications issues.
Analog changes, however, are determined based on a configured deadband value. The deadband
value can be a set quantity of counts, a percent of change, or another change detection method,
depending on the protocol and implementation. In basic terms, an analog value that changes in
relation to a deadband value is considered significant by the protocol driver, insomuch that the
protocol creates an event containing that value. When the IED is queried for any changes, it
replies with all events, binary and analog, from the time the last query was processed. The analog
event contains the time at which that analog value excursion beyond the deadband occurred, as
well as the analog value at the time of the excursion. Setting the deadband value to 0, or to a very
low number, results in excessive analog change values queuing to the next poll reply.
Consider an IED with 100 analog points, each with a deadband value of 0. If each analog value
stays the same, the change request poll will result in 100 analog values. For each change in value,
the IED creates another event. For example, if the Phase A current increases by a value of one,
and if it does so 15 times between the last change poll and the present change poll, then the IED
will reply with all 100 analogs, as well as 15 change events, for the one analog that changed.
Typically, most analogs change by at least one count between polls, so the change poll response
from the IED can rapidly grow to an enormous number and completely block all other
communications on that channel.
Note that the polling system does not typically use analog event values. Usually, SCADA or
communications gateways only display the latest present value. Unless the polling system is
archiving the analog events, only the latest change is used; the others are disregarded. If the
polling system uses the analog values in a logic scheme, it is not usually desirable to process all
change data from the time of the last poll, but rather to only apply the latest value to the logic
algorithm. Excessive analog changes, and the resulting large poll responses, often cause
communications timeouts, channels toggling between online and offline, delayed data gathering,
and delays in control operations.
Consider the source of the analog point to determine the likelihood of rapid changes when
configuring an analog deadband. Above all, consider what data your system needs and why you
are using analog deadbands. For example, setting a deadband to a value of 1 on a line voltage
may be excessive because changes this small may be of no consequence to the system. Any
changes in frequency, however, may be of great consequence to the system and should be
reported immediately. In some cases, a user may want to receive immediate notice of any binary
changes and a steady update of all static analog values. As we observed, setting analog deadbands
at small values will not necessarily result in immediate retrieval of the values because the amount
of change data returned may be so great as to cause update delays or even cause the channel to go
offline.
The following steps are the preferred method of obtaining binary event change data, as well as all
the latest static analog values for an IED (using DNP3 as an example):
Step 1. Configure each analog deadband value in the IED at a number that is large enough
so that it is not exceeded.
Step 2. Configure the client to poll the IED with a Class 1, 2, 3 poll every second.
Step 3. Configure the client to poll the IED with a custom analog input poll (static data
only) at the desired interval.

SEL Application Guide 2017-01 Date Code 20170105

 11

Unsolicited Reporting
Excessive traffic from events is not limited to poll and reply scenarios. Consider protocols that
support unsolicited messaging, such as DNP3. The IED reports changes to the RTAC without
being polled and only when something significant occurs. This may seem efficient during
commissioning, but consider that when an event occurs which is significant enough to affect
values on one IED, that it will also likely affect many values on many other IEDs. Furthermore, if
these IEDs are on a multi-drop network, a storm of unsolicited responses could cause message
collisions as all of the devices try to communicate at the same time. Even with collision
avoidance algorithms, the multiple iterations of retries and collisions could introduce significant
delays and the potential for loss of event data.
If, instead, the client polls each IED in-turn, then we can avoid communications collisions and
gather all data deterministically. Placing each client on a separate communications channel also
reduces the potential for communications collisions.
The IEC 61850 GOOSE protocol is a good example of an unsolicited protocol that can sometimes
be efficient but can also cause excessive communications issues. For example, GOOSE sends out
a multicast message when a significant change of data occurs. It does not get a reply, but rather it
continues to send the same message at intervals until the timeout expires or GOOSE must report
another change. This way, GOOSE is more likely to ensure that the recipient receives the
message. When a significant event occurs, however, IEDs using GOOSE messaging can start a
storm of network traffic that actually hampers the ability for GOOSE message subscribers to
receive messages in a timely manner. This situation can also cause a loss of event data because
packets are dropped by attached network devices. Correct message triggers (such as analog
deadbands) that limit data to what is needed, as well as appropriate GOOSE message
configuration, are all critical when using GOOSE protocol.
Network Global Variable Lists (NGVLs) are user-defined lists of variables that the RTAC sends
to predefined broadcast addresses. By this means, the RTAC can communicate data to another
RTAC or group of RTACs over Ethernet on a cyclic basis or triggered by data change. As
previously discussed with reference to GOOSE, NGVL data traffic may be acceptable during
testing, but when real-world events trigger the sending of many NGVL messages, the RTAC can
flood the network, causing communications and CPU usage issues. As with any protocol, do not
configure NGVLs to transmit messages more frequently than they can be processed by the target
devices. For example, if you configure RTAC A to transmit NGVL messages to RTAC B on a
cyclic basis, ensure the transmit cycle is not faster than the configured task cycle time on
RTAC B. Otherwise, you will cause needless network bandwidth usage as well as potential buffer
overflow issues on RTAC B.

BACKGROUND PROCESS EXECUTION

EtherCAT
EtherCAT®1 is a protocol used to transfer information between SEL Axion® I/O modules and an
RTAC (e.g., an SEL-2240). Although EtherCAT runs within the process cycle, its execution is
not configurable by the user beyond the configuration of individual I/O modules. The EtherCAT
train communicates with all I/O modules at a given rate, regardless of the user-configured task
cycle time. Because of this high rate of message processing, it is possible, therefore, that a project

1
EtherCAT is a registered trademark and patented technology, licensed by Beckhoff Automation
GmbH, Germany.

Date Code 20170105 SEL Application Guide 2017-01

12

executes well within the configured task cycle, but that CPU usage on the RTAC appears quite
high (i.e., near or at 100 percent).
Take, for example, an Axion project configured with only one 16 AI card, a few small programs,
and a small HMI configured at a 1000 ms task cycle time. The total task cycle time is 3.7 ms,
which is too small to appear as a percent of task cycle usage on the Dashboard, as Figure 8 shows.

Figure 8 Dashboard Task Usage

If we look at the CPU_Burden_Percent in SystemTags while online with the project, however,
we can see that the CPU burden value spikes as high as 30 percent. In this instance, the CPU is
busy processing EtherCAT traffic, even though the project itself is not busy.

Web Interface and Optional HMI

All RTACs have a web interface that provides tools for managing settings for individual RTACs.
Settings include user accounts, Lightweight Directory Access Protocol (LDAP) authentication,
Ethernet interface configuration, certificate creation, and others. In addition, the web interface has
several display features that provide functions such as real-time data monitoring, sequence-of-
events (SOE) viewing, and collected power system event viewing. Also, most SEL RTAC
products offer an optional, user-configurable HMI to provide a configurable, visual representation
of data, including single-line diagrams, annunciation, controls, and other standard HMI functions.
The RTAC uses background tasks to communicate data between the web browser and the real-
time engine.

Web Usage and CPU Burden

Although there are no settings to directly control web traffic, there are project and HMI-related
settings that affect the quantity of data and frequency of updates. The number of tags configured
in the Tag Processor for Live Data viewing, for example, will proportionally increase traffic to
the web interface. The data traffic between the RTAC and web browser can put a significant
burden on the PC CPU, network traffic, and RTAC CPU, which may introduce some lag in the
web interface response. To observe the CPU burden percentage of an online RTAC project, click
System Tags > Tags > CPU Burden. To observe the effect that the web interface has on CPU
burden, open a web browser, enter the RTAC IP address, and log in. You may observe that the
CPU burden can spike as high as 100 percent.
CPU burden is usually not an issue; it only means that the CPU is busy while, in this example, the
background web task communicates with the web browser. Because web interface
communication is a low-priority task, it does not interfere with execution of user-configurable
processes in an RTAC project. As mentioned previously, the RTAC ensures that the Main and
Automation Tasks complete execution of all processes. In fact, as the CPU burden rises, the lower
priority tasks, such as the web interface, are given less execution time to ensure that the POUs
defined in the project have priority.
As mentioned, if the RTAC project is large enough to use most or all of the configured task cycle
time, the web interface demonstrates a noticeable lag in responsiveness and higher CPU usage.
You should configure your RTAC project to keep CPU burden below 70 percent, which provides
the best web interface performance. To decrease CPU usage in your project, we recommend
optimizing polling traffic and increasing the configured task cycle time to at least 140 percent of
the SystemTotal_task_timer, discussed in Task Timers on page 15. As mentioned previously,

SEL Application Guide 2017-01 Date Code 20170105

 13

EtherCAT drives CPU burden higher than normal. Because you cannot adjust EtherCAT usage,
you should focus on task cycle time, poll rates, number of IEDs, and number of tags in your
RTAC project to reduce CPU usage. Generally, the quantity of tags configured with Live Data
enabled in the Tag Processor is the only user setting that affects traffic on the RTAC web
interface. As expected, viewing SOE logs or event data in the Event Collection window also adds
traffic burden while the pages populate their respective web interface views.

HMI Traffic and CPU Burden

CPU usage can also spike when you are viewing the optional RTAC HMI. The HMI uses a
background JavaScript Object Notation (JSON) task to transfer human-readable text from the
RTAC to the web browser for tag updates. You can configure the cyclic poll interval of one or
more JSON queries (or Request Groups) in the HMI project in the ACSELERATOR Diagram
Builder™ SEL-5035 Software. Select Settings > Project Settings > Request Groups to add and
configure request groups. In the Properties tab for each control on your diagram, you can select
the tag for the control and identify the Request Group in which the tag is polled. Assign tags that
require more frequent updating to one Request Group and configure the poll interval as
necessary. Assign tags that require less frequent updating to a different Request Group, and
configure the poll interval as needed. If you leave all tags in the default Request Group, the HMI
polls all tags at the same time, which may result in slower updates overall (depending on
bandwidth and RTAC burden). Trend controls in the HMI also have a configurable poll request
interval, which you define in the trend control. In the same way that polling an IED more
frequently than required wastes bandwidth and resources, setting HMI poll request intervals to the
minimum value may create enough extra burden that it slows down overall updates in the HMI.
The best approach is to poll as frequently as required, but not more so.
Not only does the tag poll interval affect the performance of an RTAC HMI project, but affecting
HMI responsiveness to an even greater degree are the quantity of tags being updated and the
number and size of configured diagrams. Embedding graphics in your HMI diagrams gives you
almost limitless design options, however, it also increases the size, and thus the load time, of
diagrams. For example, a high-resolution device image (> 300 ppi) may display well on the HMI,
but it increases the overall display burden without adding much more visual benefit compared to
a low-resolution picture. Furthermore, showing more detail in an image may actually distract the
user from the most important information on the screen.
Additionally, one of the inherent limitations of bitmap images is that they show less quality as
you zoom in closer on the image. You can minimize this loss of detail by using a higher
resolution image, but this also increases the overall size of the diagram, resulting in a longer
initial load time. For imported graphics, therefore, we recommend using lower resolution images
for reference purposes in order to enhance HMI performance, even at the expense of providing
less detail. If you require high-resolution detail, however, it is generally better to draw images in
Diagram Builder. Diagram Builder images are built using vector graphics, which can scale to any
size but are stored as graphical formulas rather than bitmaps. The image byte size is thus
relatively small, which allows for faster load times. Furthermore, because it is a vector graphic,
zooming in and out does not affect visual image quality.

Date Code 20170105 SEL Application Guide 2017-01

14

Indirect Tagging
In some HMI projects, you may have many similar diagrams that display data for many different
devices. For example, consider a group of buildings on a college campus, each building with an
electric meter and each meter with an HMI diagram. Tags from each meter in the RTAC project
are used directly in each corresponding component of the diagram and are then repeated for each
building. The result is the following:
• Number of Diagrams = (number of meters)
• Number of HMI tags = (number of meters) • (number of tags per meter)
The following is an example of this type of RTAC HMI project:
• Buildings, each with a single meter = 50
• Tags per meter = 110
• HMI polling interval = 2 seconds (default)
• Number of individual HMI diagrams = 50
• Number of individual imported meter images as PNG files = 50
The result is an initial HMI load burden of 50 screens with 50 imported images and a tag update
of 5500 values every 2 seconds. Most likely, this amount of burden will cause a lag in
performance.
Alternatively, there is a method referred to as “shared tags” or “indirect tagging” that can greatly
improve HMI responsiveness. You can create a single diagram that represents one building.
Using the example RTAC project above, follow these steps:
Step 1. Create a single set of virtual tags in the RTAC project that represents all the
required tags from one meter. In this example, there would be 110 virtual tags.
Step 2. Map the virtual tags to the components on the single diagram instead of mapping
them directly to a particular meter.
Step 3. Create a binary control for each meter or building to provide the user a means of
selecting which meter values populate on the HMI.
Step 4. Program conditional mapping in the RTAC to populate the virtual tag list with
values from Meter 1, Meter 2, etc., as the specified by the user in the HMI.
The following illustrates the improvement of the RTAC HMI project using indirect tagging:
• Buildings, each with single meter = 50
• Tags per meter = 110
• HMI polling interval = 2 seconds (default)
• Number of individual HMI diagrams = 1
• Number of individual imported meter images as PNG files = 1
The result is an initial HMI load burden of 1 screen with 1 imported image and a tag update of
110 values every 2 seconds. The result is a vast improvement over the traditional HMI project,
with faster initial HMI load timers, faster tag updates, better screen responsiveness, and less CPU
burden on the RTAC.
Figure 9 exemplifies selectively mapping meter tags to virtual tags. You can select the control tag
global_building1 from the HMI. The Tag Processor assigns tags from the SEL_735_1 meter to

SEL Application Guide 2017-01 Date Code 20170105

 15

the virtual tags. If you select the control global_building2, then the Tag Processor assigns
SEL_735_2 meter values. The actual HMI diagram displayed does not change, but the values
change to correspond to the requested meter. The diagram title also changes to display the value
from tag HMI_Tags.Building_Name.
Note that this is a basic example. You can expand the capabilities of this basic indirect tagging
method by assigning the Visible attribute of various images (such as other meter PNG files) to the
control tag; this way, the picture of the specific meter in use on the single HMI diagram appears
or disappears depending on which building you select.

Figure 9 Indirect Tag Mapping

ANALYSIS TOOLS
The RTAC includes several statistical tools that can aid you in analyzing overall processing
burden. These tools have been referred to at different points throughout this document; in this
section, we describe these tools in more detail, including how you can use them as a way of
evaluating RTAC project performance.

Task Timers
While online with a project, select Tags from the Device Tree. Figure 10 shows the overall
SystemTotal_task_timer, which displays how long it takes (in microseconds) to execute all
processes in the Main Task cycle. Note that in your project max_us will most likely have a much
higher value than average_us. The reason for this is the extra time needed during startup. Click in
the Prepared value column on the row labeled reset_all_task_timers, set the Value to TRUE,
and force the value by pressing <F6> on your keyboard. Press <Shift+F6> to unforce it. This
resets the timers so max_us more accurately reflects the maximum number of microseconds each
task iteration uses. Check that all timers are momentarily set to 0 and that they have repopulated
with presently computed values.

Date Code 20170105 SEL Application Guide 2017-01

16

Figure 10 System Total Task Timer

Configure the cycle time on the Main Controller tab to be at least 140 percent of the
SystemTotal_task_timer average_us. Setting the cycle time at a significantly greater value than
the time actually used gives the RTAC time to perform the background tasks mentioned
previously. We recommend that you configure the cycle time to an interval that meets your
system requirements, but not any faster. Note that if the SystemTotal_task_timer is only a few
milliseconds and you are using the HMI, you may need to increase the cycle time to much more
than 140 percent to give the RTAC time to process HMI traffic.
Also, be aware that each POU has timers. Expand the timers to observe how long each process
takes. You can also view the task timers of processes in other tasks, such as the Automation Task.
To accomplish this from the Home ribbon, click Tools > Watch Values > Watch 1 to open a
watch window and type the following in the Expression column of the window:
SEL_RTAC.Application.Automation_SystemTotal_task_timer
See Figure 11 for an example of a watch window with the expression entered.

Figure 11 Automation Task Timers

The most important application of task timers is comparing them to the configured task cycle
time. This information is precalculated for you and appears on the RTAC web interface on the
Dashboard page as the percentage of task cycle time. Figure 12 shows these values, as well as
memory usage and the number of users logged in. If the Main Task Usage or the Automation

SEL Application Guide 2017-01 Date Code 20170105

 17

Task Usage numbers are 100 percent or more, your project is not configured efficiently enough to
execute all the processes in the configured task cycle times. Note that the number of users logged
in includes any Open Database Connectivity (ODBC) connections, which includes
ACSELERATOR RTAC connections and web interface connections.

Figure 12 RTAC Web Interface Dashboard Statistics

CPU Usage
As mentioned previously, you can view the average CPU burden of an online project by selecting
SystemTags from the Device Tree, clicking Tags, and then viewing CPU_Burden_Percent
(a snapshot burden percent) and CPU_Burden_Percent_5_Second_Average. A high CPU
burden may indicate that the task is having difficulty processing the configured tasks, but not
necessarily. High polling rates, fast protocols (such as MIRRORED BITS or GOOSE), large
numbers of IED connections, and faster task cycle times all contribute to CPU burden. Often,
however, high CPU burden is a result of web or HMI usage; CPU burden also increases when
EtherCAT is running. A high percentage of task cycle time usage, as indicated on the RTAC web
interface Dashboard page (Figure 12), results in increased CPU usage. If the RTAC does not have
an HMI and is not using EtherCAT, if no users are logged into the web interface, and if the
percentage of task cycle time is sufficiently low, then high CPU burden may indicate excessive
protocol traffic and processing. As stated previously, this is most often caused by insufficient
deadband values on analog points and the resulting large quantity of analog event data. You can
verify if IEDs are sending large quantities of event data by using the Comm Monitor, described in
Comm Monitor on page 18 and in the ACSELERATOR RTAC Instruction Manual.

Protocol Statistics
When working on a project online, you can view protocol statistics in the Controller tab of the
individual protocol you are attempting to analyze. POU pins are the inputs that control the
process and the outputs that result from that process. The Controller tab shows the actual values
of all the input and output POU pin settings that have Visible set to TRUE in the POU Pin
Settings tab (see Figure 13). For example, while online, you can view many different POU pin
values, including the following:
• Message_Sent_Count. The number of messages or polls the RTAC sent to an IED.
• Message_Received_Count. The number of replies received from the IED.
• Offline. The IED is failing to communicate.
• Auto_Configuration_Pending. In SEL protocol, polling has not yet started because
automatic configuration is not yet complete.

Date Code 20170105 SEL Application Guide 2017-01

18

Some POU pins are common between protocols but many are unique to a specific protocol. For
example, DNP3 client has a POU pin called IIN_2_3_Buffer_Overflow. This is a Boolean value
transmitted by the polled device and indicates that the device is queuing events more frequently
than they are being polled, received by the client, and acknowledged. The usual cause of this
situation is misconfigured analog input deadbands on the IED.

Figure 13 POU Pin Settings

Comm Monitor
When you suspect a protocol issue or another communications issue, the best approach to
analyzing the problem is to view it. You can view raw communications data by using the Comm
Monitor. While online with an RTAC project, select Comm Monitor from the Home ribbon.
Select the connection type from Capture Source drop-down menu, and then select the device
(see Figure 14). A window displays the transmit-and-receive byte stream for the selected device.
You can stop the capture and open it in Wireshark for data analysis. Wireshark is a free, open-
source protocol analysis tool that you can download from the internet (see wireshark.org).

Figure 14 Comm Monitor

Consider a scenario in which there appears to be a significant delay in a control sent from
SCADA to an RTAC and then from the RTAC to an IED. Perform the following steps to analyze
the cause of the latency:
Step 1. Open a Comm Monitor session between the RTAC and SCADA.
Step 2. Open a Comm Monitor session between the RTAC and the IED.
Step 3. Send the control from SCADA.

SEL Application Guide 2017-01 Date Code 20170105

 19

Step 4. Stop both Comm Monitor sessions when SCADA determines that the control was
successful.
Step 5. In the Comm Monitor (or Wireshark), view the time stamp of the command from
SCADA, the time stamp of the command from the RTAC to the IED, and the time
stamp of the indication reply from the IED. By evaluating the relationships of these
message time stamps, you can determine whether the delay occurred after the
RTAC received the control or after the RTAC sent the control to the IED; you can
also determine if the control feedback from the IED was delayed rather than that
the control itself was delayed.

THE RTAC PLATFORM

Every product in the SEL RTAC family has unique characteristics that make each one more
suitable for particular applications. For example, quantity of serial ports, amount of I/O, form
factor, memory, and processing power varies among RTACs. Part of creating a high-performance
system, besides the items we have discussed thus far, includes choosing the correct hardware for
the job. Several RTAC platforms (e.g., SEL-3530, SEL-3530-4, and SEL-2241) have similar
memory quantities and processing capabilities. The SEL-3505, however, has less memory and
processing ability, which makes it more suitable for less tag- and communications-intensive
applications. The SEL-3555, on the other hand, has processing power that is an entire order of
magnitude greater than the SEL-3530.
This section gives some general guidelines on choosing an RTAC platform to best suit your
system requirements. Each platform has different configurations for I/O, communications ports,
etc.; this information is covered in detail in flyers, data sheets, and instruction manuals for the
individual products, which are all available on the SEL website. While it is impossible to give the
exact number of tags or IEDs that should be used for each RTAC, Table 1 provides general
recommendations. Keep in mind that poll rate, number of tags per IED, and other factors greatly
influence how many IEDs you can actually configure in RTAC projects.

Table 1 General Guidelines for RTAC Platform Selection

Task Cycle at
Recommended
RTAC Platform Maximum Tags Recommended
Maximum IEDs
Maximum IEDs (ms)
SEL-3530
80–100 25,000 500+
SEL-3530-4
SEL-2241 and others
45–65 25,000 500+
with Axion I/O
SEL-3505 <20 5,000 500+
SEL-3555 250–300 100,000 500+

We recommend conducting preliminary testing on projects to ensure the combination of the

number of tags, protocol connections, task cycle, and HMI usage results in satisfactory
performance. For example, your desktop computer performs many simultaneous tasks, but you
would not expect it to run every software package at the same time and still perform in a
satisfactory manner. By quickly configuring an RTAC with the approximate number of tags,
IEDs, etc., that your project requires, you can observe if CPU and task cycle usages are able to
meet your performance requirements. After you connect devices and change tag values, you will

Date Code 20170105 SEL Application Guide 2017-01

20

have to consider additional burden, but the preliminary configuration will at least give you an
idea if what you require is feasible with the platform you are testing.

CONCLUSION
The SEL RTAC family provides powerful and flexible automation platforms that can gather data
from IEDs, manipulate data to create useful information, perform custom control logic, and report
data to polling entities such as SCADA systems through industry-standard protocols, email and
text messaging, and through visualization in a web-based HMI. Our recommended best practices
for RTAC configuration, as discussed in this application guide, include the following:
• Poll for data only as frequently as necessary
• Poll for only the data you need
• Report only the data you need
• Process the data (per task cycle time) only as fast as necessary
By following these basic RTAC guidelines, you can improve overall system performance, ensure
correct execution of user logic, and improve web responsiveness.

FACTORY ASSISTANCE
We appreciate your interest in SEL products and services. If you have questions or comments,
please contact us at:
Schweitzer Engineering Laboratories, Inc.
2350 NE Hopkins Court
Pullman, WA 99163-5603 U.S.A.
Tel: +1.509.332.1890
Fax: +1.509.332.7990
selinc.com
info@selinc.com

© 2017 by Schweitzer Engineering Laboratories, Inc.

All rights reserved.

All brand or product names appearing in this document are the

trademark or registered trademark of their respective holders. No
SEL trademarks may be used without written permission.

*AG2017-01*
SEL products appearing in this document may be covered by
U.S. and Foreign patents.

SEL Application Guide 2017-01 Date Code 20170105