Xrdocs Io Asr9k Tutorials XR Embedded Packet Tracer
Xrdocs Io Asr9k Tutorials XR Embedded Packet Tracer
Aleksandar Vidakovic
Principal Engineer XR Routing Deployment & Escalation, Cisco. Follow
Save to PDF
O N T H I S PA G E
INTRODUCTION
PA C K E T T R A C E R YO U T U B E D E M O V I D E O
X R E M B E D D E D PA C K E T T R A C E R F R A M E W O R K A R C H I T E C T U R E
USER INTERACTION
X R E M B E D D E D PA C K E T T R A C E R P E R F O R M A N C E C O N S I D E R AT I O N S
USE CASES
X R E M B E D D E D PA C K E T T R A C E R R E S T R I C T I O N S A N D L I M I TAT I O N S
A P P E N D I X 1 : X R E M B E D D E D PA C K E T T R A C E R F R A M E W O R K A R C H I T E C T U R E D E TA I L S
Introduction
IOS XR Embedded Packet Tracer is a framework that provides the user with a capability to trace custom ows through
the router, for service validation or troubleshooting purposes.
IOS XR Embedded Packet Tracer is protocol agnostic, it works on any type of unicast or multicast packets.
This document is a user guide, with additional insight into the architecture of the XR Packet Tracer.
IOS XR Embedded Packet Tracer support starts with IOS XR release 7.1.2 and ASR 9000. Other XR product families will be
supported in future. XR release 7.1.2 provides the very basic functionality that the packet tracer can deliver. Further
development directions will depend much on your feedback.
For a quick overview of what you can expect from the XR Embedded Packet Tracer, watch this short video:
When packet tracing is enabled on an interface, the Network Processor (NP) checks whether received packets are
matching the speci ed condition. If packet matches the condition, a ag is set in the internal packet header. This ag in
the internal packet header allows for the tracing of this packet on all elements in the data-path and punt-path inside the
router.
In XR release 7.1.2 packet tracing is supported only in ASR9000 data-path. Support is available on 3rd, 4th and 5th
generation line cards (aka Tomahawk, Lightspeed and Lightspeed Plus). Packet tracing support by processes
participating in the punt/inject path will be available in future IS XR releases.
On ASR9000 network processor microcode (in case of Tomahawk) and packet processing engine code (in case of
Lightspeed) participate in the packet tracer framework. HW ASICs (FIA, XBAR, PHY) do not have the capability to
participate in the packet tracer framework. Therefore any actions performed by microcode are reported to the packet
tracer infrastructure, but actions peformed by HW ASICs are not.
User Interaction
The main pillar of the XR Embedded Packet Tracer architecture is simplicity of user experience.
At this stage of XR Embedded Packet Tracer framework development, user interface is provided through a CLI.
User interaction with the packet tracer framework is entirely contained in the user mode. There is no need for any
con guration changes to enable the packet tracer functionality.
clear packet-trace
Clears all bu ered packet-trace conditions. Command is allowed only while packet tracing is inactive.
conditions all
clear packet-trace
Resets all packet-trace counters to zero.
counters all
packet-trace condition
Specify interfaces on which you expect to receive packets that you want to trace through the router.
interface interface
packet-trace condition n
o set o set value value Specify set(s) of the O set/Value/Mask that de ne the ow of interest.
mask mask
show packet-trace
See all counters registered with the packet tracer framework along with their descriptions.
description
See conditions bu ered by the pkt_trace_master process running on the active RP and the packet tracer status (Active/Inactive). The detailed
show packet-trace status
option of the command shows which processes are registered with the packet tracer framework on every card in the router. If packet tracer
[detail]
status is Active, output also shows which conditions were successfully programmed in data-path.
packet-trace start
packet-trace stop
clear packet-trace counters all
clear packet-trace conditions all
By design, packet tracer conditions can be cleared only while packet tracing is inactive. Interpertation of packet tracer
results would othewrise we be dubious.
packet-trace condition interface hu0/5/0/6
packet-trace condition interface hu0/5/0/7
packet-trace condition interface hu0/5/0/8
When tracing on sub-interfaces, the O set/Value/Mask speci cation must take into account the dot1q or QinQ
encapsulation.
De ning a ow as a set of O set/Value/Mask triplets allows the packet tracer framework to be completely protocol
agnostic. The O set/Value/Mask can represent any part of any header in the protocol stack and does not even have to
end on the header boundary.
To address the usability aspect of this approach, we have developed the “XR Packet Tracer Condition Generator Web
App”.
Source code and installation instructions of this Web App are available on GitHub: XR Embedded Packet Tracer -
Condition Generator.
This Web App allows you to draw the protocol stack in your frame of interest, specify which of them are of interest for
de ning the condition and nally enter the values (with optional masks) that de ne your ow of interest.
Starting page of the Web App shows the supported protocol headers.
Click on the + sign to add a header to the stack in the desired order. Clicking on the - sign removes from the stack the
last header of that type. If you want to reset the stack completely, just reload the page.
Make sure you add all the headers before the one on which you want to match the tra c because the o set calculation
depends on it. Don’t forget the PW control word if it’s in use. ;)
The outermost header should be the rst one you select. Then click on the other headers to draw the protocol stack until
the innermost header that you want to match on. For example:
Click on the checkbox next to the headers on which you want to match. You can click on more than one. A frame for
each selected header will apper to the right. Enter the value/mask of your choice and click on Submit button in the frame.
Click on the Copy icon to copy the O set/Value/Mask to clipboard.
packet-trace condition 1 Offset 53 Value 0x01 Mask 0xff
packet-trace condition 3 Offset 60 Value 0xc0a84d02 Mask 0xffffff00
On ASR 9000, you can specify a maximum of three 4-octet O set/Value/Mask sets.
Use the show packet-trace status command to check which conditions were bu ered so far by the pkt_trace_master
process running on the active RP and what is the aggregate packet tracer status (active/inactive).
Buffered Conditions:
Interface HundredGigE0/5/0/6
1 offset 53 value 0x1 mask 0xff
2 offset 56 value 0xc0a84d01 mask 0xffffffff
3 offset 60 value 0xc0a84d02 mask 0xffffffff
Status: Active
RP/0/RSP0/CPU0:CORE-TOP#
The detailed option of the status command “show packet-trace status detail” can be used to see which processes are
registered with the packet tracer framework on every card in the router. If packet tracer status is Active, you can also
verify which conditions were programmed in data-path. Packet tracer conditions are broadcast to all participatig
processes when the packet-trace start command is issued, but only the NPs that own the interfaces speci ed in the
packet tracer condition are programming it in HW.
#1 spp_pd
Last errors:
#2 netio_pd
Last errors:
#3 prm_server_to
Last errors:
#4 spio_pd_LACP
Last errors:
#5 spio_pd_ARP
Last errors:
#6 spio_pd_LLDP
Last errors:
#1 prm_server_to
Interfaces: 1
HundredGigE0/5/0/6
Conditions: 3
1 offset 53 value 0x1 mask 0xff
2 offset 56 value 0xc0a84d01 mask 0xffffffff
3 offset 60 value 0xc0a84d02 mask 0xffffffff
Last errors:
------------------------------------------------------------
Buffered Conditions:
Interface HundredGigE0/5/0/6
1 offset 53 value 0x1 mask 0xff
2 offset 56 value 0xc0a84d01 mask 0xffffffff
3 offset 60 value 0xc0a84d02 mask 0xffffffff
Status: Active
------------------------------------------------------------
Location: 0/RSP0/CPU0
#1 spp_pd
Last errors:
#2 netio_pd
Last errors:
RP/0/RSP0/CPU0:CORE-TOP#
show packet-trace result [counter <name> [source <source>] [location <location>]]
The simple form of this command shows the aggregate status of all non-zero counters. In particular, you can see the
following:
Counter source. In case of packet tracing on ASR9k data-path, source represents the NP.
Counter name.
Counter type: drop or pass
Last Attribute. With every counter update the owner of the counter may decide to provide an additional explanation
along with the counter update. In case of drop counters, you should expect to see the drop reason.
Counter value
Sample output:
When using the “show packet-trace results counter <_name_>" option, you can see:
the most recent 1023 increments of the given counter. Note that the packet tracer framework may receive a counter
increment that is higher than one, depending on how the process that collects data-path counter from NP updates the
packet tracer framework.
the timestamp of the cunter increment
The asterisk next to the counter name shows the most recent update. This is important if the counter was updated more
than 1023 times, in which case the oldest entries are overwritten.
Sample output:
------------------------------------------
Location: 0/5/CPU0
------------------------------------------
Use the following command to see which counters are registered with the packet tracer framework and their
descriptions:
show packet-trace descriptions
Performance impact on Lightspeed and Lightspeed Plus is negligeable even on the NP where the incoming packets are
checked against the speci ed condition. For example, NP load remains 28% in test setup with and without packet tracing.
Performance impact on Tomahawk line cards needs to be considered before enabling packet tracing condition on
interfaces owned by Tomahawk NP. If the NP utilisation is ~20%, enabling packet tracer may increase the NP utilisation to
~60%. Note that performance impact is not observed if Tomahawk NP is the egress NP.
Use Cases
The followig three use cases illustrate what can you expect to achieve with packet tracer on ASR 9000.
I’m sure you will agree that this is one of the most di cult issues to troubleshoot: packet pertaining to a L2VPN PW are
dropped somewhere in the core. XR Embedded Packet Tracer is the only feature that allows you to troubleshoot this
easily.
The following image shows the topology, relevant con gurations, protocol stacks, test ow (ICMP echo) direction and the
point where packet tracing is applied:
Protocol stack on any ow will depend very much on the network con guration. You can either derive it by walking the
control path from encapsulation PE to the P node where packet tracer is enabled to see what kind of header rewrites are
performed on each node (e.g.: is dot1q header stripped on encapsulation PE, how many labels are pushed, is PW control
word enabled, etc.) or you can run a monitor session once on the P router to con rm the protocol stack on the ow of
interest.
Packet tracer condition in this use case was developed using the XR Embedded Packet Tracer - Condition Generator
Web App.
The O set/Value/Mask triplets in below snapshot represent the following elds of the IPv4 packet encapsulated into
L2VPN frame:
ICMP protocol
source IPv4 address
The simple set of commands below was su cient to con rm that all packets of this ow are successfuly sent towards the
egress interface:
RP/0/RSP0/CPU0:CORE-TOP#packet-trace stop
RP/0/RSP0/CPU0:CORE-TOP#clear packet-trace conditions all
RP/0/RSP0/CPU0:CORE-TOP#clear packet-trace counters all
RP/0/RSP0/CPU0:CORE-TOP#packet-trace condition interface hu0/5/0/6
RP/0/RSP0/CPU0:CORE-TOP#packet-trace condition 1 Offset 53 Value 0x01 Mask 0xff
RP/0/RSP0/CPU0:CORE-TOP#packet-trace condition 2 Offset 56 Value 0xc0a84d01 Mask 0xffffffff
RP/0/RSP0/CPU0:CORE-TOP#packet-trace condition 3 Offset 60 Value 0xc0a84d02 Mask 0xffffffff
RP/0/RSP0/CPU0:CORE-TOP#show packet-trace status
------------------------------------------------------------
Buffered Conditions:
Interface HundredGigE0/5/0/6
1 offset 53 value 0x1 mask 0xff
2 offset 56 value 0xc0a84d01 mask 0xffffffff
3 offset 60 value 0xc0a84d02 mask 0xffffffff
Status: Inactive
RP/0/RSP0/CPU0:CORE-TOP#packet-trace start
RP/0/RSP0/CPU0:CORE-TOP#show packet-trace status
------------------------------------------------------------
Buffered Conditions:
Interface HundredGigE0/5/0/6
1 offset 53 value 0x1 mask 0xff
2 offset 56 value 0xc0a84d01 mask 0xffffffff
3 offset 60 value 0xc0a84d02 mask 0xffffffff
Status: Active
Packet tracer framework can explicity report packet drops if the packet drop was a decision made by any entity that
participates in the packet tracer framework. This means that packets dropped by the NP microcode are explicitly
reported.
In this use case an egress policer was applied to the Hu0/5/0/1 interface of the P router:
Note that, as the policer drop is a decision made by the NP microcode, you can see the PACKET_EGR_DROP counter
increment in the output of show packet-trace result counter.
You can also see that the additional information passed with the PACKET_EGR_DROP counter increment explains the drop
reason. In this use case drop reason was RSV_DROP_QOS_DENY , which is a policer drop in NP microcode parlance.
You can also see that the number of drops matches exactly the drops in the output of show policy-map interface hu0/5/0/1
output command.
Further you can observe using the “show packet-trace results counter <_name_>" the drop reason in all increments of
`PACKET_EGR_DROP` counter.
Class exp4
Classification statistics (packets/bytes) (rate - kbps)
Matched : 1000/144000 8
Transmitted : N/A
Total Dropped : 77/11088 1
Policing statistics (packets/bytes) (rate - kbps)
Policed(conform) : 923/132912 7
Policed(exceed) : 77/11088 1
Policed(violate) : 0/0 0
Policed and dropped : 77/11088
Class class-default
Classification statistics (packets/bytes) (rate - kbps)
Matched : 25/2052 0
Transmitted : N/A
Total Dropped : N/A
RP/0/RSP0/CPU0:CORE-TOP#
RP/0/RSP0/CPU0:CORE-TOP#sh packet-trace results counter PACKET_EGR_DROP location 0/5/CPU0
T: D - Drop Counter; P - Pass Counter; M - Marking Master Counter; * - Last Updated Index of Counter
------------------------------------------
Location: 0/5/CPU0
------------------------------------------
Packets dropped by any HW ASIC (e.g. FIA, Xbar) are not explicitly reported. The number of such packet drops can be
inferred from the mismatch between the number of expected and counted packets. E.g. if PACKET_FROM_FABRIC count
on egress NP is less than the PACKET_TO_FABRIC count on ingress NP, this indicates that packets are dropped in FIA.
While this doesn’t directly explain the drop reason, it does indicate where are the drops happening and helps point
further troubleshooting in the right direction.
In this use case the egress policer on Hu0/5/0/1 was replaced with a shaper of the same rate:
NP Tra c Manager is a hardware component of the ASR 9000 network processor. As the Tra c Manager receives the
packet after the packet has already been processed by the NP microcode, drops by Tra c Manager are not reported to
the packet tracer framework. They can, however, be observed in the QoS statistics.
Class exp4
Classification statistics (packets/bytes) (rate - kbps)
Matched : 1000/1344000 606
Transmitted : 75/100800 38
Total Dropped : 925/1243200 0
Queueing statistics
Queue ID : 122898
High watermark : N/A
Inst-queue-len (packets) : 0
Avg-queue-len : N/A
Taildropped(packets/bytes) : 925/1243200
Queue(conform) : 75/100800 38
Queue(exceed) : 0/0 0
RED random drops(packets/bytes) : 0/0
Class class-default
Classification statistics (packets/bytes) (rate - kbps)
Matched : 2/176 0
Transmitted : 4/340 0
Total Dropped : 0/0 0
Queueing statistics
Queue ID : 122899
High watermark : N/A
Inst-queue-len (packets) : 0
Avg-queue-len : N/A
Taildropped(packets/bytes) : 0/0
Queue(conform) : 4/340 0
Queue(exceed) : 0/0 0
RED random drops(packets/bytes) : 0/0
This use case demonstrates how packet tracer framework can be used to trace punted packets. It also shows the
speci c way in which ASR 9000 is responding to ICMP echo requests.
Applied packet tracer condition to match VLAN ID 2020 and IPv4 address 202.202.202.201:
Buffered Conditions:
Interface HundredGigE0/5/0/0
1 offset 14 value 0x7e4 mask 0xfff
2 offset 34 value 0xcacacac9 mask 0xffffffff
Status: Active
RP/0/RSP0/CPU0:CORE-TOP#
You can also observe that NetIO on IOS XR routers reuses the bu er carrying the ICMP echo request packet when it
generates the echo reply, thus preserving the packet tracer ag in the internal packet header.
XR release 7.1.2:
Packet marking is supported on 5th, 4th and 3rd generation line cards (aka Lightspeed Plus, Lighstpeed, Tomahawk).
Packet tracing is supported in the NP microcode of 5th, 4th and 3rd generation line cards.
You can specify a maximum of three 4-octet O set/Value/Mask sets.
Embedded Packet Tracer is not HA (high availability) aware. Speci ed packet tracer conditions are not synchronised
with the standby RP.
By design, packet tracer conditions cannot be updated while packet tracing is active.
Packet tracer master process pkt_trace_master on active route processor (RP) card is responsible for user interaction and
sending instructions to processes participating in the packet tracer framework. When packet-trace start command is
issued, pkt_trace_master process broadcasts the speci ed conditions to all participating processes. Receiving process
detects whether the condition applies to entities it owns and acts accordingly. That way only the NPs that own the
interfaces speci ed in the packet tracer condition programme the condition in HW.
When packet tracing is enabled on an interface, the Network Processor (NP) checks whether received packets are
matching the speci ed condition. If packet matches the condition, a ag is set in the internal packet header. This ag in
the internal packet header allows for the tracing of this packet on all elements in the data-path and punt-path inside the
router.
On every line card (LC) and route processor card (RP) a pkt_trace_agent process maintas the array of counters
registered by each process participating in the packet tracer framework. Processes participating in the packet tracer
framework communicate counter updates to the pkt_trace_agent process. When you issue the show packet-trace result
command, the pkt_trace_master process on active RP polls data from all cards and displays the non-zero counters.
SHARE ON
Leave a Comment
3 Comments
1 Login
Name
F
freealx − ⚑
6 months ago
0 0 Reply ⥅
V
Veras Santos − ⚑
a year ago edited
Hi,
This is a great post, thank you for that !
I'm having a hard time to nd the right offset values to match a source/destination ipv4
address for a packet crossing a GRE tunnel.
So header stack should be something like that:
ethernet + ipv4 + gre + ipv4.
The packet tracer condition generation tool doesn't account for that case.
Any idea how I can nd the right offset values for that ow?
0 0 Reply ⥅
Vikram Sisodia − ⚑
3 years ago
0 0 Reply ⥅