What are Packet Captures - A Brief Introduction to Packet Captures

Packet capture is a activity of capturing data packets crossing networking devices

There are 2 types - Partial packet capture and Deep packet capture

Partial packet capture just record headers without recording content of datagrams, used
for basic troubleshooting upto L4

Deep packet capture will give us everything that a packet can tell, doing a deep packet
analysis is like investigating in a forensic lab,

it is used in advanced troubleshooting like troubleshooting at L7, troubleshooting for

performance related issues, traffic patterns etc

There are 2 ways of looking at traffic coming to any device, either collect captures on the
ingress of the device or collect captures on the egress interface of the device behind the
device in question

It might be sometimes necessary to collect captures on the egress interface, for example
in case our device is dropping packets even before it is processing it or if we have to
collect captures for large data as captures on some devices are limited by buffer size

Collecting captures on ASA

You can enable captures on ASA either from CLI or from ASDM

Enable captures on ASDM

Go to wizards and select packet capture wizard, it will take you through 6 simple self
explanatory steps, once done with captures select save captures. This has been illustrated
in Scenario 1

Enable captures in CLI

This is the syntax to apply capture

capture <name of capture>

These are the options available

Capture packets that match access-list, when you specify access-list make
access-
sure that you specify the traffic in both direction if you want to capture bi-
list
directional traffic
Default is 512 KB and you can configure it upto 32 MB, you do not need to
change this in most cases. Just a note of caution - applying captures will add
buffer
to memory utilization so keep an eye on memory before enabling captures
with max buffer
circular-
Overwrite buffer from beginning when full, default is non-circular
buffer
EtherType is a two-octet field in an Ethernet frame. It is used to indicate
which protocol is encapsulated in the PayLoad of an Ethernet Frame. Default
ethernet- is IP
type
Here is link for iana assigned ethernet type numbers
http://www.iana.org/assignments/ethernet-numbers
headers- Capture only L2, L3 and L4 headers of packet without data in them, useful
only for collecting partial packet capture
interface Used to specify the interface on which you want to apply the capture
 Capture packets matching five-tuple - 5 tuple consists of

-> Type of protocol - eg ip, gre, esp, icmp etc>

match -> Source Destination IP

-> and other specific detail related to type of protocol specified for example
in case of tcp it would be src dst port or in case of icmp it would be icmp
type (optional)
packet- Defines maximum length of each packet to capture, default is 1518 bytes
length which is the mtu in most cases, maximum is 9216 bytes
Display captured packets in real-time. Warning: using this option with a
real-time slow console connection may result in an excessive amount of non-displayed
packets due to performance limitations. This is very rarely useful
This keyword enables you to check the output of packet tracer for each
packet, note that this will show packet tracer output only for inbound
packets. This is useful in cases when you want to check the various checks in
trace
firewall for consecutive packets as the normal packet tracer command will
always show you output for new connection check the view packet capture
section to learn how to check the trace output.
These are the various option available here

asp-drop Capture packets dropped with a particular reason

isakmp Capture encrypted and decrypted ISAKMP payloads

raw-data Capture inbound and outbound packets on one or more

interfaces
type
tls-proxy Capture decrypted inbound and outbound data from TLS
Proxy on one or more interfaces

webvpn Capture WebVPN transactions for a specified user

You need to know what you are looking for when you want to collect these
captures, for example asp drop captures might generate lot of output so
unless you dont know what kind of drop you are looking for you might end
up looking at lot of packets
Example of capture

capture capin interface inside match ip host 1.1.1.1 host 2.2.2.2 ----> this will use
defaults for other parameters

Viewing captures

You can view captures in 2 ways view it on CLI/ASDM or in other words view it on the
device itself or you can view it on a packet analyser after exporting it in pcap form

Let us examine each of them closely

Viewing it on the device itself

You can watch the captures in real time when you enable it on asdm or you can watch it
real time when you enable capture on cli using the option "real-time" (not really
recommended as it may lead to excessive amount of non displayed packets in some
cases)

Once you are done with capturing you can view them by issueing the command show
capture <capname> this will display minimum information - src dst ip, src dst port,
timestamp and ethertype

You can view some more information by using the extended form of show cap
<capname>
show cap <capname> <one of the keywords below>

access-list Display packets matching access-list

Display <number> of packets in capture - lets you display specified number
count
of packets
Display more information for each packet - like src dst mac address, ttl, ip id
detail
- this has been illustrated in Scenario 1
dump Display hex dump for each packet - shows datagram in hex
packet- Display packet <number> in capture - lets you view captures starting from a
number specified packet number
Display extended trace information for each packet - used if capture is set
trace using the trace keyword as mentioned above, this will show the output of
packet tracer for each packet in the inbound direction

Viewing it on a packet anaylser tool

You can export these captures and save them on your PC and view it using a packet
anaylser tool like wireshark (open source tool available for free on internet). there are 2
ways of doing this

Export via https

For this you need to enable http server on your ASA and you need to know the
credentials used to access asa via asdm (default is no username no password)

Comamnds to enable http server

asa#config t

asa(config)# http server enable

asa(config)# crypto key generate rsa modulus 1024

Note: This is for creating keys because we communicate with asa via https, if you have
ssh access you probably have these keys

Once you have enabled http server on asa go to your browser and give the following in
the url field

https://<ip address of asa>/capture/<capname>/pcap

if it is in multiple context mode you have to specify the context

https://<ip address of asa>/capture/<context name>/<capname>/pcap

After you enter this you will be prompted for username password and once you enter that
the captures are stored on your PC and you can open them in a packet anaylser tool

Export via copy command

copy /pcap capture:

disk
flash
ftp
smb
system
tftp

As metioned before in some cases you might need to capture packets on devices directly
connected to asa and in most cases it is a switch connected to ASA and in such cases you
can also span the switchport to collect captures

Here is a link which will help you setup a span on your catalyst switch

http://www.cisco.com/en/US/products/hw/switches/ps708/
products_tech_note09186a008015c612.shtml#config

Once done always make sure that you remove the captures using the command

no capture <capname>

Troubleshooting simple scenario's using packet capture

Scenario - 1 - Basic Connectivity

To verify if ASA is dropping any packet - simple connectivity issues

Topology:

So here we nat all users to interface ip

Commands to capture traffic

Inside:

access-list capi extended permit ip host 192.168.1.2 host 4.2.2.2

access-list capi extended permit ip host 4.2.2.2 host 192.168.1.2

capture capin interface inside access-list capi

Or

capture capin interface inside match ip host 192.168.1.2 host 4.2.2.2

[this is possible in asa 8.0 and above and we do not need to be in config mode to put
apply an capture]
Outside:

access-list capo extended permit ip host a.b.c.d host 4.2.2.2

access-list capo extended permit ip host 4.2.2.2 host a.b.c.d

capture capout interface outside access-list capo

Or

capture capout interface outside match ip host a.b.c.d host 4.2.2.2

[Note that we are using the natted ip - so for capture use the ip addresses that you
expect to see on the wire after all processing is done for egress interface and before any
processing is done for ingress interface]

We can also apply capture using ASDM and the below screen shots show the steps for
that

Ping from 192.168.1.2 to 4.2.2.2

Pings Succed
show capture capin

192.168.1.2> 4.2.2.2: icmp: echo request

4.2.2.2 > 192.168.1.2: icmp: echo reply

192.168.1.2 > 4.2.2.2: icmp: echo request

4.2.2.2 > 192.168.1.2: icmp: echo reply

show capture capout

a.b.c.d> 4.2.2.2: icmp: echo request

4.2.2.2 > a.b.c.d: icmp: echo reply

a.b.c.d > 4.2.2.2: icmp: echo request

4.2.2.2 > a.b.c.d: icmp: echo reply

So you can see we have captured bidirectional traffic

If you want to see more details you can use the detailed keyword at the end

show capture capin detail

 000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo request
(ttl 128, id 52241)

0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply
(ttl 127, id 16992)

000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo

request (ttl 128, id 52242)

0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply
(ttl 127, id 16993)

000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo

request (ttl 128, id 52243)

0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply
(ttl 127, id 17008)

000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo

request (ttl 128, id 52244)

0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply
(ttl 127, id 17251)

show capture capout detail

0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl
128, id 52241)

0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl
127, id 16992)

0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl
128, id 52242)

0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl
127, id 16993)

0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl
128, id 52243)

0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl
127, id 17008)

0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl
128, id 52244)

0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl
127, id 17251)

You can see we have some additonal information here like mac address, ip id, ttl etc.

TIP: For basic connectivity issues always check the following

-> interface access-list

-> nat rules

-> if pings are not working check for

inspect icmp -> Check if you have this in the policy-map, you can either add this or
explicitly add acl's to permit icmp packets in access-lists on the lower security level
interfaces

icmp deny any outside -> check if you have any deny statements like this, this
statement means that we deny any icmp traffic on outisde interface
Once you have checked the above use packet tracer and packet capture to isolate the
issue further

Scenario - 2 - ICMP works but TCP traffic does not work

Users report that they can ping a particular server but cannot access any TCP services on
it

Topology:

This is a typical case of asymmetric routing were users complain that they can ping the
server but cannot access it on any of the tcp services

Lets see how captures help us in such scenarios

Commands to capture traffic

Inside:

access-list capi extended permit ip host 192.168.1.2 host x.x.x.x

access-list capi extended permit ip host x.x.x.x host 192.168.1.2

capture capin interface inside access-list capi

Or

capture capin interface inside match ip host 192.168.1.2 host x.x.x.x

[this is possible in asa 8.0 and above and we do not need to be in config mode to put
apply an capture]

Outside:

access-list capo extended permit ip host a.b.c.d host x.x.x.x

access-list capo extended permit ip host x.x.x.x host a.b.c.d

capture capout interface outside access-list capo

Or

capture capout interface outside match ip host a.b.c.d host x.x.x.x

[Note that we are using the natted ip - so for capture use the ip addresses that you
expect to see on the wire after all processing is done for egress interface and before any
processing is done for ingress interface]

When the user tries to browse a webpage on this server

Capture Outputs

show cap capin

192.168.1.2.58458 > x.x.x.x.80: S 2331481362:2331481362(0) win 5840 <mss

1460,sackOK,timestamp 2359730964 0,nop,wscale 7>

192.168.1.2.58458 > x.x.x.x.80: . ack 712296789 win 5840

show cap capout

a.b.c.d .58458 > x.x.x.x.80: S 126872520:126872520(0) win 5840 <mss

1380,sackOK,timestamp 2359730964 0,nop,wscale 7>

Captures on client

192.168.1.2.58458 > x.x.x.x.80: S 2331481362:2331481362(0) win 5840 <mss

1460,sackOK,timestamp 2359730964 0,nop,wscale 7>
 192.168.1.2.58458 > x.x.x.x.80: S 126872520:126872520(0) win 5840 <mss
1380,sackOK,timestamp 2359730964 0,nop,wscale 7>

192.168.1.2.58458 > x.x.x.x.80: . ack 712296789 win 5840

So as you can see the ASA just saw one side of traffic and dropped the ack from the
client to the server becuase it did not see the syn-ack go to the client

So this suggests that there is asymmetric routing

Solution:

Correct this assymetric routing and make sure that ASA see's both sides of traffic. For
any security appliance performing tcp checks it is important that it see's both sides of
traffic. Sometimes it is unavoidable and we have to live with asymmetric routing in that
case we can configure tcp state bypass for this traffic (you need to run asa version 8.2.1
and later )

http://www.cisco.com/en/US/products/ps6120/
products_configuration_example09186a0080b2d922.shtml
Scenario - 3 - Capture Backplane traffic to troubleshoot module related issues

Some times you might need to capture backplane captures hwen troubleshooting module
related issues

Here is the command to enable backplane captures on dataplane

capture <capname> interface asa_dataplane

Some scenario's where these could be useful

-> Some websites not accesible when traffic passes through csc module

-> If dataplane communication issue is reported in the logs

Here is the command to enable backplane captures on control plane

capture <capname> interface cplane

We will need control plane captures to troubleshoot issues related to communication

between asa and module

If the issue is one of the above it will be helpful to attach the captures while opening a
TAC case

Scenario - 4 - VPN troubleshooting using captures

A variety of VPN issues can be troubleshooted using packet captures. Packet captures are
easy to read and understand if we know what exactly we need to capture. As far as VPN
is concered we are mainly concerned about the traffic between peer IP's on the internet
facing side and traffic between internal subnets on the internal side.

You can capture the traffic in the same way as explained in the previous sections, the
intention on this section is to give an idea on what captures to apply for specific issues

Lan to Lan Tunnel (Site to Site)

 Tunnel not coming UP, show crypto isakmp sa does not show up anything on
either sides
o There is something wrong in the config the tunnel was never initiated
 Tunnel not coming UP, show crypto isakmp sa shows that tunnel is initited on
one of the side but on the responder nothing shows up in this output
o The phase one for IPSEC VPN uses udp 500 so apply captures for this on
both sides and verify that you are actually getting the packets on the
responder side. If you dont receive the packets something in the way is
blokcing these packets. Check if you have the required ports open on the
upstream devices and check with both of the ISP's if they have the
required ports open, if you receive the packets and still nothing shows up
its probably a configuration issue check configuration
 Tunnel is UP, show crypto isakmp sa shows that the tunnel is up (state is
MM_ACTIVE on ASA and QM_IDLE on Router) but you are unable to pass
any traffic between the 2 sites
o Apply captures between the peer ip's and check that there is 2 way traffic
for ip protocol 50 or 51 (ESP or AH depending on what you are using).
Again as before if you see packets leaving one side and not reaching the
other check upstrean devices and check with both the ISP's
 If you see 2 way traffic for ESP or AH check if you see encap's/decap's in the
output of show crypto ipsec sa, if you don't see incremtns then you might want to
apply captures to check if we are sending and receiving packets for the traffic
between the 2 subnets
 If any of the device is behind a nat device and nat traversal is enabled you will
capture packets for udp 4500 instead of ESP or AH

Remote Access VPN

  Most of the above is applicable to this as well, just one addition and that is
sometimes we might use tcp 10000 for phase 2, so as i said before it is very
important to know what traffic we are trying to capture in order to make
troubleshooting easy using packet capture

EXAMPLE NAT 192.168.100.158->195.46.0.158

firepower# show cap

capture capin type raw-data trace interface Outside [Capturing - 0 bytes]
match tcp host 195.46.0.158 eq https any capture capout type raw-data trace interface Inside
[Capturing - 0 bytes]
match tcp any host 192.168.100.158 eq https capture asp type asp-drop all [Capturing - 456
bytes] capture cap2 type raw-data trace interface Outside [Capturing - 360 bytes]
match tcp host 195.46.0.158 any eq www capture cap1 type raw-data trace interface Inside
[Capturing - 360 bytes]
match tcp host 192.168.100.158 any eq www firepower# show cap cap1

8 packets captured

1: 11:41:45.561936 192.168.100.158.33702 > 52.79.197.188.80: S

2828631718:2828631718(0) win 29200 <mss 1460,sackOK,timestamp 771256418 0,nop,wscale
7>
2: 11:41:45.562287 192.168.100.158.52514 > 120.24.59.150.80: S
3235982492:3235982492(0) win 29200 <mss 1460,sackOK,timestamp 771256418 0,nop,wscale
7>
3: 11:41:46.563447 192.168.100.158.33702 > 52.79.197.188.80: S
2828631718:2828631718(0) win 29200 <mss 1460,sackOK,timestamp 771257420 0,nop,wscale
7>
4: 11:41:46.563538 192.168.100.158.52514 > 120.24.59.150.80: S
3235982492:3235982492(0) win 29200 <mss 1460,sackOK,timestamp 771257420 0,nop,wscale
7>
5: 11:41:48.567673 192.168.100.158.33702 > 52.79.197.188.80: S
2828631718:2828631718(0) win 29200 <mss 1460,sackOK,timestamp 771259424 0,nop,wscale
7>
6: 11:41:48.567750 192.168.100.158.52514 > 120.24.59.150.80: S
3235982492:3235982492(0) win 29200 <mss 1460,sackOK,timestamp 771259424 0,nop,wscale
7>
7: 11:41:52.576065 192.168.100.158.33702 > 52.79.197.188.80: S
2828631718:2828631718(0) win 29200 <mss 1460,sackOK,timestamp 771263432 0,nop,wscale
7>
 8: 11:41:52.576157 192.168.100.158.52514 > 120.24.59.150.80: S
3235982492:3235982492(0) win 29200 <mss 1460,sackOK,timestamp 771263432 0,nop,wscale
7>
8 packets shown
firepower# show cap cap2

10 packets captured

1: 11:41:45.562364 195.46.0.158.33702 > 52.79.197.188.80: S 3646823577:3646823577(0)

win 29200 <mss 1380,sackOK,timestamp 771256418 0,nop,wscale 7>
2: 11:41:45.562592 195.46.0.158.52514 > 120.24.59.150.80: S 3384770433:3384770433(0)
win 29200 <mss 1380,sackOK,timestamp 771256418 0,nop,wscale 7>
3: 11:41:46.563676 195.46.0.158.33702 > 52.79.197.188.80: S 3646823577:3646823577(0)
win 29200 <mss 1380,sackOK,timestamp 771257420 0,nop,wscale 7>
4: 11:41:46.563722 195.46.0.158.52514 > 120.24.59.150.80: S 3384770433:3384770433(0)
win 29200 <mss 1380,sackOK,timestamp 771257420 0,nop,wscale 7>
5: 11:41:48.567918 195.46.0.158.33702 > 52.79.197.188.80: S 3646823577:3646823577(0)
win 29200 <mss 1380,sackOK,timestamp 771259424 0,nop,wscale 7>
6: 11:41:48.567963 195.46.0.158.52514 > 120.24.59.150.80: S 3384770433:3384770433(0)
win 29200 <mss 1380,sackOK,timestamp 771259424 0,nop,wscale 7>
7: 11:41:52.576279 195.46.0.158.33702 > 52.79.197.188.80: S 3646823577:3646823577(0)
win 29200 <mss 1380,sackOK,timestamp 771263432 0,nop,wscale 7>
8: 11:41:52.576340 195.46.0.158.52514 > 120.24.59.150.80: S 3384770433:3384770433(0)
win 29200 <mss 1380,sackOK,timestamp 771263432 0,nop,wscale 7>
9: 11:42:00.601149 195.46.0.158.33702 > 52.79.197.188.80: S 3646823577:3646823577(0)
win 29200 <mss 1380,sackOK,timestamp 771271456 0,nop,wscale 7>
10: 11:42:00.601195 195.46.0.158.52514 > 120.24.59.150.80: S 3384770433:3384770433(0)
win 29200 <mss 1380,sackOK,timestamp 771271456 0,nop,wscale 7>
10 packets shown