Carrier Grade NAT - Observations and Recommendations

Chris Grundemann | North American IPv6 Summit | 11 April 2012

Agenda
CGN Technology CGN Challenges CGN Architectures Conclusions

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Starting with the Basics

CGN TECHNOLOGY

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Network Address Translation (NAT)

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

PAT and Address Overloading

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

NAT and the End to End Principle

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

NAT444
NAT Table NAT Table
76.121.26.3:2001<->10.1.0.2:1025

DHCPv4 Server

10.1.0.2:1025<->192.168.0.2:1025

CGN builds NAT mapping using public and private IPv4

IPv4 Internet

Changes DA and forwards IPv4 packet to host

Changes SA of packet, sends upstream

CGN changes DA CGN of packet, sends ISP Router packet to HGW IPv4 Packet SA 76.121.26.3:2001 IPv4 Packet DA 76.121.26.3:2001
7

Home Router

IPv4 Host
192.168.0.2 IPv4 Packet SA 192.168.0.2

IPv4 Packet SA 10.1.0.2 IPv4 Packet DA 192.168.0.2

4/13/12

IPv4 Packet DA 10.1.0.2

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Dual-Stack Lite
NAT Table
76.121.26.3:2001<-> 2001::1|192.168.0.2:1025

Modified DHCP Server Removes IPv6

header, DHCP forwards IPv4 lease packet to host contains IPv6 addr, CGN Encapsulates packet Host with IPv6 header, sends obtains address from it to CGN Router

CGN builds NAT mapping using IPv6, IPv4, and port, then performs NAT
IPv4 Internet

CGN translates DA, CGN adds IPv6 tunnel ISP Router header, sends IPv4 Packet packet to HGW IPv6 Header SA 76.121.26.3:2001 IPv4 Packet IPv6 Header DA 76.121.26.3:2001
8

Home Router

IPv4 Host
192.168.0.2 IPv4 Packet SA 192.168.0.2

IPv4 Packet SA 192.168.0.2 IPv4 Packet DA 192.168.0.2

4/13/12

IPv4 Packet DA 192.168.0.2

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Address sharing needed when IPv6 is not available

Address sharing

IPv4 Addresses
9

Supply Purchased/ reclaimed Supply Assigned by ARIN IPv4 demand drops with IPv6 adoption

Demand subscriber/ CE growth

Cable Television Laboratories, Inc. 2010. All Rights Reserved. Proprietary/Confidential.

Time

4/13/12

Typical Access Technology Transition Timeline

Connectivity Type IPv4 Native

Time
NAT444 NAT444 DS-Lite

IPv6

None

6RD

Native

Native

10

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

The Evil in Necessary Evil

CGN CHALLENGES

11

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

CGN Testing Background

CableLabs first conducted CGN testing in 2010
o

NAT444 only Both NAT444 and DS-Lite About one a quarter

Second round June Sep, 2011

o

Additional CGN testing in IPv6 interop events

o

12

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Overview of test scenarios

Single and dual ISP networks with one or more users on multiple home networks Test applications include
o o o o o o o o
13

Video services e.g. Netflix, YouTube, iClips, Silverlight Audio streaming e.g. Pandora, Internet Archive Peer-to-peer e.g. on line gaming, uTorrent FTP large file transfers SIP calls e.g. X-Lite, Skype Video chat e.g. Skype, OoVoo Social networking e.g. Facebook, Webkinz Web conferencing e.g. GoToMeeting
4/13/12

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

Client devices and gateways used for testing

Laptops running Vista, Win 7 and MAC OS Gaming consoles Tablet devices iPhone and Android smartphones CE devices
o

Blu Ray players, Smart TVs Most vendors represented

4/13/12

CPE routers
o

14

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

Observations
The following types of applications behaved erratically or had the potential to break:
o o o o

Video streaming, e.g. Netflix, YouTube Peer-to-peer, e.g. uTorrent, Bittorent, Limewire On line gaming, e.g. X-box FTP file transfer Different NAT types (full cone, partial cone) perform differently

Performance dependent on home gateway

o

Observed behaviors were exacerbated when multiple users or multiple home networks were involved User experience further degraded when crossing ISPs and when hairpinning through the same CGN
15
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential. 4/13/12

Log volumes
150 - 450 bytes/connection * 33k - 216k connections per sub per day -------------------------------------------------------------5 - 96 MB / user / day

Thats potentially over 1 PB per 1M subs per month Its also over 20Mbps for just the log stream
16
Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential. 4/13/12

CGN Challenges
Poor quality of experience for advanced services
o

Peer-to-peer, video streaming, gaming, etc.

Negative impact to targeted advertising/geo-location Logging requirements for lawful intercept

o

Petabytes of data

17

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Workarounds
Challenges P2P SIP (cannot initiate/ receive calls) uTorrent (seeding does not work) P2P Gaming Degraded experience for services such as Netflix, video streaming Slower Download rates (some clients) Negative impact to targeted advertising/geolocation Logging requirements for lawful intercept Workarounds Use Proxies for Peer to Peer applications Port Control Protocol Software Upgrade from Manufacturer Port Control Protocol Deploy tested home-routers from an approved list No known workarounds (Try larger MTU) Distributed CGN, Regional IP and Port assignments Deterministic NAT, Data compression, Bulk port assignment Large enough shared transition space Distributed CGN, VRF (MPLS/VPN)
4/13/12

Overlapping Addressing / NAT Zones Impacts to traffic engineering

18

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

Port Control Protocol (PCP)

PCP is an IETF protocol
o o

Expected to be an RFC soon Allows an IPv6 or IPv4 host/router to control how incoming IPv6 or IPv4 packets are translated and forwarded by a network address translator (NAT) or simple firewall PCP can solve a number problems identified with CGN Requires CPE Router and CGN support Requires that trust boundary be extended to subscriber for port assignment
4/13/12

Challenges
o o

19

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

Summary
Significant improvement year over year
o o o

CGN improvements Content provider updates (X-Box live, Netflix) Application updates (X-Lite, uTorrent) Degradations in P2P, streaming applications Additional impacts to hairpinned DS-Lite connections

CGN experience not as good as un-NATed IPv4

o

DS-Lite and NAT444 perform similarly

o

Troubleshooting issues will be difficult More: draft-donley-nat444-impacts

20
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential. 4/13/12

Looking for Answers

CGN ARCHITECTURES

21

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

4/13/12

Architectural Constraints
Relative deployment cost (day 1 cost)
o

Ease of implementation

Impact on routing: Changes required in current routing infrastructure

o o

Traffic Engineering: Allows MSO to distribute/route traffic Load Balancing: Sharing load between different devices

Scalability: Response to increased traffic/subscriber growth Subscriber IP addressing

o o

Size of Private Subnet needed Number of Public Addresses used

Geo-location: Granularity of geolocation information obtained On-net server deployment: Ease of placement of various servers
22
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential. 4/13/12

Architecture Centralized

Head End
CMTS CMTS CGN CGN RTR

Central Location

IPv6 Internet
RTR

Head End
CMTS CMTS RTR

Core Network
RTR

IPv4 Internet

Head End
CMTS CMTS

23

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Architecture - Distributed

Head End
CMTS CMTS CGN RTR

IPv6 Internet

Head End
CMTS CMTS CGN RTR

Core Network
RTR

RTR

IPv4 Internet

Head End
CMTS CMTS CGN

24

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

4/13/12

Architecture Hybrid (Phased approach)

Head End
CMTS CMTS RTR

IPv6 Internet Regional Peering Point

Large Head End

CMTS CMTS CGN RTR

Core Network
RTR

CGN

RTR

IPv4 Internet

Head End
CMTS CMTS

25

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

4/13/12

Recommendation
A phased hybrid approach is recommended
o o

Start with Regionalized CGNs Add CGNs as needed locally as the CGN user base grows Offers ISPs easy starting point and wide reach Low impact to routing and traffic engineering Offers the most flexible scalability over time

Rationale
o o o

26

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Further Considerations
Subscriber differentiation Routing CGN Traffic Redundancy Load balancing & Scalability Server location & NAT bypass IP Addressing Geo location Logging Security Address Reputation
27
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential. 4/13/12

NAT Bypass and Server Location

Goal: Optimizing local traffic and subscriber access to advanced services Server Location (in a NAT444 environment)
o

Any internal (e.g. voice, video) or 3rd party (e.g. CDN) application servers that are placed inside the CGN should offer better performance This is less important for basic services such as web and email Dont send traffic through the CGN unnecessarily Use native dynamic routing to reach servers inside the CGN Add servers to CGN VPN, if in use

NAT444 CGN Bypass

o o o

DS-Lite: Enable IPv6 on all servers (all IPv4 goes through CGN)
28
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential. 4/13/12

IP Addressing: Public outside addressing

Number of addresses required determined by number of CGN subscribers and compression algorithm
Start low; ~8x

Where to get addresses?

o

Re-purposing existing addresses

Renumber infrastructure to IPv6 or private IPv4 Renumber customers to inside CGN addresses

o o o

Acquire new addresses transfer market? Reserve addresses now Does not need to be contiguous space
Should not be an issue at low compression ratios
4/13/12

Port restrictions
o
29
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

IP Addressing: Inside Addressing

NAT444: Use a single network-wide pool of inside addresses
o o

100.64.0.0/10 Shared Transition Space Assign local (per site) blocks out of larger pool for operational clarity, logging, the ability to insert local CGNs, and potential geo-location benefits

DS-Lite: Any addresses are acceptable and can be reused per tunnel

30

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

4/13/12

Geolocation
Local (per-site) CGNs will offer roughly equal granularity to what is available today Regional CGNs will dilute geo-location data One idea to minimize this dilution is to use separate outside pools of addresses which correspond to the per-site private subnets
o

These public pools should be loose, to borrow from the next pool if needed
Either borrow from an adjacent pool, or higher level pool

Pr1 /17
NAT

Pr2 /20
NAT

Pr3 /16
NAT

Pr4 /18
NAT

Per headend private subnets carved from inside CGN space Public subnets carved from outside CGN space

Regional CGN

Pu1 /20

Pu2 /23

Pu3 /19

Pu4 /21

Public subnets use SWIP and RDNS to identify their particular headend
31
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

4/13/12

Log Reduction Strategies

Port block reservations
o

Reduce logging up to 100x Reduces volume, but not search time See next slide

Log compression
o

Deterministic reservation
o

32

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

Proposal: Deterministic Port Reservation

draft-donley-behave-deterministic-cgn Collect inside range, outside range, compression ratio
o o o o o

Compression ratio inside/outside Inside range/compression ratio = ports/user Set aside well-known ports (<1024) & dynamic overflow range Pre-reserve port ranges for each internal IP address Allow dynamic reservation above that threshold
Remote logging only required for dynamic reservations Still need state logging locally for every active connection

Limitations:
o o
33

Requires low compression ratios Requires configuration change control process

4/13/12

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

Deterministic NAT Illustrated

DHCP

Reserved Port (e.g. 80)

CGN Device

IP 1 Reserved Pool

Static, PCP, portal, etc.

CGN Mapping Table

Subscriber 1 (DHCP STP Address 1) Subscriber 2 (DHCP STP Address 2) Subscriber 3 (DHCP STP Address 3) Subscriber 4 (DHCP STP Address 4)
34

IP 1, Port Pool 1 IP 1, Port Pool 2 Pool IP 1, Port exhausted Pool 3 IP 1, Port Pool 4
IP 1 Logging Bulk Pool
4/13/12

Required

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

Security Considerations
CGN Inside IP Space Filtering:
o o o

Block CGN routes from being advertised to and from peers Block traffic with CGN source or destination IPs at borders This filtering likely does not happen on the CGN device CGN device becomes target for DOS and other IP-focused attacks from outside your network CGN device is also bottleneck for attacks sourced from CGN subscriber networks

DOS Mitigation at the CGN:

o

35

Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

4/13/12

IP Address Reputation
IP blacklisting is more problematic with multiple subscribers behind a single outside IP
o o

All subs behind that IP are affected Any sub behind that IP can cause the listing Secure transactions (Banking, Storefronts, etc.) Email spam lists (Spamhaus, etc.) Individual website blocking (comment spam, etc.) Requires CGN logging
4/13/12

Examples:
o o o

Difficult to troubleshoot
o
36
Cable Television Laboratories, Inc. 2011. All Rights Reserved. Proprietary/Confidential.

The Big Picture and Whats Next

CONCLUSIONS

37

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12

IPv6 Offers a Better Experience than Shared IPv4

IPv4 traffic passes through ISP NAT, resulting in a diminished experience IPv6 traffic goes directly to the Internet, offering a better experience
Web, email (normal) Video streaming (degraded) P2P (dropped)

ISP NAT Device

Dual-Stack IPv4/IPv6 Network

IPv4 Internet
IPv4/IPv6 Remote Device

Home Router
(IPv4 & IPv6)

Dual-Stack Customer Device (e.g. PC, TV)

ISP Network
All IPv6 traffic (normal)

IPv6 Internet

Requires Dual-Stack (IPv4 & IPv6) PC and Home Gateway

38
Cable Television Laboratories, Inc. 2010. All Rights Reserved. Proprietary/Confidential. 4/13/12

We still have a lot of work to do!

Experience Gap Address sharing will affect subscriber IPv4 experience
IPv4 Exhaustion

Quality of Experience
Now
39

IPv6 experience will improve as ecosystem adds support

IPv6 IPv4

Time
Cable Television Laboratories, Inc. 2010. All Rights Reserved. Proprietary/Confidential. 4/13/12

In Short
IPv6 is the answer to IPv4 address exhaustion CGN can support legacy IPv4 systems for some time Deploying CGN will impact your customers
o o

P2P, VoIP, gaming, video, streaming & geolocation, etc. For many, a necessary evil to maintain IPv4 service Optimize routing, latency and jitter Reduce logging requirements Improve targeted advertising results Mitigate the impact on your customers
4/13/12

A properly designed architecture can help

o o o o
40

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

Questions?
Chris Grundemann
c.grundemann@cablelabs.com http://chrisgrundemann.com

41

Cable Television Laboratories, Inc. 2012. All Rights Reserved. Proprietary/Confidential.

4/13/12