KT Internet Segment Troubleshooting

Introduction:

----------------------

Internet segment is responsible for the following major services

1- myabl
2- Call center SIP
3- Active Synch (mails on mobile devices)
4- All remaining DMZ services like IPsec tunnels etc/

There are 6 steps to verify Internet segment

1- Check Physical connectivity of the following Boxes

2- Verifying end-to-end MAC Address
3- Failover status and configurations
4- Check internet links & IBGP neighbourship
5- Trace ABL live IP
6- Traffic shifting from Tw1 to Ptcl and vice versa
7- Traffic shifting from KT to KR

Step 1: Check Physical connectivity of the following Boxes

a- Ping the inside and Management IPs from your PC to check physical connectivity

Management Health Check

Name of Device Inside IPs
IPs
KT internet fpr 2120 Active 10.133.50.30 10.21.254.19 (Check with System ip Show process CPU
10.21.254.20 +
KT internet fpr 2120 Standby 10.133.50.31 Go to NMS and verify
(Check with System ip
internet Rt Pri AT 4351-01 10.133.50.51 Check ping from System and from VSS Node Status as well see
internet Rt Pri AT 4351-02 10.133.50.52 Check ping from System and from VSS the topology diagram
+ NMS Top 10 view
Internet SW-1 Outside KT 10.133.50.35 Check ping from System and from VSS
Internet SW-2 Outside KT 10.133.50.36 Check ping from System and from VSS
FMC internet and B2B 10.133.50.33 Check ping from System and from VSS

ping 10.133.50.33

Pinging 10.133.50.33 with 32 bytes of data:

Reply from 10.133.50.33: bytes=32 time<1ms TTL=63

Reply from 10.133.50.33: bytes=32 time<1ms TTL=63

Reply from 10.133.50.33: bytes=32 time<1ms TTL=63

Reply from 10.133.50.33: bytes=32 time<1ms TTL=63

Ping statistics for 10.133.50.33:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

 Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\Administrator>ping 10.133.50.35

Pinging 10.133.50.35 with 32 bytes of data:

Request timed out.

Reply from 10.133.50.35: bytes=32 time=1ms TTL=254

Reply from 10.133.50.35: bytes=32 time=7ms TTL=254

Reply from 10.133.50.35: bytes=32 time=3ms TTL=254

Ping statistics for 10.133.50.35:

Packets: Sent = 4, Received = 3, Lost = 1 (25% loss),

Approximate round trip times in milli-seconds:

Minimum = 1ms, Maximum = 7ms, Average = 3ms

C:\Users\Administrator>ping 10.133.50.36

Pinging 10.133.50.36 with 32 bytes of data:

Request timed out.

Reply from 10.133.50.36: bytes=32 time=1ms TTL=252

Reply from 10.133.50.36: bytes=32 time=1ms TTL=252

Reply from 10.133.50.36: bytes=32 time=1ms TTL=252

Ping statistics for 10.133.50.36:

Packets: Sent = 4, Received = 3, Lost = 1 (25% loss),

Approximate round trip times in milli-seconds:

Minimum = 1ms, Maximum = 1ms, Average = 1ms

C:\Users\Administrator>ping 10.133.50.51

Pinging 10.133.50.51 with 32 bytes of data:

Reply from 10.133.50.51: bytes=32 time=1ms TTL=254

Reply from 10.133.50.51: bytes=32 time=1ms TTL=254

Reply from 10.133.50.51: bytes=32 time=1ms TTL=254

Reply from 10.133.50.51: bytes=32 time=1ms TTL=254

Ping statistics for 10.133.50.51:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 1ms, Maximum = 1ms, Average = 1ms

C:\Users\Administrator>ping 10.133.50.52
Pinging 10.133.50.52 with 32 bytes of data:

Reply from 10.133.50.52: bytes=32 time=1ms TTL=254

Reply from 10.133.50.52: bytes=32 time=1ms TTL=254

Reply from 10.133.50.52: bytes=32 time=1ms TTL=254

Reply from 10.133.50.52: bytes=32 time=1ms TTL=254

Ping statistics for 10.133.50.52:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 1ms, Maximum = 1ms, Average = 1ms

KT-VSS#p 10.133.50.51

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.133.50.51, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

KT-VSS#p 10.133.50.52

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.133.50.52, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

KT-VSS#p 10.133.50.33

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.133.50.33, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

KT-VSS#p 10.133.50.35

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.133.50.35, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

KT-VSS#p 10.133.50.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.133.50.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

Step-2 Verifying end-to-end MAC Address

b- Check the arp on active fpr so that it can be verified that the systems are live and reachable

KT-INT-FPR2120-01# sh arp

KT-OUTSIDE 103.247.66.2 0000.0c07.acfa 1735

KT-OUTSIDE 103.247.66.4 7802.b166.5140 1735 // KT internet router’s interface ip

KT-OUTSIDE 103.247.66.3 7802.b166.5260 1735

KT-RDX 192.168.252.163 0050.56a0.17da 10562

{

 Lets take an example (Verify MAC address on N5k)

AT-N5K-1# sh mac address-table | inc 17da

* 116 0050.56a0.17da dynamic 10 F F Po10

Same Mac is found on N5K

KT-Oracle-Access-mgr 192.168.249.4 0050.56a0.b7c6 197

KT-Digital-Banking 192.168.249.43 0050.56a0.aee1 8971

KT-INTERNET-BANKING 192.168.50.112 0050.56bf.15ec 23

KT-FCDB-Prod 192.168.251.68 0050.56a0.2c7f 20

KT-DMZ-1 192.168.250.18 0050.56a0.df0e 2851

KT-CallCenter 192.168.250.196 0050.56ba.1bd5 0

Step 3: Failover status and configurations [if required and recommended by Network team]

ver 1.1.1.1 255.255.255.248 standby 1.1.1.2

Checking Failover Status

In good days, active standby on FMC should be look like the following
KT-INT-FPR2120-01# show failover

Failover On

Failover unit Primary

Failover LAN Interface: Failover Port-channel15 (up)

Reconnect timeout 0:00:00

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 1 of 1042 maximum

MAC Address Move Notification Interval not set

failover replication http

Version: Ours 9.12(2)151, Mate 9.12(2)151

Serial Number: Ours JAD23200E7D, Mate JAD23200E6R

Last Failover at: 02:10:18 UTC Jan 29 2020

This host: Primary - Active

Active time: 101406 (sec)

The above output will give you good idea that which device is active now and when was last failover occurred.

Step 4: Check internet links & IBGp neighbourship

Internet router pri (10.133.50.51)

Check Transworld link health

ping 110.93.219.69 so 110.93.219.70 re 500 si 500

Internet router Sec (10.133.50.52)

Check PTCL link health

ping 221.120.209.129 so 221.120.209.130 re 500 si 500

Investigation IBGP status

There should be the following neighbourships up on given routers

KT internet router Primary: 3 IBGP, 1 EBGP with TW1

KT internet router Secondary: 2 IBGP, 1 EBGP with PTCL
KR internet router Primary: 3 IBGP, 1 EBGP with PTCL
KR internet router Secondary: 2 IBGP, 1 EBGP with TW1

a- Check Neighbourship of IBGP

AT-INT-RTR4351-01#sh ip bgp summary
BGP router identifier 192.168.221.9, local AS number 58515

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.200.200.2 4 58515 1467 1471 219 0 0 22:07:11 10 //IBGP WITH KR PTCL router 1
10.200.200.41 4 58515 1439 1443 219 0 0 21:42:27 3 //IBGP with KR TW1 router 2
110.93.219.69 4 38193 16639 15472 219 0 0 1w2d 1 //EBGP with KR PTCL
192.168.221.10 4 58515 191554 191552 219 0 0 17w1d 321 //IBGP with KT Router 2

AT-INT-RTR4351-02#sh ip bgp summ

BGP router identifier 192.168.221.10, local AS number 58515

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.200.200.141 4 58515 0 0 1 0 0 1d04h Active //IBGP with KR-internet-RTR-PRI PTCL

192.168.221.9 4 58515 186849 186839 86 0 0 16w5d 35 //IBGP with KT internet RTR Primary
221.120.209.129 4 17557 169036 186065 86 0 0 16w5d //TW1 neighbourship

Step 5: Trace ABL live IP as follow

Trace the live ip by visiting tracerout.org and selecting the server there. An example server in this regard

https://hax.at/trace/trace.php

103.247.66.1
 Step 6: Traffic shifting from Tw1 to Ptcl and vice versa

If traffic need to be shifted to PTCL link at KT then you have to shut Tw1 links at both KT and KR as follows:

AT-INT-RTR4351-01 (10.133.50.51)

AT-INT-RTR4351-01#conf t

interface GigabitEthernet0/0/2

shut

KR-RTR-INT-Sec (10.128.100.36)

KR-RTR-INT-Sec#conf t

int Gi0/0

shut

If you want to revert, the traffic to TW1 no shut the Tw1 links at KT and KR routers

Step 7: Traffic shifting from KT to KR

Shifting Traffic to KR Tw1 Link

For this, you only have to shut the Tw1 link at Kt primary router

AT-INT-RTR4351-01 (10.133.50.51)

AT-INT-RTR4351-01#conf t

interface GigabitEthernet0/0/2

shut

To revert the traffic to TW1 at KT side just no shut the above port

Verifications:

Try to log in myabl from outside internet toverify the services

Appendix--A
How to configure failover

Execute the following command in exec mode to do failover and make the device active

KT-INT-FPR2120-01# failover ?

active Make this system to be the active unit of the failover pair

exec Execute command on the designated unit

reload-standby Force standby unit to reboot

reset Force a unit or failover group to an unfailed state

KT-INT-FPR2120-01# failover active

Failover configurations: (For information only)

log in 10.133.50.30

KT-INT-FPR2120-01# sh run | inc failover

failover

failover lan unit primary

failover lan interface Failover Port-channel15

failover replication http

failover link Failover Port-channel15

failover interface ip Failo