Issues with routing

[mathourthy]

Hi. Nice to be part of this.

Im really hoping you can help me. I’ve got some issues on a network I’m responsible for at work which consists of three locations. These are all in the UK and we have offices in London, Manchester, and Bristol.

We have MPLS circuits between these three sites which has worked fine up until we recently added new firewalls in each location (a couple of days ago), and now we cannot seem to connect (or keep a connection) to any of the remote offices (I’m in Manchester fyi). We are getting disconnected after about 30 seconds and I have absolutely no idea why.

The new firewalls are Juniper SRX 300’s connected to a Cisco Catalyst 3750-24PS switch. We also have Cisco 2900 routers in each office that connect the MPLS circuits between offices. The IP ranges in use are below

London

192.168.12.0/24
Firewall IP is 192.168.12.254
Switch IP is 192.168.12.2
Cisco Router IP is 192.168.12.1

Manchester

192.168.11.0/24
Firewall IP is 192.168.11.254
Switch IP is 192.168.11.2
Cisco Router IP is 192.168.11.1

Bristol

192.168.10.0/24
Firewall IP is 192.168.10.254
Switch IP is 192.168.10.2
Cisco Router IP is 192.168.10.1

Is this enough for you to determine what is wrong, or would you need more info?

Thanks for any help you can give me.

[phenomlab]

@mathourthy said in Issues with routing:

I feel embarassed to tell you that the previous firewall was a Cisco PIX

I won’t judge you - don’t worry. At least it’s gone now. The reason as to why this wasn’t an issue before is pretty obvious now. The PIX firewall range did not offer IP Routing (until firmware v7.x which wasn’t compatible with all of the range, so may not have worked depending on the model) or Layer 3 technology, so you would have had to have a router in place to send traffic destined for other networks elsewhere.

This now explains the issue you are experiencing because you effectively have two routing devices on the same subnet, and as they are both capable of ARP resolution, the incorrect one is sending the response back. After 30 seconds, because no valid response was received, the exchange is discarded.

What we need to do here is to alleviate this issue - this is also known as “black hole” routing meaning that packets are never delivered and simply disappear. As you have a Cisco 3750 switch, you do the following (based on the “Manchester” office)

• Create a new VLAN on the Cisco switch, but have it so it is layer 2. For example

conf t
int vlan 900
no ip address
description ***** MPLS Router *****
int gi0/1
switchport mode access
switchport access vlan 900
description ***** MPLS Router *****
int gi0/2
switchport mode access
switchport access vlan 900
description ***** MPLS Router *****

By specifying no ip address on the new VLAN, we effectively make it Layer 2 so it cannot route any traffic itself.

• Change the IP address of the router so that it is in a completely different subnet - for example, 172.16.0.0/24, so you would use 172.16.0.1 255.255.255.0. This will of course attract downtime, so do it when there are no users around.
• Connect the router to port gi0/2 on the switch
• Change the default gateway on your DHCP (I’m assuming) clients so that the firewall each each location becomes the default gateway - in the case of Manchester, that will be 192.168.11.254.
• On the firewall, create a new interface and identify is as “MPLS”. Give it an IP address - for example, 172.16.0.254 255.255.255.0 and connect that into port gi0/1 on the switch
• On the router, create a last resort route of 0.0.0.0 0.0.0.0 172.16.0.254 so that it sends any traffic it cannot resolve back to the firewall
• On the firewall, create static routes for 192.168.12.0 255.255.255.0 172.16.0.1 and 192.168.10.0 255.255.255.0 172.16.0.1 so that any traffic looking to reach these subnets via the new firewall default gateway will be passed to the MPLS router and vice-versa
• On the firewall, crate policies between the LAN and this new “MPLS” interface and vice-versa so traffic can pass between the segments

Taking this route means the below

• You remove the MPLS router from the same network as the firewall - effectively, placing it behind the firewall and out of the direct reach of connected clients. This way, the only “router” that can respond is the firewall itself which is what we want.
• The firewall becomes the default gateway for all connected devices in each office network

Obviously, you’ll need to do some repatching in all offices to support this new layout.

What you will land up with once all sites are completed is the below

Hopefully this makes sense.

[phenomlab]

@mathourthy Hi, and welcome to Sudonix

At first guess looking at the above, this sounds like asymmetric routing to me. Here’s a bit more information around that issue.

https://www.networkstraining.com/what-is-asymmetric-routing/

You mentioned that the connection when it does establish lasts for around 30 seconds and then breaks? TCP/IP by default has a session lifetime of 30 seconds, so this sounds very much like the initial request never receives a response from the remote side -or much worse, another router tries to respond and because there is no match, will terminate the connection as a result. What manufacturer and model where the firewalls you were using before?

Ethernet routing uses something called ARP (Address Resolution Protocol) which means that any router on the same subnet can “hear” that request, and will attempt to resolve it - even if it is not the default gateway (which refers to the exit point on your network where traffic is sent if it cannot be resolved locally, such as internet etc). Because of this, routing devices (anything that is Layer 3 compatible) on the same subnet will send back offers of resolution and if the routing path differs from what was originally requested (for example, my default gateway is 192.168.10.1 yet the firewall responds with an offer but has the IP address of 192.168.10.254), then confusion arises and because the firewall has no match for the original request (and no return path), it discards it.

SPI (Stateful Packet Inspection) is another reason and works along the same lines, although I don’t think this is the immediate issue here.

I also note that you are using Cisco products which were EOL some time ago. This is out of scope for this particular discussion, but you should replace these - particularly if they are internet facing as without updates, they will be vulnerable.

[mathourthy]

@phenomlab thanks for the really quick response. I appreciate it.

Answers below

@phenomlab said in Issues with routing:

What manufacturer and model where the firewalls you were using before?

Given your comment about the Cisco kit being end of life, I feel embarassed to tell you that the previous firewall was a Cisco PIX that we finally threw out last week and replaced with the Junipers. The company I work for has no real concept of why you need to replace kit on a schedule and like to get the most out of something before that is on its knees and virtually dead.

I think I understand a bit more now having read the link you posted (thanks btw), and it does sound quite like what we are experiencing. I guess I just want to understand why this wasn’t an issue before but is now, and what I can do to resolve it

[phenomlab]

@mathourthy said in Issues with routing:

I feel embarassed to tell you that the previous firewall was a Cisco PIX

I won’t judge you - don’t worry. At least it’s gone now. The reason as to why this wasn’t an issue before is pretty obvious now. The PIX firewall range did not offer IP Routing (until firmware v7.x which wasn’t compatible with all of the range, so may not have worked depending on the model) or Layer 3 technology, so you would have had to have a router in place to send traffic destined for other networks elsewhere.

This now explains the issue you are experiencing because you effectively have two routing devices on the same subnet, and as they are both capable of ARP resolution, the incorrect one is sending the response back. After 30 seconds, because no valid response was received, the exchange is discarded.

What we need to do here is to alleviate this issue - this is also known as “black hole” routing meaning that packets are never delivered and simply disappear. As you have a Cisco 3750 switch, you do the following (based on the “Manchester” office)

• Create a new VLAN on the Cisco switch, but have it so it is layer 2. For example

conf t
int vlan 900
no ip address
description ***** MPLS Router *****
int gi0/1
switchport mode access
switchport access vlan 900
description ***** MPLS Router *****
int gi0/2
switchport mode access
switchport access vlan 900
description ***** MPLS Router *****

By specifying no ip address on the new VLAN, we effectively make it Layer 2 so it cannot route any traffic itself.

• Change the IP address of the router so that it is in a completely different subnet - for example, 172.16.0.0/24, so you would use 172.16.0.1 255.255.255.0. This will of course attract downtime, so do it when there are no users around.
• Connect the router to port gi0/2 on the switch
• Change the default gateway on your DHCP (I’m assuming) clients so that the firewall each each location becomes the default gateway - in the case of Manchester, that will be 192.168.11.254.
• On the firewall, create a new interface and identify is as “MPLS”. Give it an IP address - for example, 172.16.0.254 255.255.255.0 and connect that into port gi0/1 on the switch
• On the router, create a last resort route of 0.0.0.0 0.0.0.0 172.16.0.254 so that it sends any traffic it cannot resolve back to the firewall
• On the firewall, create static routes for 192.168.12.0 255.255.255.0 172.16.0.1 and 192.168.10.0 255.255.255.0 172.16.0.1 so that any traffic looking to reach these subnets via the new firewall default gateway will be passed to the MPLS router and vice-versa
• On the firewall, crate policies between the LAN and this new “MPLS” interface and vice-versa so traffic can pass between the segments

Taking this route means the below

• You remove the MPLS router from the same network as the firewall - effectively, placing it behind the firewall and out of the direct reach of connected clients. This way, the only “router” that can respond is the firewall itself which is what we want.
• The firewall becomes the default gateway for all connected devices in each office network

Obviously, you’ll need to do some repatching in all offices to support this new layout.

What you will land up with once all sites are completed is the below

Hopefully this makes sense.

[mathourthy]

@phenomlab OMG thanks so much for this, and yeah - seeing it in picture format, this does make a lot of sense. I’ll get onto this now as everyone finished at 5 (same for London and the manufacturing plants here and Bristol). Are you available here this evening (I’m assuming you’re in the UK?) in case I run into any issues? I know it’s an ask, but I feel like I’m close to getting this fixed now.

I’m going to get the IT helpdesk guys in London and Bristol to come back into the office so we can do these all at once. Wish me luck!!!

[phenomlab]

@mathourthy Yes, I’ll be here if you need me. Once you’ve moved the first site, you should still be able to ping the other locations as normal

[mathourthy]

@phenomlab Thanks. Just finished in Manchester, and everything seems to work as it did before - I also have had a connection open to a PC (RDP) in Bristol which has been connected without breaks for about 20 minutes. I had to revamp the config you provided because those switch ports were already in use, but everything else is good to go.

I created the firewall rules on each firewall (in each location) and we are passing traffic with no restrictions. Just waiting for the guys in London and Bristol to finish up and then we’ll perform more tests.

[phenomlab]

@mathourthy sounds positive so far

What sizes are your MPLS circuits and internet circuits?

[mathourthy]

@phenomlab see below (these are for both internet and MPLS)

London = 100Mbps
Manchester = 100Mbps
Bristol = 100Mbps

All are provided from the same vendor. Why ?

[phenomlab]

@mathourthy Because you could easily dump the MPLS and use IPSec tunnels between the sites. This not only constitutes a saving in terms of no longer needing the MPLS, but traffic passing across these links would be encrypted by default, which isn’t the case with MPLS networks.

The upside of MPLS is that they are 100% private networks, and are not exposed to the internet. This makes them pretty secure by default. In contrast, the downside of IPSec VPN tunnels is that they are public, and exposed to the internet, so could easily be the target of a DDoS attack.

You could also consider SD-WAN as an alternative technology.

[mathourthy]

@phenomlab Yeah, that’s a good point. To be honest, I’ve just taken over the IT stuff here, and still working my way through. I’ll need to have a look at the MPLS contract to see when it comes up for renewal, but thanks for the tip.

[mathourthy]

My guy in Bristol is having issues so need to get on a call with him. I’ll be back as soon as I can.

EDIT - seems the port was dodgy on his switch. We’ve changed it for another one and assigned it the same VLAN. he has an RDP session open to a spare machine here in Manchester.

[mathourthy]

@phenomlab some good news. Since making the changes in Manchester and Bristol, it seems that both sides are now working without any breaks (and have been for the past 20+ minutes). I’m just waiting for the changes to be made in London, so so far so good.

[phenomlab]

@mathourthy great! Thanks for the update.

[mathourthy]

@phenomlab Ok, just heard back from my guy in London. He’s finished his end, and…

IT WORKS !!! Thank you so much for everything. You are an absolute LEGEND, and a genius to boot.

[phenomlab]

@mathourthy Glad to hear it’s sorted. Let me know if you get any more issues.