Routing problems with two interfaces

[sonicbuddha]

I run a small, low power machine with two interfaces as a file and torrent server. Both interfaces are on the same LAN. I typically assign different services to different interfaces, such as AFP, SMB, SSH to one and the Azureus to the other. This has worked fine in 8.04 but with 10.04 if I bring up the machine with both interfaces active packets cannot find their way back through my firewall (LAN traffic is fine). A traceroute stops right at my firewall. This is not exclusive to my network environment, either- I've tested with a virtual machine at work as well and packets cannot find their way back beyond the first router. I'm not the best at troubleshooting but the basic network issues so I'm at a loss what occurred between 8.04 and 10.04 that is causing this problem.

[iponeverything]

Your network layout is not clear. Is your "firewall" the same machine.

[sonicbuddha]

No, this is a single machine behind a firewall which is another machine. There is no firewall software running on the server unless there is something preconfigured that I do not know about. This is the same configuration that is running on the virtual machine that I have tested in other environments. I invite you to attempt the same configuration using VirtualBox with a server install (not desktop).

[sonicbuddha]

For argument's sake, I created a default install of Debian 5.0.4 in a virtual machine and there is no problem with two interfaces and package routing. The kernel is off by several versions, however (2.6.26), assuming that this could be at all kernel related.

Edit:
I updated to the most recent backported kernel in Debian 5.0.4, 2.6.32-bpo.5, and this issue does not exist. It seems Ubuntu isolated.

[iponeverything]

Turn off sending and listening to ICMP redirects, if that does not fix it post your routing table. It could very well be that you have two default routes. If that is case, just delete one of them and prevent its recurrence by configuring the interfaces in /etc/network/interfaces so that you can control which interface is going to get the gateway.

[sonicbuddha]

I'll try turning off sending and listening to ICMP redirects but having two default routes should not be a problem- and isn't a problem for 8.04, my Debian test install, or any other operating system I've tested with (OS X, Windows). For not specific traffic it simply picks up the first default and for applications bound to specific applications they use the first default they can- the address they are bound to. 10.04 is the only install I've ever had that choked with more than one default route (thus the post).

[iponeverything]

Nice of you to clear things up.

Two equal metric default routes on the same segment *on separate network devices* have always caused problems for me. Logically, how is that you would expect it to work correctly?

If you don't want to remove one them to test or just out of principal or because "it worked before" that is your business. Good luck.

[sonicbuddha]

I have tested with a single interface- there is no routing problem with a single interface. I have also tested by setting only one interface to to have a gateway, which also works.

I am not against testing and have done so prior to my post and am open and eager for more suggestions. My understanding of routing tables is that the first route that matches the destination is used- a second default should be, the majority of the time, ignored.

My reluctance to settle on using only one interface with a gateway is because I am reluctant to reduce functionality without explanation. Two interfaces with gateway settings should work and, prior to 10.04 as well as with other operating systems, does work.

[iponeverything]

I've had different a experience under linux and I don't have much of an understanding of the MS network stack as they do a wonderful job of hiding what exactly the kernel is doing and of making it difficult to find out. Fortunately my world has always revolved UNIX variants or Linux. I have never needed use osX in a server setting, because of the availability better FOSS alternatives.

Every time I have accidentally ended up with two identical default routes over separate interfaces on a machine, it has never worked correctly -- and in the past 17 years it has happened to me more that few times. As for "other operating systems" working, lots of vendors have/had broken or incomplete IPv4 implantations. BTW - just make sure I was not going senile, I just tested it 8.10 and 9.04 -- neither one works as configured:

0.0.0.0 10.111.111.254 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.111.111.254 0.0.0.0 UG 0 0 0 eth1

Increasing the metric on either default route does work. But, it doesn't really buy anything.

How is not having two identical default gateways on the same segment over different interfaces "reduced functionality" ?

If anything two identical default gateways on the same segment over different interfaces is not extra functionality, its just broken. Making one of the defaults a higher metric might work, except for the fact that you have different services bound to the separate interfaces.

So - If you don't mind me asking - why are using two interfaces on same segment with each offering a different set of services?

If you want to make use of both interfaces, I suggest bonding them or setting up one as hot standby for automatic failover if the primary becomes inactive.

If you really do want to find out what is happing with regard to your traffic with your unique configuration, use an instance of tcpdump on each of the interfaces and send the output to a files. Then run some tests, it should not be hard to deduce how things are getting mucked.

[iponeverything]

For argument's sake, I created a default install of Debian 5.0.4 in a virtual machine and there is no problem with two interfaces and package routing. The kernel is off by several versions, however (2.6.26), assuming that this could be at all kernel related.

Edit:
I updated to the most recent backported kernel in Debian 5.0.4, 2.6.32-bpo.5, and this issue does not exist. It seems Ubuntu isolated.

Let me see the routing table for this and the output from ifconfig, just to make sure we are the same page.