I have a Solaris 10U11 1/13 system with the latest CPU installed (kernel patch 150400-38) and a gigaswift (ce) quad ethernet card. The interfaces are connected as such:
ce0 18.104.22.168/29 - ISP #1 (gw: 22.214.171.124)
ce1 10.1.1.1/21 - LAN
ce2 192.168.1.1/29 - DMZ
ce3 126.96.36.199/23 - ISP #2 (gw: 188.8.131.52)
Originally, all DMZ & LAN traffic were going out ISP #1 and my ipf & ipnat rules were working great. I recently added ISP #2 and now I'd like all traffic from the DMZ/ce2 to go out ISP#1/ce0 and all traffic from the LAN/ce1 to go out ISP #2/ce3.
My updated ipf/ipnat rules seem to be working fine but the routing is not. Since I technically now have two default routes, Solaris seems to be round-robining between them from both LAN/ce1 & DMZ/ce2 . Here's my routing table:
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ---- ------- ---------
default 184.108.40.206 UG 1 25108 ce3
default 220.127.116.11 UG 1 2316 ce0
10.1.0.0 10.1.1.1 U 1 4717 ce1
18.104.22.168 22.214.171.124 U 1 1 ce3
126.96.36.199 188.8.131.52 U 1 88 ce0
192.168.1.0 192.168.1.1 U 1 141 ce2
184.108.40.206 10.1.1.1 U 1 0 ce1
127.0.0.1 127.0.0.1 UH 2 75 lo0
If I remove the 220.127.116.11 default route, all traffic to/from the DMZ/ce2 works great. If I remove the 18.104.22.168 default route, all traffic to/from the LAN/ce1 works great. With neither, nothing works and with both, I see lots of dropped packets (ping tests) because the ICMP packets are going o ut the wrong interface.
From what I've read, I need some sort of source-based or policy-based routing solution. I've also read on MOS that some users use ipfilter as a workaround. This is the part I'm trying to figure out but not having much luck.
I tried adding these rules:
pass out log quick on ce0 to ce3:22.214.171.124 from 10.1.1.0/21 to any
pass out log quick on ce3 to ce0:126.96.36.199 from 192.168.1.0/29 to any
And since I wasn't sure if ipnat rules applied before or after ipf, I added these:
pass out log quick on ce0 to ce3:188.8.131.52 from 184.108.40.206 to any
pass out log quick on ce3 to ce0:220.127.116.11 from 18.104.22.168 to any
This didn't help. I was still getting dropped ICMP packets from the respective LAN/DMZ interfaces. I then added:
block out log quick on ce3 to ce0:22.214.171.124 from 10.1.1.0/21 to any
block out log quick on ce0 to ce3:126.96.36.199 from 192.168.1.0/29 to any
And that helped a little. Now, I don't get dropped packets within a single ping command but subsequent ping commands don't always use the correct outgoing so zero packets are returned.
Someone also mentioned populating /etc/gateways using routed but that didn't seem to help. However, I took a page from routed and also added these network routes:
route add 10.1.1.0 188.8.131.52
route add 192.168.1.0 184.108.40.206
I had these routes in place while testing the above ipf rules but I'm not sure if they were required.
At this point, I feel like I'm chasing my tail and would appreciate any ideas or if I'm doing something wrong.