I have a Solaris 10U11 1/13 system with the latest CPU installed (kernel patch 150400-38) and a gigaswift (ce) quad ethernet card. The interfaces are connected as such:
ce0 220.127.116.11/29 - ISP #1 (gw: 18.104.22.168)
ce1 10.1.1.1/21 - LAN
ce2 192.168.1.1/29 - DMZ
ce3 22.214.171.124/23 - ISP #2 (gw: 126.96.36.199)
Originally, all DMZ & LAN traffic were going out ISP #1 and my ipf & ipnat rules were working great. I recently added ISP #2 and now I'd like all traffic from the DMZ/ce2 to go out ISP#1/ce0 and all traffic from the LAN/ce1 to go out ISP #2/ce3.
My updated ipf/ipnat rules seem to be working fine but the routing is not. Since I technically now have two default routes, Solaris seems to be round-robining between them from both LAN/ce1 & DMZ/ce2 . Here's my routing table:
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ---- ------- ---------
default 188.8.131.52 UG 1 25108 ce3
default 184.108.40.206 UG 1 2316 ce0
10.1.0.0 10.1.1.1 U 1 4717 ce1
220.127.116.11 18.104.22.168 U 1 1 ce3
22.214.171.124 126.96.36.199 U 1 88 ce0
192.168.1.0 192.168.1.1 U 1 141 ce2
188.8.131.52 10.1.1.1 U 1 0 ce1
127.0.0.1 127.0.0.1 UH 2 75 lo0
If I remove the 184.108.40.206 default route, all traffic to/from the DMZ/ce2 works great. If I remove the 220.127.116.11 default route, all traffic to/from the LAN/ce1 works great. With neither, nothing works and with both, I see lots of dropped packets (ping tests) because the ICMP packets are going o ut the wrong interface.
From what I've read, I need some sort of source-based or policy-based routing solution. I've also read on MOS that some users use ipfilter as a workaround. This is the part I'm trying to figure out but not having much luck.
I tried adding these rules:
pass out log quick on ce0 to ce3:18.104.22.168 from 10.1.1.0/21 to any
pass out log quick on ce3 to ce0:22.214.171.124 from 192.168.1.0/29 to any
And since I wasn't sure if ipnat rules applied before or after ipf, I added these:
pass out log quick on ce0 to ce3:126.96.36.199 from 188.8.131.52 to any
pass out log quick on ce3 to ce0:184.108.40.206 from 220.127.116.11 to any
This didn't help. I was still getting dropped ICMP packets from the respective LAN/DMZ interfaces. I then added:
block out log quick on ce3 to ce0:18.104.22.168 from 10.1.1.0/21 to any
block out log quick on ce0 to ce3:22.214.171.124 from 192.168.1.0/29 to any
And that helped a little. Now, I don't get dropped packets within a single ping command but subsequent ping commands don't always use the correct outgoing so zero packets are returned.
Someone also mentioned populating /etc/gateways using routed but that didn't seem to help. However, I took a page from routed and also added these network routes:
route add 10.1.1.0 126.96.36.199
route add 192.168.1.0 188.8.131.52
I had these routes in place while testing the above ipf rules but I'm not sure if they were required.
At this point, I feel like I'm chasing my tail and would appreciate any ideas or if I'm doing something wrong.