Crossbow oops
I realized that I've been posting a lot about OpenSolaris. Today's no exception. I've been attempting to set up a vuln-dev lab here at work using OpenSolaris, Crossbow, and xVM. I love the things crossbow lets me do. I can virtualize the entire network stack, creating virtual switches, virtual NICs, and virtual VLANs. These last two days, though, have been a bit of a challenge as I work through what I think is a bug.
To set the stage, I have to do a bit of explaining. I have two separate etherstubs (an etherstub is a virtual switch). One etherstub is called xenswitch1, which is responsible for the 192.168.3.0/24 virtual network. The other etherstub is called xenswitch2 and is responsible for the 192.168.4.0/24 virtual network. The host has a vnic on each network, with the IP of .1. Here's the console output in case what I just wrote doesn't make sense:
root@shawn-vulndev:~# dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
nge0 phys 1500 up -- --
xenswitch1 etherstub 1500 unknown -- --
xenswitch2 etherstub 1500 unknown -- --
xenvnic0 vnic 9000 up -- xenswitch1
xenvnic1 vnic 9000 up -- xenswitch2
xvm29_0 vnic 1500 up -- xenswitch2
xvm30_0 vnic 1500 up -- xenswitch1
xvm31_0 vnic 1500 up -- xenswitch1
root@shawn-vulndev:~# ifconfig xenvnic0
xenvnic0: flags=1100843 mtu 9000 index 7
inet 192.168.3.1 netmask ffffff00 broadcast 192.168.3.255
ether 2:8:20:e6:fc:37
root@shawn-vulndev:~# ifconfig xenvnic1
xenvnic1: flags=1100843 mtu 9000 index 6
inet 192.168.4.1 netmask ffffff00 broadcast 192.168.4.255
ether 2:8:20:a3:e3:4
I have a VM on each etherstub as well. I have a Windows Server 2008 Enterprise VM on xenswitch1 with an IP of 192.168.3.3 and an Ubuntu Desktop 10.04 VM on xenswitch2 with an IP of 192.168.4.2. Both networks are fully NATed.
You would think that if I were to ping the Win2k8 VM from the Ubuntu VM, the ICMP packet would go outbound from xenvnic1 to xenvnic0. However, the bug I found is that if Ubuntu sends a packet to the 192.168.3.0/24 network, the packet goes outbound from the xenvnic0 interface. All other traffic is treated like normal and goes outbound from the xenvnic1 interface.
I haven't found a solution, yet. I hope the explanation of the issue was clear. I'm always looking for pointers in doing this better, especially in this situation.
AddToAny
Tags for Crossbow oops
Powered By:
Comments
Update
Update:
OpenSolaris is respecting the routing tables. Unfortunately, it's not applying my NAT rules before applying the routing rules. That is the problem. From what I can tell, there is no possible solution at this point. Stay tuned for more information.
Pre-Routing NAT rules
I would imagine that because ICMP is a weird protocol, it's missing your NAT rules. I know on Linux they have a PREROUTING chain that allows you to modify the packet before routing. Do they have something similar for OpenSolaris?
Can you post your NAT setup?
--Ben
nat rules
I'd have to see the ipnat.conf to verify, but generally the "nat" rules are applied as the traffic goes out the "public" interface. If you wanted traffic going from the .3 net to the .4 net (and vice versa) to be nat'd, you would have to have at least 2 more lines in ipnat.conf. This is just a *SWAG* :)
TommyTheKid
IPNat Rules
map nge0 192.168.3.0/24 -> 0/32 portmap tcp/udp auto
map nge0 192.168.3.0/24 -> 0/32
rdr nge0 0.0.0.0/0 port 3389 -> 192.168.3.4 port 3389
map nge0 192.168.4.0/24 -> 0/32 portmap tcp/udp auto
map nge0 192.168.4.0/24 -> 0/32
New IPNat Rules
TommyTheKid helped me on IRC. I owe him a big thanks. Here's the new rules:
# outbound from xenvnic0 -> nge0
map nge0 192.168.3.0/24 -> 0/32 portmap tcp/udp auto
map nge0 192.168.3.0/24 -> 0/32
# outbound from xenvnic1 -> nge0
map nge0 192.168.4.0/24 -> 0/32 portmap tcp/udp auto
map nge0 192.168.4.0/24 -> 0/32
# from xenvnic1 -> xenvnic0
map xenvnic0 192.168.4.0/24 -> 0/32 portmap tcp/udp auto
map xenvnic0 192.168.4.0/24 -> 0/32
# from xenvnic0 -> xenvnic1
map xenvnic1 192.168.3.0/24 -> 0/32 portmap tcp/udp auto
map xenvnic1 192.168.3.0/24 -> 0/32
# inbound WTS access
rdr nge0 0.0.0.0/0 port 3389 -> 192.168.3.4 port 3389