0xfeedface Tech musings and rants

I realized that I've been posting a lot about OpenSolaris. Today's no exception. I've been attempting to set up a vuln-dev lab here at work using OpenSolaris, Crossbow, and xVM. I love the things crossbow lets me do. I can virtualize the entire network stack, creating virtual switches, virtual NICs, and virtual VLANs. These last two days, though, have been a bit of a challenge as I work through what I think is a bug.

To set the stage, I have to do a bit of explaining. I have two separate etherstubs (an etherstub is a virtual switch). One etherstub is called xenswitch1, which is responsible for the 192.168.3.0/24 virtual network. The other etherstub is called xenswitch2 and is responsible for the 192.168.4.0/24 virtual network. The host has a vnic on each network, with the IP of .1. Here's the console output in case what I just wrote doesn't make sense:

root@shawn-vulndev:~# dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
nge0 phys 1500 up -- --
xenswitch1 etherstub 1500 unknown -- --
xenswitch2 etherstub 1500 unknown -- --
xenvnic0 vnic 9000 up -- xenswitch1
xenvnic1 vnic 9000 up -- xenswitch2
xvm29_0 vnic 1500 up -- xenswitch2
xvm30_0 vnic 1500 up -- xenswitch1
xvm31_0 vnic 1500 up -- xenswitch1
root@shawn-vulndev:~# ifconfig xenvnic0
xenvnic0: flags=1100843 mtu 9000 index 7
inet 192.168.3.1 netmask ffffff00 broadcast 192.168.3.255
ether 2:8:20:e6:fc:37
root@shawn-vulndev:~# ifconfig xenvnic1
xenvnic1: flags=1100843 mtu 9000 index 6
inet 192.168.4.1 netmask ffffff00 broadcast 192.168.4.255
ether 2:8:20:a3:e3:4

I have a VM on each etherstub as well. I have a Windows Server 2008 Enterprise VM on xenswitch1 with an IP of 192.168.3.3 and an Ubuntu Desktop 10.04 VM on xenswitch2 with an IP of 192.168.4.2. Both networks are fully NATed.

You would think that if I were to ping the Win2k8 VM from the Ubuntu VM, the ICMP packet would go outbound from xenvnic1 to xenvnic0. However, the bug I found is that if Ubuntu sends a packet to the 192.168.3.0/24 network, the packet goes outbound from the xenvnic0 interface. All other traffic is treated like normal and goes outbound from the xenvnic1 interface.

I haven't found a solution, yet. I hope the explanation of the issue was clear. I'm always looking for pointers in doing this better, especially in this situation.

I've been tasked with designing and implementing a set of systems to serve as a NAS and a dedicated virus scanning machine. Three systems will be involved: a Windows Server 2003 box acting as a domain controller, a Windows Server 2008 box acting as a dedicated virus scanning machine for file uploads, and an OpenSolaris NAS. The OpenSolaris NAS will be authenticating via Active Directory and serving files over CIFS/SMB.

Because of how large this project is, I decided first to test configurations in a lab. When Windows Server acts as a domain controller, it likes to take full control over the network. It likes to serve DHCP, DNS, NTP, and act as the gateway. I needed to be able to have the virtual lab, then, on its own private network. I first tried VirtualBox, since it can natively do host-based networking. However, I learned that VirtualBox's support for host-based networking is practically broken in OpenSolaris hosts. Naturally, I turned to xVM.

Prior to choosing xVM, I knew OpenSolaris's cool networking feature Crossbow could do some pretty cool things. Crossbow can simulate a virtual layer three ethernet switch and I can set up virtual NICs (VNICs) and VLANs. Using crossbow and this tutorial, I was able to set up a private network to host my lab. I won't dive into the details in how to do it, since it's laid out really nicely in that tutorial (complete with pictures, yay!). One thing it didn't discuss, however, is that in order for your VNIC configuration to persist upon reboots, you cannot use NWAM. You have to disable NWAM via svcadm disable network/physical:nwam and set up oldschool static IP configuration via /etc/hostname.[vnic] and svcadm enable network/physical:default.

To sum up, OpenSolaris mixed with xVM and Crossbow provides an amazing virtual machine and lab solution. Crossbow is so simple to use and easy to integrate with other technologies, like xVM.

Overall, Sun xVM is easy to configure and run. Setting up a virtual machine on a headless VirtualBox installation takes a lot of thought. Setting up a VM with xVM takes a single command with virt-install. In OpenSolaris Build 133, you will be able to have the VM automatically start upon booting up the host OS. I'm very glad I switched from VirtualBox to xVM. xVM appears to be stable and fast.

I have only to gripes about xVM. First, my server has only 4GB RAM installed. xVM limits my host OS to 2GB memory. Since ZFS loves to use all available memory when it's not needed by other applications, I notice that ZFS performance has degraded slightly. Second, xVM uses VNC for remote controlling VMs. VNC is an antiquated, insecure, and inefficient protocol. I prefer the RDP VirtualBox offers over xVM's VNC.

From an administrative perspective, xVM beats VirtualBox hands-down. When combined with ZFS snapshots, xVM offers a complete solution I am looking for in a VM server. xVM has a lot of potential, especially when mixed with other OpenSolaris technologies like Crossbow.

In the end, xVM wins over VirtualBox for server environments.