0xfeedface Tech musings and rants

As you may have heard, OpenSolaris is finally and officially dead. Oracle is killing off the distribution of one of the greatest open source enterprise operating systems today. Developers will still be able to run Solaris for free via Solaris 11 Express, which is supposedly debuting by the end of 2011.

Garrett D'Amore, a Nexenta employee, has written a detailed blog post about the subject. On IRC, he stated: "there is a mass exodus of talent leaving Oracle." Oracle is distancing itself from a very intelligent community. It's sad to see so much talent leave such a powerful product, but great things will happen with Illumos. Illumos will likely end up as a true fork of the OS.

I'm excited to see what Illumos brings to the table. The great talent leaving Oracle will be joining Illumos. I'm just sad it had to come to this. Hopefully, Illumos will provide a great enterprise-class OS with easy upgrade paths from OpenSolaris.

I realized that I've been posting a lot about OpenSolaris. Today's no exception. I've been attempting to set up a vuln-dev lab here at work using OpenSolaris, Crossbow, and xVM. I love the things crossbow lets me do. I can virtualize the entire network stack, creating virtual switches, virtual NICs, and virtual VLANs. These last two days, though, have been a bit of a challenge as I work through what I think is a bug.

To set the stage, I have to do a bit of explaining. I have two separate etherstubs (an etherstub is a virtual switch). One etherstub is called xenswitch1, which is responsible for the 192.168.3.0/24 virtual network. The other etherstub is called xenswitch2 and is responsible for the 192.168.4.0/24 virtual network. The host has a vnic on each network, with the IP of .1. Here's the console output in case what I just wrote doesn't make sense:

root@shawn-vulndev:~# dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
nge0 phys 1500 up -- --
xenswitch1 etherstub 1500 unknown -- --
xenswitch2 etherstub 1500 unknown -- --
xenvnic0 vnic 9000 up -- xenswitch1
xenvnic1 vnic 9000 up -- xenswitch2
xvm29_0 vnic 1500 up -- xenswitch2
xvm30_0 vnic 1500 up -- xenswitch1
xvm31_0 vnic 1500 up -- xenswitch1
root@shawn-vulndev:~# ifconfig xenvnic0
xenvnic0: flags=1100843 mtu 9000 index 7
inet 192.168.3.1 netmask ffffff00 broadcast 192.168.3.255
ether 2:8:20:e6:fc:37
root@shawn-vulndev:~# ifconfig xenvnic1
xenvnic1: flags=1100843 mtu 9000 index 6
inet 192.168.4.1 netmask ffffff00 broadcast 192.168.4.255
ether 2:8:20:a3:e3:4

I have a VM on each etherstub as well. I have a Windows Server 2008 Enterprise VM on xenswitch1 with an IP of 192.168.3.3 and an Ubuntu Desktop 10.04 VM on xenswitch2 with an IP of 192.168.4.2. Both networks are fully NATed.

You would think that if I were to ping the Win2k8 VM from the Ubuntu VM, the ICMP packet would go outbound from xenvnic1 to xenvnic0. However, the bug I found is that if Ubuntu sends a packet to the 192.168.3.0/24 network, the packet goes outbound from the xenvnic0 interface. All other traffic is treated like normal and goes outbound from the xenvnic1 interface.

I haven't found a solution, yet. I hope the explanation of the issue was clear. I'm always looking for pointers in doing this better, especially in this situation.

I've been tasked with designing and implementing a set of systems to serve as a NAS and a dedicated virus scanning machine. Three systems will be involved: a Windows Server 2003 box acting as a domain controller, a Windows Server 2008 box acting as a dedicated virus scanning machine for file uploads, and an OpenSolaris NAS. The OpenSolaris NAS will be authenticating via Active Directory and serving files over CIFS/SMB.

Because of how large this project is, I decided first to test configurations in a lab. When Windows Server acts as a domain controller, it likes to take full control over the network. It likes to serve DHCP, DNS, NTP, and act as the gateway. I needed to be able to have the virtual lab, then, on its own private network. I first tried VirtualBox, since it can natively do host-based networking. However, I learned that VirtualBox's support for host-based networking is practically broken in OpenSolaris hosts. Naturally, I turned to xVM.

Prior to choosing xVM, I knew OpenSolaris's cool networking feature Crossbow could do some pretty cool things. Crossbow can simulate a virtual layer three ethernet switch and I can set up virtual NICs (VNICs) and VLANs. Using crossbow and this tutorial, I was able to set up a private network to host my lab. I won't dive into the details in how to do it, since it's laid out really nicely in that tutorial (complete with pictures, yay!). One thing it didn't discuss, however, is that in order for your VNIC configuration to persist upon reboots, you cannot use NWAM. You have to disable NWAM via svcadm disable network/physical:nwam and set up oldschool static IP configuration via /etc/hostname.[vnic] and svcadm enable network/physical:default.

To sum up, OpenSolaris mixed with xVM and Crossbow provides an amazing virtual machine and lab solution. Crossbow is so simple to use and easy to integrate with other technologies, like xVM.

One of OpenSolaris's heralding features is a next-gen filesystem called ZFS. Managing ZFS backups could not be easier. All you need to run is zfs snapshot tank/dataset@backup. If you want to replicate or store that snapshot on another machine, you can run zfs send tank/dataset@backup > backup.zfs.

I'll be demoing how to use GPG2 to encrypt ZFS backups created with zfs send. Here's the commands I used to do it:

root@shawn-desktop:~# chmod A+user:shawn:full_set:fd:allow /datumz/
root@shawn-desktop:~# logout
shawn@shawn-desktop:~$ zfs create datumz/demo
shawn@shawn-desktop:~$ echo "this is a test" > /datumz/demo/testfile
shawn@shawn-desktop:~$ zfs snapshot datumz/demo@backup
shawn@shawn-desktop:~$ zfs send datumz/demo@backup | gpg2 -e -r "Shawn Webb" -o datumz_demo\@backup.zfs.gpg
Warning: using insecure memory!
shawn@shawn-desktop:~$ zfs destroy -r datumz/demo
shawn@shawn-desktop:~$ gpg2 -d datumz_demo\@backup.zfs.gpg | zfs recv datumz/demo
Warning: using insecure memory!
You need a passphrase to unlock the secret key for
user: "Shawn Webb "
4096-bit RSA key, ID F751C7BB, created 2010-04-27 (main key ID 5B242D25)
can't connect to `/home/shawn/.gnupg/S.gpg-agent': No such file or directory
Warning: using insecure memory!
gpg: encrypted with 4096-bit RSA key, ID F751C7BB, created 2010-04-27
"Shawn Webb "
shawn@shawn-desktop:~$ ls /datumz/demo/
testfile

So you can see how easy it is to manage and create snapshots and backups of ZFS datasets. I love that the backup files are never stored in plaintext. They're stored encrypted.

I completely ditched Linux at home. I'm now running OpenSolaris on all machines I actively own and use. The only exception is my netbook, which is running Win7 Starter. I don't really count that--I only use it for travel.

These past few days, I've been spending my time reading endless amounts of documentation and Google results figuring out why NFSv4 isn't working with me. I had NFS working a few months ago. Why would NFS stop working on a fresh install of OpenSolaris build 134?

After a few emails on the nfs-discuss mailing list, Tom Haynes, the project lead for OpenSolaris's NFS implementation, had helped me out quite a bit. He helped me figure out that the user "nobody" needs read access to the share. I never read that in any of the documentation.

Essentially, autofs tries to mount the share. Root permissions are needed to mount filesystems, causing autofs to authenticate as the root user--not the currently-logged in user as I had assumed. On the server, NFS will map UID 0 to the user "nobody". Since my user was the only one with read access, the mount naturally failed.

After giving nobody read access to the share, I could use the share like normal. Here's the minimum ACLs needed for an NFSv4 share:
$ /bin/ls -vd /tank/shares/shawn
d---------+ 13 shawn root 20 Apr 20 10:28 /tank/shares/shawn
0:user:nobody:list_directory/read_data/read_xattr/read_attributes
/read_acl:allow
1:user:shawn:list_directory/read_data/add_file/write_data
/add_subdirectory/append_data/read_xattr/write_xattr/execute
/delete_child/read_attributes/write_attributes/delete/read_acl
/write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow


Thanks Tom for helping me figure this out. I appreciate the time you spent.

Upgrading Drupal could not be easier when it's run on an OpenSolaris server. I just recently upgraded Drupal from 6.15 to 6.16. The upgrade process took less than three minutes to complete. Here's the steps I used:

1) download tarball
2) zfs snapshot
3) rm -rf
4) untar tarball
5) cp -r .zfs/snapshot/[snapname]/sites/* sites

If something bad happens, I can just revert back to the snapshot. The process would have to be changed a bit if you have custom mods to your Drupal installation. My installation is a stock install with added modules and themes. Let me know if there's a more efficient way to upgrade Drupal on an OpenSolaris server.

If you're seeing this blog entry, that means you've hit the new site! The migration is nearly complete. Let me know if there are any bugs.

UPDATE 2010-03-11 1:42 PM: Double-NATing is a big no-no. I spent hours upon hours on the phone with Comcast only to find out that my double-NAT setup was not correct.

After Oracle's complete acquisition of Sun, certain OpenSolaris users doubted OpenSolaris would survive. However, in a recent blog post, we are assured OpenSolaris will continue to be open and will receive the same, if not more, devoted development time. I'm happy to see Oracle not killing OpenSolaris. What I would love to see is for them to stop development of Linux's BTRFS and to relicense ZFS with a GPL-compatible license and port ZFS to Linux. I'm also excited for the day when KDE4 is stable and officially supported, complete with a KDE version of Gnome's TimeSlider. For now, we will have to wait and see how this turns out.

Overall, Sun xVM is easy to configure and run. Setting up a virtual machine on a headless VirtualBox installation takes a lot of thought. Setting up a VM with xVM takes a single command with virt-install. In OpenSolaris Build 133, you will be able to have the VM automatically start upon booting up the host OS. I'm very glad I switched from VirtualBox to xVM. xVM appears to be stable and fast.

I have only to gripes about xVM. First, my server has only 4GB RAM installed. xVM limits my host OS to 2GB memory. Since ZFS loves to use all available memory when it's not needed by other applications, I notice that ZFS performance has degraded slightly. Second, xVM uses VNC for remote controlling VMs. VNC is an antiquated, insecure, and inefficient protocol. I prefer the RDP VirtualBox offers over xVM's VNC.

From an administrative perspective, xVM beats VirtualBox hands-down. When combined with ZFS snapshots, xVM offers a complete solution I am looking for in a VM server. xVM has a lot of potential, especially when mixed with other OpenSolaris technologies like Crossbow.

In the end, xVM wins over VirtualBox for server environments.