I could have accomplished this project via constantly swapping harddrives when I needed to change hypervisors, but that seemed more like accepting the problem than a solution.
My initial plan was to divide my six harddrives into 4 logical volumes at the raid controller level. Unfortunately the raid controller for servers I was using (I'm going to refrain from plugging any one vendor, at least as long as I'm operating on other people's hardware) doesn't provide this functionality. For a brief moment I flirted with the idea of cutting up the disk using partitions, until a closer look at ESXi and Xen's installation revealed they don't provide any granularity in setup of the partition tables beyond selecting an installation volume.
Instead of telling this story as it happened, I'm going to first go over the hiccups I encountered on the way. That way, incase you're like me and already started this project before doing your research, you can plan a bit before reading the whole article.
VMware Boot Bank Corruption
Not sure how I accomplished this on the first server, because I couldn't replicate it on the second. I assumed it was because Microsoft was automounting the VMware Boot Banks (they are formatted fat32). Anywho, for quite a long time I was stuck with the error "Not a VMware Boot Bank". VMware provides very little information on this error (and not applicable to my situation). Unfortunately VMware also doesn't seem to provide a way to fix a corrupted Boot Bank, so I was forced to reinstall. Afterwards I ran this command to prevent Windows from auto-mounting these volumes:
diskpart automount disable exit
Windows installing onto the wrong drives
While Microsoft is nice enough to ask you which drive you'd like to install Windows on, that doesn't mean it'll listen. Unfortunately while it will install your C: drive to the selected partition, it will install the 100MB System Reserved partition which it boots from to where it sees fit. On a system with lots of unformatted drives like mine, and where I was installing Windows to not the first volume, it chose to put the system reserved partition and the MBR for windows on the first volume set, ensuring that it would be booted automatically. I ended up fixing this by only leaving in the disks for Windows during it's installation process, then inserting the other disks after it was done.
XenServer is scared of VMFS volumes
Citrix, it's ok to be scared, but you need to face your fears. Every time I booted the Xen installer it would die when it got the point of scanning the drives with the error "Could not parse Sgdisk". From what I found via a quick google, this is due to an inability of the Xen 6 installer to handle VMFS volumes. Citrix, this is no way to convert followers away from VMware. I resolved this by making sure the disk I was installing Xen to was formatted and pulling the other disks out of the server during the installation of XenServer.
XenServer is scared of GPT volumes
Not sure if this was just from me, or why this was occuring, but I had to follow these instructions to get past a GPT error which occurred during the installation of Xen 6:
1. Boot from the XenServer 6.0.0 install CDROM.
Pretty simple.
Boot everything from Xen 6 (Syslinux 4.04)
Citrix was nice enough to include both menu.c32 and chain.c32 in their boot partition. I didn't check to see if they mounted this within their OS or not, and simply edited it out of band anyways. Since I was already done with most everything at this point, I didn't change it to use the menu config but instead left it with requiring you to enter the label at the boot prompt.
The file you want to edit this time is extlinux.conf. Again, I suggest making a backup first:
Boot everything from Bootmgr
First note, I never actually go this working. Booting Grub from Bootmgr is easy. I'm sure it's possible somehow to boot Syslinux from Bootmgr, but after accomplishing the above tasks this seemed like a trivial exercise. Still, I figured I'd list what I did.
First I pulled the mbr off each of the disks:
Then I copied the files I created to the root of the C: drive for Windows. Then within BCD edit I ran:
This seemed to boot Grub just fine, but everything else gave an error about no operating system found. I have not spent more time to troubleshoot it.
End Result
Everything available from Grub. I know it's cheesy to take a picture of a monitor, but until someone gets me some test boxes with out of band management (hint hint, Cisco and/or Fujitsu), this is how it's going to be.
Credit: Information for this solution was gathered from numerous websites in addition to independent research. In addition to the websites listed above, some of the others which provided instrumental information are:
http://www.legacycode.net/
http://www.msfn.org/board/
Boot everything from Xen 6 (Syslinux 4.04)
Citrix was nice enough to include both menu.c32 and chain.c32 in their boot partition. I didn't check to see if they mounted this within their OS or not, and simply edited it out of band anyways. Since I was already done with most everything at this point, I didn't change it to use the menu config but instead left it with requiring you to enter the label at the boot prompt.
The file you want to edit this time is extlinux.conf. Again, I suggest making a backup first:
# location mbr serial 0 115200 default xe prompt 1 timeout 50 label xe # XenServer kernel mboot.c32 append /boot/xen.gz dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M console= vga=mode-0x0311 --- /boot/vmlinuz-2.6-xen root=LABEL=root-ckxrtmdk ro xencons=hvc console=hvc0 console=tty0 quiet vga=785 splash --- /boot/initrd-2.6-xen.img label xe-serial # XenServer (Serial) kernel mboot.c32 append /boot/xen.gz com1=115200,8n1 console=com1,vga dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M --- /boot/vmlinuz-2.6-xen root=LABEL=root-ckxrtmdk ro console=tty0 xencons=hvc console=hvc0 --- /boot/initrd-2.6-xen.img label safe # XenServer in Safe Mode kernel mboot.c32 append /boot/xen.gz nosmp noreboot noirqbalance acpi=off noapic dom0_mem=752M com1=115200,8n1 console=com1,vga --- /boot/vmlinuz-2.6-xen nousb root=LABEL=root-ckxrtmdk ro console=tty0 xencons=hvc console=hvc0 --- /boot/initrd-2.6-xen.img label fallback # XenServer (Xen 4.1.1 / Linux 2.6.32.12-0.7.1.xs6.0.0.529.170661xen) kernel mboot.c32 append /boot/xen-4.1.1.gz dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M --- /boot/vmlinuz-2.6.32.12-0.7.1.xs6.0.0.529.170661xen root=LABEL=root-ckxrtmdk ro xencons=hvc console=hvc0 console=tty0 --- /boot/initrd-2.6.32.12-0.7.1.xs6.0.0.529.170661xen.img label fallback-serial # XenServer (Serial, Xen 4.1.1 / Linux 2.6.32.12-0.7.1.xs6.0.0.529.170661xen) kernel mboot.c32 append /boot/xen-4.1.1.gz com1=115200,8n1 console=com1,vga dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M --- /boot/vmlinuz-2.6.32.12-0.7.1.xs6.0.0.529.170661xen root=LABEL=root-ckxrtmdk ro console=tty0 xencons=hvc console=hvc0 --- /boot/initrd-2.6.32.12-0.7.1.xs6.0.0.529.170661xen.img label redhat com32 chain.c32 append hd0 label win com32 chain.c32 append hd1 label esxi com32 chain.c32 append hd2
Boot everything from Bootmgr
First note, I never actually go this working. Booting Grub from Bootmgr is easy. I'm sure it's possible somehow to boot Syslinux from Bootmgr, but after accomplishing the above tasks this seemed like a trivial exercise. Still, I figured I'd list what I did.
First I pulled the mbr off each of the disks:
dd if=/dev/sda of=/tmp/linux.bin bs=512 count=1 dd if=/dev/sdc of=/tmp/syslinux.bin bs=512 count=1 dd if=/dev/sdd of=/tmp/xen.bin bs=512 count=1
Then I copied the files I created to the root of the C: drive for Windows. Then within BCD edit I ran:
bcdedit /create /d “Linux” /application BOOTSECTOR
bcdedit /set {ID} device partition=c:
bcdedit /set {ID} path \linux.bin
bcdedit /displayorder {ID} /addlast
bcdedit /create /d “ESXi 5” /application BOOTSECTOR
bcdedit /set {ID} device partition=c:
bcdedit /set {ID} path \syslinux.bin
bcdedit /displayorder {ID} /addlast
bcdedit /create /d “Xen 6” /application BOOTSECTOR
bcdedit /set {ID} device partition=c:
bcdedit /set {ID} path \xen.bin
bcdedit /displayorder {ID} /addlast
bcdedit /timeout 30
{ID} should be replaced in each entry here by the ID which BCDedit gives you for that entry after you run the /create command. When I was finished the configuration looked like this:
Windows Boot Manager
--------------------
identifier {bootmgr}
device partition=E:
description Windows Boot Manager
locale en-US
inherit {globalsettings}
default {default}
resumeobject {69db61ae-5dc5-11e1-8bc1-bb2a8301fdf5}
displayorder {default}
{53d01ece-5e3e-11e1-94cf-9e5144662cd0}
{657ad556-5e3e-11e1-94cf-9e5144662cd0}
{c4a9ea4a-6a30-11e1-8e1a-f59982b714d6}
toolsdisplayorder {memdiag}
timeout 10
Windows Boot Loader
-------------------
identifier {default}
device partition=C:
path \Windows\system32\winload.exe
description Windows Server 2008 R2
locale en-US
inherit {bootloadersettings}
recoverysequence {69db61b0-5dc5-11e1-8bc1-bb2a8301fdf5}
recoveryenabled Yes
osdevice partition=C:
systemroot \Windows
resumeobject {69db61ae-5dc5-11e1-8bc1-bb2a8301fdf5}
nx OptOut
Real-mode Boot Sector
---------------------
identifier {53d01ece-5e3e-11e1-94cf-9e5144662cd0}
device partition=C:
path \syslinux.bin
description ESXi
Real-mode Boot Sector
---------------------
identifier {657ad556-5e3e-11e1-94cf-9e5144662cd0}
device partition=C:
path \linux.bin
description RedHat
Real-mode Boot Sector
---------------------
identifier {c4a9ea4a-6a30-11e1-8e1a-f59982b714d6}
device partition=C:
path \xen.bin
description Xen
This seemed to boot Grub just fine, but everything else gave an error about no operating system found. I have not spent more time to troubleshoot it.
End Result
Everything available from Grub. I know it's cheesy to take a picture of a monitor, but until someone gets me some test boxes with out of band management (hint hint, Cisco and/or Fujitsu), this is how it's going to be.
Credit: Information for this solution was gathered from numerous websites in addition to independent research. In addition to the websites listed above, some of the others which provided instrumental information are:
http://www.legacycode.net/
http://www.msfn.org/board/
