Linux P2V

Introduction

P2V Linux migrations are a combination of science, art and luck.

P2V stands for Physical to Virtual. In other words, it is the process or procedure of moving a running system (operating system and everything installed) from a physical machine to a virtual machine.

This page describes some of the (un)usual steps necessary to convert a Linux system into a virtual machine running under VMware ESXi Server. The same steps should mostly apply to other virtual platorms/hypervisors like VirtualBox, KVM and QEMU.

The focus of this P2V explanation is on Red Hat and CentOS guests as they are not only supported, but benefit from kudzu and rescue disk capability built-in. Other distributions can of course be converted but the exact steps will vary.

So, let's get started.

What to use (or not)

The following software products claim to do P2V, but in fact do not seem to support Linux, so YMMV

• Virtual Server Migration Toolkit
• EZP2V

These products do support Linux in some way...

• VMware Converter

VMware converter will work, however any options such as resizing the disks and post migration configuration are greyed-out. This is due to the fact that it just does a raw block-by-block copy of the source disk. It is most useful if your target for migration is ESX 3.x

• Platespin PowerConvert?

PowerConvert works (somewhat) with Linux but it does not support LVM and in my experience is an unreliable product with lackluster support.

• liveview looks promising, however it only runs on Windows (due to dependency on VMDK disk mounter) and works with vmware-server, not ESX server. It has "limited" Linux support.

These products or methods offer full support for Linux...

• UltimateP2V appears to be worthy of consideration.
• Good ol' dd + netcat, followed by rescue disk of some kind (to fix the modules and make a new initrd).
• MondoRescue - whilst primarily a backup/restore utility it lends itself extremely well to P2P/P2V migrations

Preparation

You will want to have the necessary tools in place as well as some calculations. Consider the follow aspects of your system.

• How much physical RAM? Is it over or under-utilized?

• How much Swap space and where?

• Disk type - IDE or SATA and the disk device will be /dev/hda, SCSI will be /dev/sda. You may also have multiple disks (hdb, sdb, etc).

• Disk size - use a command such as sudo sfdisk -s. Blocks are in 1KB units, do the math to figure out the equivalents in MB by dividing by 1024 and in GB by dividing by 1024 again.
  • Example:

jetson:~> sudo sfdisk -s
/dev/hda:  39070080
total: 39070080 blocks

(39070080/1024) = 38154.375 MB
(38154.375/1024) = 37.260 GB

• Partition layout -know exactly the disk partitions, sizes and filesystem types. This can be gleaned from the output of `fdisk -l /dev/sda` and the content of /etc/fstab. Understand that a disk partition is not the same as the filesystem installed on it. Some disk partitions don't have a filesystem at all, such as those used for swap, LVM or software raid (md).

• Rescue disk - this may be necessary for the recovery of the system once the disk data has been converted over. Conversion puts the system into a "new" environment of emulated devices, and some cases kudzu will not quite get you there or won't even be available (kudzu is a Red Hat software, not normally found on other Linux distributions).

• Knowledge of destination environment. For instance, see http://www.vmware.com/pdf/GuestOS_guide.pdf for vi3.

• ISOs - you will want to have the following ISOs for easy access and to map to the CDROM device in the guest.
  • Install disc #1 for the Red Hat or CentOS version of the source system
  • Knoppix (recommended) or System Rescue CD

Preparing the source system

Take the time to consider and perform the following tasks while the source system is still running in its native state.

• Zero-out each of the disk partitions... this can speed up the data transfer later on. e.g.

dd if=/dev/zero of=/usr/bigfile; rm -f /usr/bigfile

• Disable any services you don't think will be necessary after the conversion, such as system-management agents (think Dell OMSA or IBM Director) and ntpd. Also open-vm-tools, cloud-init.
• Install any packages and services that may ease the transition, like virtio drivers.
• Purge out old logfiles, scratch files in /tmp and unnecessary software.
• Cleanup old/extraneous kernels. You will likely want to end up with just 2 kernels, the latest Non-SMP kernel and a previous one.
• Building a new initrd with the mptscsih (RHEL/CentOS) or BusLogic (RHEL3/CentOS3, RHEL2/CentOS2) SCSI driver loaded, this may save you from having to boot into linux rescue mode after the conversion.

For RHEL4/CentOS4, add --with=mptscsih For RHEL3/CentOS3 and earlier, use --with=BusLogic

 mkinitrd -v -f --with=BusLogic /boot/initrd-`uname -r`.img `uname -r`

Boot your system with Knoppix or System Rescue CD. The state we want is an quiesced hard drive(s) and some network connectivity. NOTHING should be running/writing to the hard drive(s). It will likely come exposed as the device file /dev/sda or /dev/sdb. Check with `dmesg` to see the drive letter assignment. Verify with `fdisk -l /dev/sda` it's the RIGHT drive.

Optional step: run md5sum /dev/sda and record the resulting hash. Usually the last 6 characters will suffice. This can take awhile but gives you a fingerprint of the hard drive data that you can use later to verify the integrity of the data after transferring to the target system. '

Preparing the target system (VM)

create a VM with the following parameters:

• Operating System: Linux. You can be more specific on vmware-server or ESX, such as Red Hat Enterprise Linux.
• Disk: slightly larger than source-system (see below). Create same number of disks as exist on the source system.
• Network (NIC): Use vlance if given the choice, can be upgrade to vmxnet later when vmware-tools is installed.
• CDROM: assign to either Knoppix ISO or System Rescue CD ISO

Boot the target system (I enter knoppix 2 at the boot prompt) and verify the disk(s) are recognized using sfdisk -s. Also verify the network is up using ifconfig eth0. You should have an IP address assigned to eth0 via DHCP or static. Now try pinging the source system e.g. ping 10.4.1.2

Network Acquisition (Disk Cloning)

This is where we transfer the bits from drive A on the source to drive B on the target. The process is functionally very similar to a network acquisition often used in the field of computer forensics.

For our part, a simple example will show how to clone the bits from a single drive: /dev/sda

You'll need to know the IP address of your target-system, which can be learned from ifconfig -a.

These commands can be used to clone the blocks to the target-system disk. I assume you have netcat (nc) installed on the source system.

First, run this on the target system

nc -l 9001 | gzip -d - | dd of=/dev/sda

Then run this on the source system

dd if=/dev/sda | gzip -c - | nc <target-system-ip> 9001

In the real-world, repeat the above process as necessary for the remaining disks. Notice the is used in the pipeline to compress/decompress on the fly.

This is the likely the slowest part of the process. Unfortunately, dd does not show a progress meter. dd will, however, display it's progress in blocks if sent the -USR1 signal see dd's info page for details. Other tools like dcfldd or dd_rescue might be more helpful in printing out a progress bar.

The network transfer speed is an important consideration when planning your scheduled outage. You may want to run some tests before your P2V, with a smaller set of data (10GB?) to get an estimate of your throughput, then run the numbers to figure out how long it will take to do the entire drive(s). This can be done while the system is still online. Also, consider the tip above about zero-filling your disk partitions beforehand that can save a lot of time since it compresses well.

Another very nice CLI tool to measure your network throughput at the TCP (OCI layer 4) is iperf.

Once your drive(s) have been bit-copied over the network to the target, shutdown your target system and remove the virtual CDROM or ISO mapping.

Optional step: run md5sum /dev/sda and verify the result matches what you saw earlier.

Extras

This is a perfect time to make some adjustments if you want to be clever about your disk & paritition sizes, the following may come in handy. Verify the partitions

fdisk -l /dev/sda

Check a filesystem.

e2fsck -f /dev/sda1

Align ext[23] filesystem size with (new) partition size of sda1

resize2fs -p /dev/sda1

First-boot

Assuming you got this far, the next step is to immediately shutdown the system again. You'll want to re-assign the NIC to vmxnet and assign the vmware-tools ISO to the CDROM (path: /usr/lib/vmware/isoimages/linux.iso). Boot up into single-user mode (at grub prompt hit e, select kernel line, e, append "single" to the line, then hit b). Install the vmware-tools (detailed elsewhere) which should get you the vmxnet driver module. Adjust network settings now! Cleanup and reboot. You should be 98% there. Congratulations!

NOTE: I had to rename /etc/rc3.d/S19vmware-tools to /etc/rc3.d/S09vmware-tools to "fix" my network bootup sequence.

On the second reboot, the kudzu command will run and (may well) deal with the remaining hardware changes.

When kudzu runs, it recognizes that certain devices (Broadcom NICs) are no longer there while others (LSI Logic card, pcnet32 NIC) had been added. Usually it is easiest to just accept what kudzu tells us & fine tune later.

See When things go wrong below if you don't get back to a login: prompt.

When things go wrong

If kudzu does not get you back to a login: prompt, the next step is to boot with the rescue disk. This entails mapping the install cd#1 ISO file to your CDROM device using the MUI or vmware-server-console. Make sure the VM BIOS is also set to use your CDROM in the boot order before the hard drive(s).

Once booted, type linux rescue at the boot prompt and shortly thereafter you will be able to type chroot /mnt/sysimage to get at your disk partitions, which should automatically be mounted there.

The first thing to look at is /etc/modules.conf (RHEL3/CentOS3) or /etc/modprobe.conf (RHEL4/CentOS4). Make sure the appropriate SCSI driver is listed, either BusLogic or mptscsih (based on what you configured this VM to use and the recommendations above).

alias scsi_hostadapter mptscsih
#or
alias scsi_hostadapter BusLogic

Also, take note of the eth0 setting, which should be either pcnet32 for the vlance device, or vmxnet for the vmxnet device. After you install the vmware-tools with the vmxnet device assigned it should be configured automatically.

alias eth0 pcnet32
#or
alias eth0 vmxnet

If you went the route of cloning individual partitions instead of the entire disk(s), it may be necessary to clone the MBR. This will be evident if you try to boot from the drive and get the message "No operating system found". The process is described here.

Tying up loose ends

Consider the new state of the system... Disable daemons you don't need like smartd, pcmcia and the like. Do you really need to run NTP anymore? Read VMware's timekeeping whitepaper, set tools.timeSync="TRUE" in the .vmx file and add clock=pit or clocksource=acpi_pm to the grub kernel line. Consider an optimized kernel or adding divider=100 (to newer 2.6 kernels).

See Also

• ESX Server 3 Systems Compatibility Guide (PDF)
• ESX Server 2.x Systems Compatibility Guide (PDF)
• Hard Disk Cloning
• UbuntuonVMware
• Wonders of 'dd' and 'netcat': Cloning OS harddrives
• The Sleuth Kit Informer
• dd_rescue - looks like a GREAT alternative to dd for P2Ving systems with failing hard drives. (It's called gddrescue on Ubuntu). Now available as GNU ddrescue from [1]
• dcfldd - another dd alternative that can update the user of its progress in terms of the amount of data transferred and how much longer operation will take.
• MondoRescue for P2P / P2V migrations (PDF)