VMware Home Lab - Save money with virtual Vyatta router and virtual ZFS storage system

I built VMware home lab like any other VMware gurus that loves to tinker stuff at home  It helps me to to learn new products and test my Powershell scripts without destroying production environment at work. :)

Below are my lab equipment hardware list:

ESXi1
Dell T110 II Power Edge server
E31220 3.1Ghz Quad Xeon processor
16GB of RAM
Five Gigabit Ethernet, including on-board NIC
240GB SATA HD 7200rpm 

ESXi2
Dell T110 II Power Edge server
E31220 3.1Ghz Quad Xeon processor
16GB of RAM
Five Gigabit Ethernet, including on-board NIC
240GB SATA HD 7200rpm

ESXi3
Dell T110 II Power Edge server
E31220 3.1Ghz Quad Xeon processor
16GB of RAM
Five Gigabit Ethernet, including on-board NIC
Perc H200 Raid controller  (3)1TB 7200rpm drives
240GB SATA HD 7200rpm

HP Procurve 1810G-24 Gigabit managed switch

What is so great about Dell T110 II? It is on VMware HCL.  nuff said.

Notice I don't have Router or SAN storage unit.  I used Vyatta virtual router for all my routing needs and virtual storage system for my storage.  Vyatta router is powerful virtual router that you can download as virtual appliance or install it on Linux for FREE.  At home, I have "router on the stick" configuration.
For my shared storage, I went with Solaris VM configured with ZFS.  I didn't want to spend more money on hardware storage system so I bought H200 Perc raid card with three 1TB drives and installed it on my ESXi host server.  ESXi server does not see the raid card because it is not supported.  But, your VMs will see the card with VMDirectPath.  Solaris VM uses VMDirectPath mode to access Perc H200 raid card.  Perc H200 raid card is connected to three 1TB hard drive.  With ZFS, I have NFS, SMB, and ISCSI storage running.  It also supports storage snapshots, de-dupes and replication with fantastic performance. If you have more of these ZFS VMs and raid cards, you can setup SRM in your home lab.  How cool is that?  It has all the enterprise features for FREE.  Just remember to check your CPU before you invest your money in virtual storage.  VMDirectPath only works wtih Xeon CPUs.  I am not sure about AMD processors...

You can learn more about ZFS configuration from the link below. Everyone is raving about it.
http://hardforum.com/showthread.php?t=1573272

Vyatta Router

Virtual Storage System

My Lab 

PLINK PLINK! - SSH commands over Powershell

Have you ever wanted to automate SSH commands using powershell? PLINK is the way to do it.  On Powershell script, you can use plink.exe command to pass commands to SSH session.  This opens the door to many possibilities.  Most IT server manufacturers use SSH sessions to manage the hardware via SSH command line. 
I am familiar with HP server devices.  On HP servers, you can use SSH command line to connect to iLO and mount CD images via HTTP.   You can use this technique to automate your ESX host server firmware update or even build ESX host server from customized CD ISO.

Example:
Scenario: I am a server administrator that wants to update firmware on 12 ESX host servers in the cluster.  I want to fully automate the update process with Powershell/PowerCLI script.  I do not want any VM downtime.  I do not want endless mouse clicks...

Requirements:

1. HP licensed iLO and IP address
2. Fully automated DRS cluster
3. PLINK.EXE ver 0.60_q1.129
4. Webserver with ISO image configured with correct MIME type

Solution:

New HP firmware update DVDs are automated by default.  You can take advantage it to automate the process

1. Get hosts in the cluster and use ForEach loop to put the server in maintenance mode one at a time.
2. Use the plink command in the script to mount the firmware update ISO image on the ESX host.
3. Reboot ESX host and Firmware DVD will kick in to automatically boot from the ISO.  It will update the firmware automatically.
4. When ESX host comes back from disconnected status, take it out of maintenance mode and put next server in to maintenance mode.
5. Rinse and repeat and you will have hands free firmware update automated script.

 So, how do you use the PLINK command to connect the ILO CD?

VMTools Update - Many ways to cut it.

Updating VMtools can be pain the a$$.  We have 6000+ VMs and it is challenging to keep all the VMTools updated.  There are many ways to update it: Script it, do it manually, use Update Manager, or use "Check and upgrade vmtools during power cycling" settings in VM option.  Thankfully, we have maintenance window for many of our VMs.  I like to take advantage of these windows reboot windows to update the vmtools.  Note that this option only works on VMs with older vmtools installed.  It will not install VMtools on VMs that does not have VMtools running.

There are so many VMs out there and I can't be selecting this options one by one.  That's where PowerCLI script comes to the rescue.

Script below will add the option:





Script below will remove the option:

Get-Snapshot - Are we there yet?

At my work, we are in the middle of performing datastore upgrade.  I am using storage vMotion to migrate the VMs live.  For storage vMotion to work you need to make sure CD is not attached and there are no snapshots on the VM.  On my storage vmotion script, I initially used Get-Snapshot to query the snapshots on the VM.  It took very, very long time to query the information and it was too slow for my taste.  Thankfully, snapshot info is available in API and I got the snapshot information quicker after I modified the script.

Get-member is my favorite command for any query.  It shows you additonal extensions from most values from Get-**** commands.  If you drill down in extensiondata, you will find wealth of information about the value. 

$vm = get-vm vm_name
$vm | Get-member

And finally, command below will give information if the VM I am looking at has snapshot.

$vm.extensiondata.snapshot

You can loop it using ForEach command to scan through the VMs.  It will be ALOT faster than using Get-SnapShot command.

Host Profile update made easy with PowerCLI GUI

You don't usually associate Powershell with GUI. However, powershell is extremely .NET friendly.  You can build pretty impressive Windows GUI interface with it. But, it is not easy writing the GUI code line by line in Powershell. That is why I like to use Sapien Primary Forms. You can build GUI inteface using the graphic utility and it will write the powershell script for you.  How awesome is that?

Script below is one of the GUI interface script that I wrote.  It checks host profile in the cluster and applies the host profile on single ESX host at a time.  It does it by putting one non-compliant host in to maintenance  mode by evacuating the VMs using DRS and vMotion.  Once the host server is place in to maintenance mode, it applies assigned host profile and takes the server out of the maintenance mode once it is completed.  I hope to post more of my GUI scripts in the future.

You can grab my script from here.

Automated ESX Virtual Machine VLAN test

At work we moved away from standalone servers to HP blade servers with two 10Gbe pass-thru configuration. I recently built 5 ESXi4.1 hosts in to our growing number of ESXi clusters. Before putting the server in to production, we let it bake in the sun for about one to two weeks. After baking is over, we test VLAN and fault tolerance on NICs. After countless clicks to Vmotion a VM_HLPER VM through ESXi hosts, I decided to make script for it. Below is the script for my automated VLAN test using VM_HELPER machine. VM_HELPER is a small VM that helps administrators run tests like this. VM_HELPER should have access to all VM VLANs with IP address.
Be sure that ESX servers are not being used in production. The script will isolate each vmnic from vSwitch0.

NFS mapping via IP or DNS?

In the place that I work, we use NFS for are datastore mounts.  We use DNS to mount our datastores (I disagree with this approach).  I was building new servers on a new physical plateform with new network configuration.  Because of this reason, I didn't want to use existing host profile to configure the server.

Since datastore mounts will stay the same, I wanted to copy datastore mount from existing ESX host servers to new ESX servers that I built.  I was being lazy and searched for the PowerCLI script.  Google took me to ic-freak.nl PowerCLI script that will copy datastore mounts from reference hosts to the new hosts.  The problem with the script was that it will only mount the datastore using IP although reference host had DNS based mounts.


vSphere will not think IP mounted and DNS mounted datastores are same even if they point to same datastore and folder.  vMotion will fail if you try to vmotion a VM between them.  You must either mount all datastores via IP or DNS, not both.

Below is the script without using Get-View statement.  Without get-view, scripts are little bit faster using .extensiondata.

   
 $viserver = Read-Host "Enter VI server"  
 $refhost = Read-Host "Enter Reference host server"  
 $cluster = Read-Host "Enter Cluster name that has new hosts"  
   
 Connect-VIServer $viserver  
 $REFHOST = Get-VMHost $refhost  
 $NEWHosts = Get-Cluster $cluster | Get-VMHost | sort name  
   
 foreach($nfs in (Get-VMhost $REFHOST | Get-Datastore | Where {$_.type -eq "NFS"} )){  
   [string]$remotePath = $nfs.extensiondata.info.nas.remotepath  
   [string]$remoteHost = $nfs.extensiondata.info.nas.remotehost  
   [string]$shareName = $nfs.Name  
        
      Foreach ($NEWHost in $NEWHosts) {  
        $NEWHOST | Get-Datastore | Where {$_.Name -eq $shareName -and $_.type -eq "NFS"} -ErrorAction SilentlyContinue  
        If ((Get-VMHost $NEWHOST | Get-Datastore | Where {$_.Name -eq $shareName -and $_.type -eq "NFS"} -ErrorAction SilentlyContinue )-eq $null){  
          Write-Host "NFS mount $shareName doesn't exist on $($NEWHOST)" -fore Red  
          New-Datastore -Nfs -VMHost $NEWHost -Name $Sharename -Path $remotePath -NfsHost $remoteHost  | Out-Null  
        }  
      }       
 }