Category Archives: ESXi

vSphere 5.5 Update 2 Released

VMware vCenter Server and ESXi 5.5 Update 2 has just been released  with a number of fixes and additional features such as vCenter Server database support for Oracle 12c, Microsoft SQL Server 2012 Service Pack 1, and Microsoft SQL Server 2014 in vCenter Server and support for ESXi hosts with 6TB of RAM in the ESXi 5.5 update 2 release! For the full list, check here.

 

One particular change was a bit buried in the release notes and was not mentioned in the blog post, which tries to say it’s not a major change. However, for VMware administrators, this might be the most important change in Update 2 and the reason they will upgrade; In Update 2, you will be able to edit virtual machines with hardware version 10 again!

For more details… refer http://blogs.vmware.com/vsphere/2014/09/vsphere-5-5-update-2-ga-available-download.html

Advertisements

Install Emulex CIM Provider in VMware ESXi

As a part of VMware vSphere administration, you all will came across this term “CIM Provider” let see what it is …

The Common Information Model (CIM) system: CIM is the interface that enables hardware-level management from remote applications via a set of standard APIs.

The CIM is an open standard that defines how computing resources can be represented and managed. It enables a framework for agentless, standards-based monitoring of hardware resources for ESXi. This framework consists of a CIM object manager, often called a CIM broker, and a set of CIM providers. CIM providers are used as the mechanism to provide management access to device drivers and underlying hardware.

Hardware vendors, which include both the server manufacturers and specific hardware device vendors, can write providers to provide monitoring and management of their particular devices. VMware also writes providers that implement monitoring of server hardware, ESX/ESXi storage infrastructure, and virtualization-specific resources. These providers run inside the ESXi system and hence are designed to be extremely lightweight and focused on specific management tasks.

The CIM object manager in ESXi implements a standard CMPI interface developers can use to plug in new providers. The CIM broker takes information from all CIM providers and presents it to the outside world via standard APIs, including WS-MAN (Web Services-Management). Figure 3 shows a diagram of the CIM management model.

So the Hardware vendors like Brocade, HP, DELL, IBM, EMC, Qlogic, Emulex etc will provide these modules and we have to install in the ESX/ESXi hosts.

 

image

Now let’s see how we can implement Emulex CIM in the vSphere infra. There are 3 main components in the Emulex Software solution for device management in the vSphere environment.

1- Emulex CIM Provider : This we have to install on the ESX/ESXi hosts, it will be available in offline bundle in ZIP format or in the VIB format.

2- Emulex OneCommand Manager  (OCM) : This we can install in any windows virtual machine or in the vCenter server itself

OCM for VMware vCenter Server and Emulex CIM provider for ESX/ESXi host is free to download from the Management tab on the following pages.

3- Emulex vCenter server plugin : Once every thing configured and ready, you can see the OCM plugin in the vSphere console and just enable the plugin.

Installation Steps

1- Download the corresponding CIM provider for the ESXi based on your version, in my case it is vSphere 5 update 2 and download the file “CIM Provider Package 3.8.15.1” and extract the ZIP file.

http://www.emulex.com/downloads/emulex/drivers/vmware/

http://www.emulex.com/downloads/emulex/drivers/vmware/vsphere-50/management/

The release notes and user manual are there for more information

2- Upload the offline bundle ZIP file (VMW-ESX-5.0.0-emulex-cim-provider-3.6.12.1-01-offline_bundle-799300.zip) to any of the Datastore via Putty or vSphere Client Datastore browser, to an ESX/ESXi host.

image

3- Login to the ESXi shell via SSH, and first check the files are copied correctly and their location

~ # cd /vmfs/volumes/Templates/emulex/

/vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex #

/vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex # ls -l

-rw——- 1 root root 7629804 Mar 5 2013 VMW-ESX-5.0.0-emulex-cim-provider-3.6.12.1-01-offline_bundle-799300.zip

/vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex #

4- Now install the offline bundle and you will get a SUCCESS message if it went properly.

/vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex #
/vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex # esxcli software vib install -d /vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex/VMW-ESX-5.0.0-emulex-cim-provider-3.6.12.1-01-offline_bundle-799300.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true
VIBs Installed: Emulex-Corporation_bootbank_emulex-cim-provider_3.6.12.1-01
VIBs Removed:
VIBs Skipped:
/vmfs/volumes/50bd9d4d-adf00e08-78e9-002655e66551/emulex #

 

image

4- Now reboot the ESXi host and you can see the Emulex Hardware details in the vSphere client

image

ESX/ESXi – Administration TIPS

How to disable or Stop the CIM agent on the ESX/ESXi host ?

Note: The CIM agent is the process providing hardware health information. Disabling this service will disable the hardware health status.

To disable the CIM agent on an ESXi host: Log in to the ESXi shell as the root user.

    chkconfig sfcbd-watchdog off
    chkconfig sfcbd off
    /etc/init.d/sfcbd-watchdog stop

    Note: Changing the chkconfig disables the sfcbd service and is persistent across reboots.

To re-enable the CIM agent on the ESXi host, run these commands:

    chkconfig sfcbd-watchdog on
    chkconfig sfcbd on
    /etc/init.d/sfcbd-watchdog start

    Note: To check the status of the agent on ESXi, run the below command.

    /etc/init.d/sfcbd-watchdog status

For troubleshooting purpose you can use the below commands to restart the CIM related services in ESXi host

#/etc/init.d/wsman restart
#/etc/init.d/sfcbd-watchdog start

Disabling a Single ESXi CIM provider when it fails or is unstable.

To see the CIM providers installed on your ESXi host:

  1. Log in to the ESXi shell as the root user.
  2. Run the command:
    esxcli system settings advanced list | grep CIM
  3. You see output similar to:
    Path: /UserVars/CIMEnabled
    Description: Enable or disable the CIM service
    Path: /UserVars/CIMLogLevel
    Description: Set the log level of the CIM Service
    Path: /UserVars/CIMWatchdogInterval
    Description: Set the watchdog polling interval for the CIM Service
    Path: /UserVars/CIMvmw_sfcbrInteropProviderEnabled
    Description: Enable or disable the CIM vmw_sfcbrInterop provider
    Path: /UserVars/CIMvmw_hdrProviderEnabled
  4. Disable a CIM provider by running the command:
    esxcli system settings advanced set -o /UserVars/CIMProviderName -i 0
    Note: To re-enable the CIM provider, run the command:
    esxcli system settings advanced set -o /UserVars/CIMProviderName -i 1
  5. To allow the changes to take effect, restart the SFCBD agent by running the command:
    /etc/init.d/sfcbd-watchdog restart

I will post a details blog regarding the use of OneCommand manager

ESXi Storage Device Naming Convention for a LUN : Part-1

This blog post came as a result of my VMware community interaction, so the questions are simple !!

1- What is the need for such a Storage Device Naming Convention for a LUN and the theory behind this.

2- Who is responsible for assigning an Unique Storage Device Name for the LUN in an ESX/ESXi host.

3- For a LUN why you need an unique and same LUN ID across the ESX/ESXi hosts in a cluster.

4- How an ESX/ESXi host can uniquely identify a LUN in a Storage Area Network.

5- What are the different types of naming standards or convention for a LUN in an ESX/ESXi host.

As we all know, for the ESX/ESXi hosts & clusters we have to create/present LUN’s from the Storage Array, to get the features like VMotion,HA,DRS etc. Now let’s see the answers for the above:

1- What is the need for such a naming convention and the theory behind this.

The need for a standard

But here comes a basic problem. If I can expose the same LUN to one or more machines, then how could I address it? In other words, how can safely I distinguish between one LUN and another? This seems to be a really trivial problem. Just stick a a unique GUID to each LUN and you are done! Or, stick a unique number. Or… a string… but hold on, things are not that easy. What if storage array maker ABC assigns GUIDs to each LUN and another vendor assigns 32-bit numbers? We have a complete mess.

To add to the confusion, we have this other concept – the serial number attached to a SCSI disk. But this doesn’t work all the time. For example, some vendors assign a serial number for each LUN, but this serial number is not guaranteed to be unique. Why, some SCSI controllers are even returning the same serial number for all exposed LUNs!

Every hardware vendor had a more-or-less proprietary method to identify LUNs exposed to a system. But if you wanted to write an application that tried to discover all the LUNs you had a hard time since your code was tied to the specific model of each array.  What if two vendors had a conflicting way to assign IDs to LUNs? You could end up with two LUNs having the same ID !!

We all know the Storage devices, I/O interfaces, SAS disks are basically used to send and receive Data by using SCSI commands and they all has to follow the SCSI (Small Computer System Interface) standards.

T10 develops standards and technical reports on I/O interfaces, particularly the series of SCSI (Small Computer System Interface) standards. T10 is a Technical Committee of the InterNational Committee on Information Technology Standards (INCITS, pronounced “insights”). INCITS is accredited by, and operates under rules that are approved by, the American National Standards Institute (ANSI).

T10 operates under INCITS and is responsible for setting standards on SCSI Storage Interfaces, SCSI architecture standards (SAM), SCSI command set standards.  As per T10, SCSI Primary Commands – 3 (SPC-3) contains the third-generation definition of the basic commands for all SCSI devices. As of now all the major Storage array vendors like EMC, Netapp, HP, DELL, IBM, Hitachi and many others follows these T10 standards, all these arrays follow the SPC-3 standards during the LUN creation, presentation, and communication to the hosts etc. Similarly the ESXi storage stack and other latest Operating systems also uses these standards to communicate to the Storage array, Access the luns etc.

So in short these are industry standards and vendor neutral so that ISV, OEM and other software/hardware vendors can develop solutions and products inside single frame work.

2- Who is responsible for assigning an Unique Storage Device Name for the LUN to an ESX/ESXi host.

There are 2 people assigns and maintain an unique name for a LUN one is the Storage Array and other is the Host, both assigns and maintains at their own level.  But what ever luns created and given from a Storage it will be unique, and it is the responsibility of the Storage to maintain the Uniqueness the array uses the T10/SPC-3 standards to maintain the Uniqueness.

That is from the SAN when we create a LUN with a LUN ID the SAN itself make sure it is unique, and we will give LUN name to understand easily. Once that LUN is presented the ESX/ESXi host will make this volume unique with UUID, and in particularly with ESX/ESXi it has different types of multiple naming conventions and representations.

So the Storage array is responsible for this and the ESX/ESXi just uses the LUN, but they follow the guidelines of T10/SPC-3 standards to maintain the Uniqueness.

Veeam Upgrade from version 6.5 to 7

As you all know the pioneer in the Virtualization backup technology, has released their latest version Veeam Backup and Replication 7. This is a major release, there are lot of new features that I will discuss in my next blog. I have just done an upgrade for my client, to be very honest, I really saw the power of Veeam simplicity and easiness. From an administrator and organizational point this matters a lot. The amazing fact is, it took below 20 minutes to upgrade with few mouse clicks !!!

The Steps are given below;

1- Download the latest version, it is in ISO format extract or burn to a CD and mount. Just run the Setup program

image

2- Click “Next” it will show you the current version of veeam components and Click “Next”

image

 

clip_image004

3- Download the license, you have to upgrade your existing 6.5 version to 7, we can apply license at this stage or after completing your upgrade.

clip_image005

4- It will automatically check the prerequisites, any dependencies will be installed automatically.

clip_image006

5- Give the local admin or domain credentials, which has full admin access to the veeam sever and its database.

clip_image007

6- Select the MSSQL database instance correctly, after that it will give a warning that “any old database will be automatically updated” This is safe, just click “Yes”

clip_image008

clip_image009

clip_image011

clip_image012

7- Now Click “Install” and Enjoy !! Relax…..

clip_image013

clip_image015

8- That’s it…. Click “Finish” you are done, no need to reboot that’s cool !! But at this stage the upgrade is not completed.

clip_image016

9- Now run the veeam program, its there in the desktop !!

clip_image017

10- Now we have to upgrade the components like, the backup transport module, instant recovery module etc… just click “OK” now its done.

clip_image019

clip_image021

11- Apply the license

clip_image022

clip_image023

Now you can resume all of your backup operations, just note the below

image

image

This is why I really love Veeam, what a cool and easy Stuff. In my next blog, I will discuss the main features and its uses.

VMware ESXi network driver install and upgrade

Recently I have to perform a network driver update in the ESXi host, it is very nice to do this. Here I am updating the Ethernet card driver  for ESXi 5. This process can be used for updating any driver in the ESXi & ESX host.

VMware uses a file package called a VIB (VMware Installation Bundle) as the mechanism for installing or upgrading software packages, drivers on an ESX server.

The file may be installed directly on an ESX server from the command line or we can deploy using VMware update manager. Here I am going to mention about the command line method of doing the upgrade.

Install or update a patch/driver on the host using these esxcli commands:

COMMAND LINE INSTALLATION

New Installation

For new installs, you should perform the following steps, Issue the following command (full path to the file must be specified):

esxcli software vib install -v {VIBFILE} or esxcli software vib install -d {OFFLINE_BUNDLE}

Upgrade Installation

The upgrade process is similar to a new install, except the command that should be issued is the following:

esxcli software vib update -v {VIBFILE} or esxcli software vib update -d {OFFLINE_BUNDLE}

Notes:

  • To install or update a .zip file, use the -d option. To install or update a .vib file use the -v option.
  • The install method has the possibility of overwriting existing drivers. If you are using 3rd party ESXi images, VMware recommends using the update method to prevent an unbootable state.
  • Depending on the certificate used to sign the VIB, you may need to change the host acceptance level. To do this, use the following command: esxcli software acceptance set –level=<level>

Usage: esxcli software acceptance set [cmd options]

Description:

Set (Sets the host acceptance level. This controls what VIBs will be allowed on a host.)

Cmd options:

–level=< VMwareCertified / VMwareAccepted / PartnerSupported / CommunitySupported >

(Specifies the acceptance level to set. Should be one of VMwareCertified / VMwareAccepted / PartnerSupported / CommunitySupported.)

STEPS

1, Download the corresponding driver from the VMware site, like given below.

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI50-Qlogic-nx_nic-50619&productId=229

clip_image002

2, Extract the nx_nic-esx50-5.0.619-743299.zip file and there you can see a file net-nx-nic-5.0.619-3vmw.500.0.0.472560.x86_64.vib which contains the Driver. And there is a ZIP file nx_nic-esx50-5.0.619-offline_bundle-743299.zip which is an offline bundle, this also contains the driver and we can also use this for the driver installation.

image

3, Migrate or power off the virtual machines running on the host and put the host into maintenance mode. The host can be put into maintenance mode from the command line with:

ESXi: # vim-cmd hostsvc/maintenance_mode_enter

ESX: # vimsh -n -e hostsvc/maintenance_mode_enter

Exit maintenance mode using the vimsh or vim-cmd command

ESXi: # vim-cmd /hostsvc/maintenance_mode_exit

ESX: # vimsh -n -e /hostsvc/maintenance_mode_exit

4, Upload the VIB file or offline bundle to the ESXi location /tmp via SCP

image

5, Then open SSH shell for ESxi and go to the directory and check for the uploaded file.

cd /tmp

clip_image001

6, To install the Driver give the command esxcli software vib install -v /tmp/net-nx-nic-5.0.619-3vmw.500.0.0.472560.x86_64.vib

clip_image001[4]

To upgrade the drivers – esxcli software vib update -d /tmp/nx_nic-esx50-5.0.619-offline_bundle-743299.zip

clip_image002

7, Verify that the VIBs are installed on your ESXi host:
# esxcli software vib list

image

8, Then reboot the ESXi host.

NOTES

if you didn’t give full path of the VIB file it will show the below error.

clip_image001[10]

If the VIB is already installed, you can see the below;

clip_image002[8]

To remove a VIB, search the installed VIB by the command esxcli software vib list

image

Once we find the correct VIB name give the command esxcli software vib remove -n net-nx-nic

clip_image005

And reboot the ESXi server.

VMware vCenter Upgrade from 4.1 to 5.0 : Part-1

There are many ways… to upgrade the vcenter from 4.1 to 5.0, customers have different scenario in their environment, I would like to discuss these common scenarios

OBJECTIVE: Upgrade the vCenter 4.1 to vCenter 5.0

SCENARIO : 

A- vCenter 4.1 and Database in the same server, the database is MSSQL 2005 Express edition (bundled MSSQL database, within the vCenter 4.1 DVD)

B- vCenter 4.1 and Database in the same server, the database is MSSQL 2005 Standard or Enterprise.

END RESULT :

A – vCenter 5.0 and Database will be in the same server, the Database will be the same (MSSQL 2005 Express edition ), but the SCHEMA will be upgraded.

B – vCenter 5.0 and Database will be in the same server, the Database will be the same MSSQL 2005 Standard or Enterprise database, but the SCHEMA will be upgraded.

MIGRATION STEPS :

1- Install SQL management studio and Make a full backup of the vCenter Server database.

image

image

Click ADD and give the location to save the backup of the database or use defaults.

2- Check Existing SYSTEM DSN for the vCenter 4.1 in the OBDC Data Source Administrator, and test the connection settings.

image

image

3- Back up the SSL certificates that are on the VirtualCenter or vCenter Server system before you upgrade to vCenter Server 5.0. The default location of the SSL certificates is %allusersprofile%\Application
Data\VMware\VMware VirtualCenter\SSL

4- Stop the VMware VirtualCenter Server service.

image

5- From the vSphere 5 DVD or its contents, run the AUTORUN.EXE installer for the vSphere 5 and select to install “vCenter Server” Also ensure all the prerequisites are installed.

image

6- The installer wizard will indicate that there is already previous version of vCenter present in the server.Click next and accept the EULA and give the license or later also we can give license.

image

7- The existing DSN will be automatically detected and click next,

image

8- Select these options, you need to upgrade the database SCHEMA to accommodate the new vCenter 5.0,

“A dialog box might appear warning you that the DSN points to an older version of a repository that must be upgraded. If you click Yes, the installer upgrades the database schema, making the database irreversibly incompatible with previous VirtualCenter versions. See the vSphere Upgrade documentation”

image

9- We can automatically or manually the vCenter agent in the ESXi hosts, its not a big deal !!!

image

10- Give the credentials and use the FQDN of the old vCenter server, else it will throw an error saying FQDN is not resolved and vCenter feature will not work. This we can neglect, but it is good to give FQDN.

image

11- Use the default location of the binaries or give the locations if need to customize.

image

12- It is better to use default ports for the vCenter, else we can change if needed. Rest of the next few steps are self explanatory and choose accordingly

13- If you are using Distributed switch, and large number of portgroups and virtual machines we can select the below option IF YOU ARE USING EPHEMERAL PORTS, BUT IT IS RECOMMEDED TO USE THE STATIC BINDING IN THE vDS so no need to use this option.

image

14- Click INSTALL button !!!! That’s it…

 

My next post will be..how to upgrade the vCenter and Database if they are in separate servers.

Magic Effect of SIOC (VMware Storage Input Output Control) in vSphere 5 : A practical study.

Recently I got a chance to implement and test the Enterprise Plus feature in vSphere called SIOC. It is really an amazing feature…..

Theory

Application performance can be impacted when servers contend for I/O resources in a shared storage environment. There is a crucial need for isolating the performance of critical applications from other, less critical workloads by appropriately prioritizing access to shared I/O resources. Storage I/O Control (SIOC), a new feature offered in VMware vSphere 4 & 5, provides a dynamic control mechanism for managing I/O resources across VMs in a cluster.

Datacenters based on VMware’s virtualization products often employ a shared storage infrastructure to service clusters of vSphere hosts. NFS, ISCSI, and Storage area networks (SANs) expose logical storage devices (LUNs) that can be shared across a cluster of ESX hosts.  Consolidating VMs’ virtual disks onto a single VMFS datastore, or NFS datastore, backed by a higher number of disks has several advantages—ease of management, better resource utilization, and higher performance (when storage is not bottlenecked).

With vSphere 4.1 we can only use SIOC with FC/ISCSI datastores, but with vSphere 5 we can use NFS.
However, there are instances when a higher than expected number of I/O-intensive VMs that share the same storage device become active at the same time. During this period of peak load, VMs contend with each other for storage resources. In such situations, lack of a control mechanism can lead to performance degradation of the VMs running critical workloads as they compete for storage resources with VMs running less critical workloads.
Storage I/O Control (SIOC), provides a fine-grained storage control mechanism by dynamically allocating portions of hosts’ I/O queues to VMs running on the vSphere hosts based on shares assigned to the VMs. Using SIOC, vSphere administrators can mitigate the performance loss of critical workloads during peak load periods by setting higher I/O priority (by means of disk shares) to those VMs running them. Setting I/O priorities for VMs results in better performance during periods of congestion.

So What is the Advantage for an VM administrator/Organization? Now a days, with vSphere 5 we can have 64TB of single LUN and with high end SAN with FLASH we are achieving more consolidation. That’s great !! but when we do this…. just like the CPU/MEMORY resource pools ensure the Computing SLA, the SIOC ensures the virtual disk STORAGE SLA and its response time.

In short, below are the benefits;

– SIOC prioritizes VMs’ access to shared I/O resources based on disk shares assigned to them. During the periods of I/O congestion, VMs are allowed to use only a fraction of the shared I/O resources in proportion to their relative priority, which is determined by the disk shares.
– If the VMs do not fully utilize their portion of the allocated I/O resources on a shared datastore, SIOC redistributes the unutilized resources to those VMs that need them in proportion to VMs’ disk shares. This results in a fair allocation of storage resources without any loss in their utilization.
– SIOC minimizes the fluctuations in performance of a critical workload during periods of I/O congestion, as much as a 26% performance benefit compared to that in an unmanaged scenario.

How Storage I/O Control Works
SIOC monitors the latency of I/Os to datastores at each ESX host sharing that device. When the average normalized datastore latency exceeds a set threshold (30ms by default), the datastore is considered to be congested, and SIOC kicks in to distribute the available storage resources to virtual machines in proportion to their shares. This is to ensure that low-priority workloads do not monopolize or reduce I/O bandwidth for high-priority workloads. SIOC accomplishes this by throttling back the storage access of the low-priority virtual machines by reducing the number of I/O queue slots available to them. Depending on the mix of virtual machines running on each ESX server and the relative I/O shares they have, SIOC may need to reduce the number of device queue slots that are available on a given ESX server.

It is important to understand the way queuing works in the VMware virtualized storage stack to have a clear understanding of how SIOC functions. SIOC leverages the existing host device queue to control I/O prioritization. Prior to vSphere 4.1, the ESX server device queues were static and virtual-machine storage access was controlled within the context of the storage traffic on a single ESX server host. With vSphere 4.1 & 5, SIOC provides datastore-wide disk scheduling that responds to congestion at the array, not just on the hostside HBA.. This provides an ability to monitor and dynamically modify the size of the device queues of each ESX server based on storage traffic and the priorities of all the virtual machines accessing the shared datastore. An example of a local host-level disk scheduler is as follows:

Figure 1 shows the local scheduler influencing ESX host-level prioritization as two virtual machines are running on the same ESX server with a single virtual disk on each.

image

Figure 1. I/O Shares for Two Virtual Machines on a Single ESX Server (Host-Level Disk Scheduler)

In the case in which I/O shares for the virtual disks (VMDKs) of each of those virtual machines are set to different values, it is the local scheduler that prioritizes the I/O traffic only in case the local HBA becomes congested. This described host-level capability has existed for several years in ESX Server prior to vSphere 4.1 & 5. It is this local-host level disk scheduler that also enforces the limits set for a given virtual-machine disk. If a limit is set for a given VMDK, the I/O will be controlled by the local disk scheduler so as to not exceed the defined amount of I/O per second.

vSphere 4.1 onwards it has added two key capabilities: (1) the enforcement of I/O prioritization across all ESX servers that share a common datastore, and (2) detection of array-side bottlenecks. These are accomplished by way of a datastore-wide distributed disk scheduler that uses I/O shares per virtual machine to determine whether device queues need to be throttled back on a given ESX server to allow a higher-priority workload to get better performance.

The datastore-wide disk scheduler totals up the disk shares for all the VMDKs that a virtual machine has on the given datastore. The scheduler then calculates what percentage of the shares the virtual machine has compared to the total number of shares of all the virtual machines running on the datastore. As described before, SIOC engages only after a certain device-level latency is detected on the datastore. Once engaged, it begins to assign fewer I/O queue slots to virtual machines with lower shares and more I/O queue slots to virtual machines with higher shares. It throttles back the I/O for the lower-priority virtual machines, those with fewer shares, in exchange for the higher-priority virtual machines getting more access to issue I/O traffic.

However, it is important to understand that the maximum number of I/O queue slots that can be used by the virtual machines on a given host cannot exceed the maximum device-queue depth for the device queue
of that ESX host.

What are the conditions required for the SIOC to work ?
– A large datastore, and lot of VMs in it and this datastore is shared between multiple ESX hosts.
– Need to set disk shares, for the vm’s inside the datastore.
– Based on the LUN/NFS type (made of SSD,SAS,SATA), select the SIOC threshold, and enable SIOC.
– The SIOC will monitor the datastore IOPS usage, latency of VM’s and monitor overall STORAGE Array performance/Latency.

SIOC don’t check the below;

How much Latency created by other physical systems to the array, that is the Array is shared for the PHYSICAL hosts also, and other backup applications/jobs etc. So it gives false ALARMS like “VMware vCenter – Alarm Non-VI workload detected on the datastore] An unmanaged I/O workload is detected on a SIOC-enabled datastore”

Tricky Question – Will it work for one ESXi host ? Simple answer – SIOC is for multiple ESXi hosts, Datastore is exposed to many hosts and in that Datastore contains a lot of VM’s. For single host also we can enable the SIOC if we have the license, but it is not intended for this CASE.

So what is the solution for a single ESXi host ? – Simple just use shares for the VMDK and the ESXi host use Host-Level Disk Scheduler and ensures the DISK SLA.

So is there any real advantage of the SIOC in Real time scenario ? the below experience of me PROVES this..

My scenario: I have 4 ESXi 5 hosts in the cluster, and many FC LUNS from the HP 3PAR storage arrays is mounted. In one or the LUN we have put around 16 VMs and this includes our vCenter and it Database, and inside that Datastore we have a less critical VM (Analytics engine, which is from the vCOPS appliance) and this consumes a lot of IOPS from the storage and eventually effects other critical VM. We enabled the SIOC and monitored for 10 days then we compared the performance of the VMDK and its LATENCY during the PEAK hours.

Below is the Datastore with many VMS and it is shared across 4 HOSTS.

image

With SIOC disabled.

image

image

With SIOC enabled.

image

image

So what is the Take away with SIOC ?

Better I/O response time, reduced latency for critical VMS and ensured disk SLA for critical VM’s. So bottom line is the crtical VM’s wont get affected during the event of storage contention and in highly consolidated environment.

 

REFERENCES:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020651

http://blogs.vmware.com/vsphere/2011/12/using-both-storage-io-control-network-io-control-for-nfs.html

http://blogs.vmware.com/vsphere/2012/03/debunking-storage-io-control-myths.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022091

http://blogs.vmware.com/vsphere/2011/09/storage-io-control-enhancements.html

http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf

http://blogs.vmware.com/performance/2010/07/sioc.html

Partition alignment in VMware vSphere 5, a DeepDrive, Part-2

In continuation to my part-1, we will see how the Virtual machine disk alignment effects the Virtualized world and eventually the performance.

Here we need to know how to align the disk partitions in OS like windows 2003, XP, RHEL 5.x etc in VMware vSphere VMFS 5 Datastore.

Let us assume the VMFS is residing above a RAID volume or a LUN from a storage array, in either case the RAID STRIP size will be from 4KB to 256KB depending upon the ARRAY and RAID levels. So now we are deployed the above mentioned operating systems on the VMFS,

What is the ISSUE with the GUEST OS disk misalignment ?

This issue is applicable to the PHSICAL and VIRTUAL also and it is not because of the VMFS layer or any other means. It is the limitation of these OS and how they do the partition in the given HDD, just like to the PHYSICAL world, in VIRTUAL world the Hypervisor also gives an HDD but it is a VIRTUAL HDD. But the OS doesn’t know if it is a virtual HDD or Actual HDD. So eventually the OS wont do the partition with correct alignment.

Leave the VITRUL world and go to PHYSICAL world,  let us assume the Guest filesystem Block/Cluster is 4KB, and it writes or reads a 4KB to the hard disk, and in turn it goes to the HDD PHYSICAL sector of 4KB. Because the partition is not ALIGNED it uses 2 PHYSICAL sectors of 4KB for this operation, here the amount of Data read/write is same but it uses 2 PHYSICAL sectors.

image

So the Harddisk HEAD need to do more work to fetch the Data to and from the PHYSICAL sectors. So for a single IO for the same 4KB data from the OS, the HDD or Array need to use 2 PHYSICAL sectors, so the IOPS response time and latency will be affected. So we will get a POOR performance. This is applicable in VIRTUAL world also,

As of now the VMFS3 and VMFS5 are already aligned to the underlying storage. The below give more info how the VMFS is aligned, open an ESXi shell or SSH and type the below;

~ # partedUtil get /vmfs/devices/disks/naa.600508b1001030394330313737300300
71380 255 63 1146734896

1 2048 1146719699 0 0
~ #

The first line displayed is disk geometry information (cylinders, heads, sectors per track and LBA [Logical Block Address] count). The second line is information about the partitions. There is only 1 partition; it starts at LBA 2048 and ends at LBA 1146719699.That’s something else to be aware of – newly created VMFS-5 partitions start at LBA 2048. This is different to previous versions of VMFS:

  • VMFS-2 created on ESX 2.x; starting LBA 63
  • VMFS-3 created on ESX 3 & 4; starting LBA 128
  • VMFS-5 created on ESXi 5; starting LBA 2048

So the VMFS is aligned with the 1MB boundary (starts from the LBA 2048), and as we all know the VMFS 5 the block size is 1MB. So the VMFS and the underlying storage is already aligned. So when a misaligned GUEST OS send’s a READ/WRITE request to the HYPERVISOR layer, that is from the VMDK to the VM SCSI controller to the VMFS and finally to the STORAGE, the STORAGE has to look for more than one PHYSICAL sector or CHUNK, this is an over head for the storage and of course it will be an over head to the VMkernel, because the VMkernel has to wait until the array does the task.

image

If the GUEST OS is aligned, for one 4KB write/read it will use only one single CHUNK from the storage, this will give GOOD Response time and LOW latency for the IOPS operation.

image

Now How to align the Guest OS;

For windows follow the below –

1- Add the required virtual HDD to the Guest OS

2- Verify the HDD is visible in the OS

image

image

3- Open Command prompt and use the Command Line Syntax below

C:\>diskpart

DISKPART> list disk

DISKPART> select disk 2

Disk 2 is now the selected disk.

DISKPART> create partition primary align=1024

image

image

 

After this format the PARTITON, Windows MSSQL databases and Exchange servers, it is recommended to format the NTFS cluster size as 64K (64 kilobytes) and for other normal server we can use 32K.

image

Generally to check a partition is aligned or not use the below command, then refer the PART 1 of this blog to do the MATH.

wmic partition get BlockSize, StartingOffset, Name, Index

image

In my case this shows both my disks having partitions that are aligned to 1024KB or 1MB …or sector2048.

(1048576 bytes)/(512 bytes/sector) = 2048 sector

To check File Allocation Unit Size – Run this command for each drive to get the file allocation unit size:

fsutil fsinfo ntfsinfo c:

image

Steps to align the partition for Linux.

To check that your existing partitions are aligned, issue the command:

fdisk –lu

The output is similar to:

image

1. Enter fdisk /dev/sd<x> where <x> is the device suffix.

2. Type n to create a new partition.

3. Type p to create a primary partition.

4. Type 1 to create partition No. 1.

5. Select the defaults to use the complete disk.

image

1. Type t to set the partition’s system ID.

2. Type 8e to set the partition system ID to 8e (Linux LVM)

3. Type x to go into expert mode.

4. Type b to adjust the starting block number.

5. Type 1 to choose partition 1.

6. Type 2048 to set it to start with the sector 2048.

7. Type w to write label and partition information to disk.

image

To check that your existing partitions are aligned, issue the command:

fdisk –lu

image

Now you can see the partition are aligned, that is started from the sector 2048 that is 1MB boundary.

NOTE:

Now in the internet there are many methods to automate the process during the TEMPLATE deployment, one method is add few 1 GB VMDK and do the partition alignment and make a template and after that when the template deployment is over, increase the VMDK for the guest OS. This will work fine for the thin disk and lazy zeroed disks, but if you do the same process for a EAGAR ZEROED disk, then we all know the outcome. Once you increase the EAGAR ZEROED disk it will become LAZY zeroed disk, so for FT and windows clusters, oracle clusters it will be a problem. Else after the VMDK increase you need to use the VMKFSTOOLS to change the VMDK from LAZY zeroed disk to EAGAR ZEROED disk. So again a management overhead, so its your decision !!

 

REFERENCE:

 

http://blogs.vmware.com/vsphere/2011/08/vsphere-50-storage-features-part-7-gpt.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036609

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003565

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003813

Partition alignment in VMware vSphere 5, a DeepDrive, Part-1

This topic has been discussed seriously for a long time… in the virtualization domain. I would like to add few insights in to this topic, with the new vSphere 5.x release there are lot of changes happened in the VMFS filesystem and also with the release of Windows 2008, 2008 R2, 2012 and RHEL 6.x, Ubuntu 12.x there is no need of doing partition alignment, but it would be good to know how these OS do the partition and handle the disk volumes.

Also the legacy OS like Windows 2003, RHEL 5.x still need partition alignment with vSpehere 5 & VMFS 5 filesystem. In short, generally this topic applies to physical, other hypervisor vendors and to VMWARE also.

There are many outdated articles in the web and below info will give a good insight to this topic.

Theory & History

As we all know a physical server or Storage array need physical HDD’s, with virtual machine it is a virtual HDD. The below image shows the hard disk geometry

clip_image001

In the case of early IDE/ATA hard disks the BIOS provides access to the hard disk through an addressing mode called Cylinder-head-sector, also known as CHS, was an early method for giving addresses to each physical block of data on a hard disk drive. CHS addressing is the process of identifying individual sectors on a disk by their position in a track, where the track is determined by the head and cylinder numbers.

In old computer system the maximum amount of addressable data was very limited – due to limitations in both the BIOS and the hard disk interface. The legacy OS like NT, DOS etc uses this method.

Modern hard disks use a recent version of the ATA standard, such as ATA-7. These disks are accessed using a different addressing mode called: logical block addressing or LBA involves a totally new way of addressing sectors. Instead of referring to a cylinder, head and sector number, each sector is instead assigned a unique “sector number”. In essence, the sectors are numbered 0, 1, 2, etc. up to (N-1), where N is the number of sectors on the disk. So all modern disk drives are now accessed using Logical Block Addressing (LBA) scheme where the sectors are simply addressed linearly from 0 to some maximum value and disk partition boundaries are defined by the start and end LBA address numbers. In LBA addressing system with each cylinder standardized to 255 heads, each head has one track with 63 logical blocks or sectors and each one has 512 bytes. You can see this info in linux

clip_image002

All the modern OS uses the LBA method to read/write data to the Harddisk.

In olden days the physical sector of the HDD is 512-byte, and this has been the standard for over 30 years. This physical sector size will match with the size of one Logical block or sector of the OS that is 512 byte, so no issue. During 2009, IDEMA (The International Disk Drive Equipment and Materials Association) and leading data storage companies introduced Advanced Format (AF) Technology in the HDD, so that the physical sector size will be 4K (4,096 bytes). Disk drives with larger physical sectors allow enhanced data protection and correction algorithms, which provide increased data reliability. Larger physical sectors also enable greater format efficiencies, thereby freeing up space for additional user data.

One of the problems of introducing this change in the media format is the potential for introducing compatibility issues with existing software and hardware. As a temporary compatibility solution, the storage industry is initially introducing disks that emulate a regular 512-byte sector disk, but make available info about the true sector size through standard ATA and SCSI commands. As a result of this emulation, there are, in essence, two sector sizes:

Logical sector: The unit that is used for logical block addressing for the media. We can also think of it as the smallest unit of write that the storage can accept. This is the “emulation.”

Physical sector: The unit for which read and write operations to the device are completed in a single operation. That is the Actual physical sector size of storage data on a disk.

Initial types of large sector media

The storage industry is quickly ramping up efforts to transition to this new Advanced Format type of storage for media having a 4 KB physical sector size. Two types of media will be released to the market:

4 KB native: Disks that directly report a 4 KB logical sector size and have a physical sector size of 4 KB – The disk can accept only 4 KB IOs to the disks. However, the software stack can provide 512-byte logical sector size support through RMW support. This media has no emulation layer and directly exposes 4 KB as its logical and physical sector size. The overall issue with this new type of media is that the majority of apps and operating systems do not query for and align I/Os to the physical sector size, which can result in unexpected failed I/Os.

512-byte emulation (512e): Disks that directly report a 512-byte logical sector size but have a physical sector size of 4 KB – Firmware translate 512 byte writes to 4k writes RMW (Read Modify Write). In today’s drives, this translation introduces a performance penalty. This media has an emulation layer as discussed in the previous section and exposes 512-bytes as its logical sector size (similar to a regular disk today), but makes its physical sector size info (4 KB) available.

Overall Windows support for large sector (4KB) media

This table documents the official Microsoft support policy for various media and their resulting reported sector sizes. See this KB article for details.

clip_image004

Windows 8, windows server 2012 and Starting with Linux Kernel Version 2.6.34 has full support to read and write 4K for the LBA by the OS. Operating systems like Windows 7, 2008, 2008 R2 still uses 512 Bytes for each logical block or sector, so one physical sector of 4K contains 8 logical sectors of size 512 Bytes.

clip_image006

512-byte emulation at the drive interface

clip_image008

clip_image010

To maintain compatibility, Western Digital, Hitachi, Toshiba emulates a 512-byte device by maintaining a 512-byte sector at the drive interface that is the firmware inside the HDD controller will do the conversion; and Seagate uses SmartAlign technology, for this emulation (firmware level). These drives are also called Advanced Format 512e. Let see how this works !!!

512-byte Read

When the host requests to read a single 512-byte logical block, the hard drive will actually read the entire 4K physical sector containing the requested 512 bytes. The 512-byte block is extracted and sent to the host. This can be done very quickly.

clip_image012

512-byte Write (Read-Modify-Write)

When the host attempts to write a single 512-byte logical block, the hard drive will first read the 4K physical sector containing the 512 bytes that are to be overwritten. Next, it will insert the 512 bytes of new data and write the entire 4K block of data back to the media. This process is called a “Read-Modify-Write”. The drive must read the existing data, modify a subset, and then write the data back to the disk. This process can require additional revolutions of the hard disk.

clip_image014

How Does Advanced Format Technology Maintain Performance?

In order to maintain top performance, it is important to ensure that writes to the disk are aligned. Ideally, writes should be done in 4K blocks, and each block will then be written to a physical 4K sector on the drive. This can be accomplished by ensuring that the OS and applications write data in 4K blocks, and that the drive is partitioned correctly. Most modern operating systems use a file system that allocates storage in 4K blocks or clusters (NTFS Cluster/File allocation unit or EXT3/4 file system block size). In a traditional hard drive, the 4K block is made up of eight 512-byte sectors (see Figure 4: 512-byte Emulated Device Sector Size).

clip_image016

In production, the RAID layer will come in to picture, these 4KB physical sectors are again combined to form a RAID STRIP size, this may vary from 4KB to 256 KB depending upon the RAID level. In the storage array also this is the same. So the RAID array controller like PERC, Intel, LSI, Smart Array etc will handle and gives a RAID volume to the OS. In both cases the OS partition needs to be aligned.

Since most modern operating systems will write in 4K blocks, it is important that each 4K logical block is aligned to a physical 4K block on the disk (see Figure 5). This is especially important because the 512e feature of the drive cannot prevent a partitioning utility from creating a misaligned partition. When misalignment occurs, a logical 4K block will reside on two physical sectors.

clip_image018

In this case, a single read or write of a 4K block will result in a read/write of two physical sectors. The impact of a “read” is minimal, whereas a single write will cause two “Read-Modify-Writes” to occur, potentially impacting performance (see Figure 6).

clip_image020

So now what will happen with the OS like XP, windows 2003, RHEL5.x etc and why we need disk Alignment in PHYSICAL world or in VIRTUAL world ?

As I mentioned the physical disks has 4KB physical sectors and RAID volume or a LUN from storage array  has 4KB to 256 KB STRIP SIZE, that is a multiple of 4KB, and the operating system has 512 bytes of logical sector. When we install the operating system, like 2003, RHEL 5.x. in a HDD or a LUN, in these OS the first 62 sectors (first track) of the HDD is reserved for BOOT area and it is hidden to the OS.

That is sectors from 0 to 62 and reserved (hidden), the master boot record (MBR) resides within these hidden sectors. The master boot record (MBR) resides within these hidden sectors. It uses the first sector of the first track for MBR data (LBA 0) and the first partition begins in the last sector of the first track, which is from (logical) block address 63. You can see this in the below;

clip_image022

Here in RHEL 5.x/LINUX older version, we can see the first partition starts from sector/LBA 63, and if you add another HDD or a LUN this host, and when you create a partition, the partition tool of these linux versions again create partition starting from sector 63.

clip_image024

Here in the above info from Windows 2003, the first partition starts from the offset 32256, in windows it won’t show the LBA/sector number, instead it shows the values in Bytes. That is an offset of 32256 means (32256/512) = 63 LBA/sector, so the partition starts from sector 63. Below is the detailed way of confirming this;

Essential Correlations: Partition Offset, File Allocation Unit Size, and Stripe Unit Size

Use the information in this section to confirm the integrity of disk partition alignment configuration for existing partitions and new implementations.

There are two correlations which when satisfied are a fundamental precondition for optimal disk I/O performance. The results of the following calculations must result in an integer value:

Partition_Offset ÷ Stripe_Unit_Size (Disk physical sector size or RAID strip size)

Stripe_Unit_Size ÷ File_Allocation_Unit_Size

Of the two, the first is by far the most important for optimal performance. The following demonstrates a common misalignment scenario: Given a starting partition offset for 32,256 bytes (31.5 KB) and stripe unit size of 4086 bytes (4 KB), the result is 31.5/4 = 7.894273127753304. This is not an integer; therefore the offset & strip unit size are not correlated.

In the second one, the NTFS cluster size or file allocation unit the default value is 4086, and we can give 32KB, 64KB etc. If it is MSSQL and EXCHANGE It is recommended to give 64KB during the formatting time, and this value is not an issue and it will be an integer. But the first one is the crucial !!!

So we have MISALIGNMENT, and we need to realign the partition, the below diagrams show pictorially;

clip_image026

So Windows 7, 8, 2008, 2008 R2, 2012, RHEL 6, Debian 6, Ubuntu 10, 11, 12, SUSE 11 onwards, automatically aligns partitions during installation. You can see this in the below;

WINDOWS 7

clip_image028

WINDOWS 2008R2

clip_image030

In Windows case the partition alignment defaults to 1024 KB or 1MB boundary (that is, startingoffset 1,048,576 bytes = 1024KB). It correlates well (as described in the previous section, 1024KB/4KB = 256 an integer) with common stripe unit sizes such as 4KB, 64 KB, 128 KB, and 256 KB as well as the less frequently used values of 512 KB and 1024 KB. That is simply the windows partition tool begin the first partition at LBA/sector 2048 (1,048,576/512 = 2048). So here not need to manual alignment and if we add another disk also it will do auto alignment.

clip_image032

RHEL 6

clip_image034

In RHEL and latest linux cases, the first partition starts from 2048 that is LBA 0 to 2047 is reserved. That is the OS is aligned to 1MB boundary, if we do the math the sector 2048 is at the offset 1,048,576 bytes (1,048,576 bytes/512 = 2048) and if we add another HDD or LUN it will do alignment automatically.

My next post will be discussing how to do the disk alignment in vSphere or any other hypervisor.

References:

http://en.wikipedia.org/wiki/Advanced_Format

http://www.tech-juice.org/2011/08/08/an-introduction-to-hard-disk-geometry/

http://www.seagate.com/tech-insights/advanced-format-4k-sector-hard-drives-master-ti/

http://en.wikipedia.org/wiki/Cylinder-head-sector

http://en.wikipedia.org/wiki/Logical_Block_Addressing

http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/

http://wiki.hetzner.de/index.php/Partition_Alignment/en

http://blogs.technet.com/b/askcore/archive/2011/09/26/alignment-changes-in-windows-2008-and-2008-r2.aspx

http://msdn.microsoft.com/en-us/library/dd758814%28v=sql.100%29.aspx

http://frankdenneman.nl/2009/05/20/windows-2008-disk-alignment/

http://support.microsoft.com/kb/2510009

http://support.microsoft.com/kb/2515143

http://technet.microsoft.com/en-us/library/ee832792.aspx#Phys

http://blogs.msdn.com/b/psssql/archive/2011/01/13/sql-server-new-drives-use-4k-sector-size.aspx

http://www.idema.org/?page_id=1936

http://wdc.custhelp.com/app/answers/detail/a_id/5655/~/how-to-install-a-wd-advanced-format-drive-on-a-non-windows-operating-system

http://msdn.microsoft.com/en-us/library/windows/desktop/hh848035%28v=vs.85%29.aspx

http://blogs.technet.com/b/filecab/archive/2011/04/26/using-4k-sector-and-advanced-format-drives-in-windows-hotfix-and-support-info-for-windows-server-2008-r2-and-windows-7.aspx

http://storage.toshiba.com/docs/services-support-documents/toshiba_4kwhitepaper.pdf

http://www.hgst.com/tech/techlib.nsf/techdocs/3D2E8D174ACEA749882577AE006F3F05/$file/AFtechbrief.pdf

http://www.seagate.com/files/docs/pdf/datasheet/disc/ds_momentus_5400_6.pdf

http://www.wdc.com/wdproducts/library/WhitePapers/ENG/2579-771430.pdf

http://www.seagate.com/docs/pdf/whitepaper/mb_smartalign_technology_faq.pdf

http://www-03.ibm.com/systems/resources/systems_i_advantages_integratedserver_pdf_vmware_storage_alignment.pdf

 

Multi-NIC vMotion in vSphere 5.x

As always VMware brings another feature in this release…..there is a huge improvement in the VMotion performance compared to vSphere 4.x

with this feature, now the VMotion traffic can span across multiple pNICS attached to the vmotion vswitch.

To configure it… follow the below….

 

image

To set up Multi-NIC vMotion in vSphere 5.x on a Standard vSwitch:

  1. Log into the vSphere Client and select the host from the inventory panel.
  2. Click the Configuration tab and select Networking.
  3. Click Add Networking and choose VMkernel as the Connection Type.
  4. Click Next.
  5. Add two or more NICs to the required standard switch.
    Note: You can create a new vSphere standard switch or use an existing vSwitch.
  6. Name the VMkernel portgroup (for example, vMotion-01), and assign a VLAN ID as required.
  7. Click Use this port group for vMotion, then click Next.
  8. Configure the IP address and subnet mask, then click Next..
  9. Click the Properties tab of the vSwitch, select the vMotion-01 portgroup, and click Edit.
  10. Click the NIC Teaming tab.
  11. Under Failover Order, select Override switch failover order.
  12. Configure the first adapter (for example, vmnic1) as active and move the second adapter (for example, vmnic3) to standby.
  13. Click OK.
  14. Under the vSwitch Properties, click Add to create a second VMkernel portgroup.
  15. Name the VMkernel portgroup (for example, vMotion-02), and assign a VLAN ID as required.
    Note: Ensure that both VMkernel interfaces participating in the vMotion have the IP address from the same IP subnet.
  16. Click Use this port group for vMotion, then click Next.
  17. Configure the IP address and subnet mask, then click Next.
  18. Click the Properties tab of the vSwitch, select the vMotion-02 portgroup, and click Edit.
  19. Click the NIC Teaming tab.
  20. Under Failover Order, select Override switch failover order.
  21. Configure the second adapter (for example, vmnic3) as active and move the first adapter (for example, vmnic1) to standby.
  22. On the Properties tab of the vSwitch, select each vMotion portgroup in turn and confirm that the active and standby adapters are the reverse of each other.

once you configured you can see like this.

image

 

 

To set up Multi-NIC vMotion in vSphere 5.x on a Distributed vSwitch:

  1. Log into the vSphere Client and click the Networking inventory.
  2. Click New vSphere Distributed Switch and choose version 5.0.0.
  3. Name the Distributed switch (for example, Multi-NIC-vMotion).
  4. Assign two uplink ports to the switch, then click Next.
  5. Select physical adapters to each of the hosts, then click Next and Finish.
  6. Expand the Distributed switch you just created, click the dvPortGroup and click Edit Settings.
  7. Name the dvPortgroup (for example, vMotion-01).
  8. Click VLAN and assign a VLAN ID as required.
  9. Click the Teaming and Failover tab, configure dvUplink1 as Active Uplink and move dvUplink2 to Standby Uplink.
  10. Right-click the Distributed vswitch, then click New Port Group.
  11. Name the dvPortgroup (for example, vMotion-02).
  12. Click VLAN and assign a VLAN ID as required, then click Next and Finish.
  13. Select the second portgroup created, then click the Teaming and Failover tab.
  14. Configure dvUplink2 as Active Uplink and move dvUplink1 to Standby Uplink.
  15. Go the Hosts and Clusters Inventory tab, select a host’s Networking, and click vSphere Distributed Switch.
  16. Click Manage Virtual Adapters and click Add to add new virtual adapter.
  17. Choose VMkernel as the Virtual Adapter Type.
  18. Select the vMotion-01 portgroup, click Use this port group for vMotion, then click Next.
  19. Configure the IP address and subnet mask, then click Next and Finish.
  20. Add another virtual adapter, then select the vMotion-02 portgroup.
  21. On the Distributed vSwitch, select each dvportgroup on VMKernel Port vmk1 and vmk2 in turn, and confirm that the active and standby uplinks are the reverse of each other.

once you configured you can see like this.

 

image 

 

    imageimage 

 

image   image

 

image

 

image image

 

My Test Results , I have used Network I/O Control to check with different bandwidth for pNICS

image

 

Note :-

No need to change the teaming policy for the VMotion vSS switch or vDS switch

No need to create Etherchannel,LACP,LAG etc for the VMotion network, ESXi will spread the Network traffic across the pNICS even for one VM. Also there are multiple uplinks so we get redundancy also, in the VMotion port groups we making other pNICS in the team as standby.

Need to set the “Fail back” option to “NO” in the vswitch teaming policy, because it is recommended, during  pSWITCH reboot the switch ports links will be up first but the switch wont be ready and the ESX traffic will immediately failback and tries to send traffic.

For vSS No need to change the other settings in the teaming policy for the port group, only we need to change the pNICS fail over order.

For vDS we need to change the settings in the port group “Fail back” option to “NO” and the failover order of the pNICS

For vDS we need to use STATIC BINDING

For 10GbE Ethernet cards, it is good to use Network I/O Control and set bandwidth & set shares for the VMotion network

 

REF:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007467

Dan Gorman's Technology News Aggregation

My Daily Readings from Zite

Virtual Reality

Eat > Sleep > Drink and Dream Virtualization

Brad Hedlund

stuff and nonsense

VCDX56

A blog focusing on day 2 day virtualization stuff

UCSguru.com

Every Cloud Has a Tin Lining.

pibytes

Experience the Datacenter Technologies

boche.net - VMware vEvangelist

Experience the Datacenter Technologies

blog.scottlowe.org

The weblog of an IT pro specializing in virtualization, networking, cloud, servers, & Macs

Eric Sloof - NTPRO.NL

Experience the Datacenter Technologies

Technodrone

Experience the Datacenter Technologies

Welcome to vSphere-land!

your ultimate VMware information destination

Michelle Laverick...

Laverick by Name, Maverick by Nature...

CloudXC

By Josh Odgers - VCDX#90

Long White Virtual Clouds

all things vmware, cloud and virtualizing business critical applications

Virtual Geek

Experience the Datacenter Technologies

Yellow Bricks

by Duncan Epping