ESXi是经裁剪和扩展后的linux,同样可以做bonding或者vlan trunk,在esxi的术语中,bongding被称为nic teaming,vlan trunk则简称为trunk。然而,esxi和linux又有些不同,下面举例说明这两种功能的配置和行为。

本实验用到一台DL385g8(安装esxi5.5u2)和两台h3c 5800交换机(利用h3c irf协议做了堆叠),它们之间网口/端口连接对应表如下:

Table 1. DL385、h3c5800网口/端口连接对应表

ESXi

h3c 5800

port1

gi1/0/26

port2

gi2/0/26

port3

gi1/0/34

port4

gi2/0/34

nic teaming

ESXi的标准交换机不支持lacp,只能配置静态的上行链路聚合。VMware的分布式交换机支持lacp,但需要部署vCenter,本文仅涉及单台ESXi,所以只能用标准交换机。

h3c 5800交换机的端口配置如下:

interface Bridge-Aggregation26
 description "DL385g8(esxi6-bond0)"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 111 300
interface GigabitEthernet1/0/26
 port link-mode bridge
 description "DL385g8:port1(esxi6)"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 111 300
 port link-aggregation group 26
interface GigabitEthernet2/0/26
 port link-mode bridge
 description "DL385g8:port2(esxi6)"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 111 300
 port link-aggregation group 26
...
interface Bridge-Aggregation34
 description "DL385g8(esxi6-bond1)"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 111 300
interface GigabitEthernet1/0/34
 port link-mode bridge
 description "DL385g8:port3(esxi6)"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 111 300
 port link-aggregation group 34
interface GigabitEthernet2/0/34
 port link-mode bridge
 description "DL385g8:port4(esxi6)"
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 111 300
 port link-aggregation group 34

esxi中的网口配置如下:

ESXi# esxcli network nic list
Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description
------  -------------  ------  ----  -----  ------  -----------------  ----  -------------------------------------------------------
vmnic0  0000:004:00.0  tg3     Up     1000  Full    a0:d3:c1:f1:03:a8  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic1  0000:004:00.1  tg3     Up     1000  Full    a0:d3:c1:f1:03:a9  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic2  0000:004:00.2  tg3     Up     1000  Full    a0:d3:c1:f1:03:aa  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic3  0000:004:00.3  tg3     Up     1000  Full    a0:d3:c1:f1:03:ab  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet

但是h3c 5800上显示2/0/26处于down的状态

[SY-S5800-HSW]dis int bri | incl /26
The brief information of interface(s) under bridge mode:
Link: ADM - administratively down; Stby - standby
Speed or Duplex: (a)/A - auto; H - half; F - full
Type: A - access; T - trunk; H - hybrid
Interface            Link Speed   Duplex Type PVID Description
GE1/0/26             UP   1G(a)   F(a)   T    1    "DL385g8:port1(esxi6)"
GE2/0/26             DOWN auto    A      T    1    "DL385g8:port2(esxi6)"

做了以下尝试:

  1. 在esxi中重启网卡

    ESXi# esxcli network nic down -n vmnic1
    ESXi# esxcli network nic up -n vmnic1
  2. 在交换机中重做link-aggregation

  3. 关闭broadcom 5719驱动的netqueue[1]特性,并重启。

    ESXi# esxcli system settings kernel set -s netNetqueueEnabled -v FALSE
    ESXi# reboot
broadcom netqueue关闭方法

可通过命令esxcfg-module -s force_netq=0,0,0,0 tg3关闭每一张1G nic的netqueue功能,0的含义请参考 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035701。

ge2/0/26仍然处于down的状态,一怒之下弃之,改用vmnic2/vmnic3来做link-aggregation

[SY-S5800-HSW]dis int bri | incl /34
The brief information of interface(s) under bridge mode:
Link: ADM - administratively down; Stby - standby
Speed or Duplex: (a)/A - auto; H - half; F - full
Type: A - access; T - trunk; H - hybrid
Interface            Link Speed   Duplex Type PVID Description
GE1/0/34             UP   1G(a)   F(a)   T    1    "DL385g8:port3(esxi6)"
GE2/0/34             UP   1G(a)   F(a)   T    1    "DL385g8:port4(esxi6)"

[SY-S5800-HSW]dis link-aggregation verbose Bridge-Aggregation 34

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregation Interface: Bridge-Aggregation34
Aggregation Mode: Static
Loadsharing Type: Shar
  Port             Status    Oper-Key
--------------------------------------------------------------------------------
  GE2/0/34         S         1
  GE1/0/34         S         1

测试

测试过程中用到的测试服务器如下:

Table 2. 测试服务器列表

服务器

ip

master

192.168.111.1

ws12r2-mgmt-2

192.168.111.47

h3c-5800-vlan111

192.168.111.254

ws12r2-mgmt-2是运行在dl385/esxi上的一台虚拟机,网卡桥接了vlan111。

在5800中对bagg34中的两个网口分别执行关闭和开启的操作

  1. 关闭interface gi1/0/24

    观察:

    • 在master中ping ws12r2-mgmt-2

      没有丢包

    • 在ws12r2-mgmt-2中ping h3c-5800-vlan111

      没有丢包

  2. interface gi1/0/24恢复至开启状态。

  3. 关闭jinterface gi2/0/24

    观察:

    • 在master中ping ws12r2-mgmt-2

      丢一个包

    • 在ws12r2-mgmt-2中ping h3c-5800-vlan111

      丢一个包

  4. interface gi2/0/24恢复至开启状态。

结论: vmware的nic teaming(route based on ip hash)与h3c 5800的link-aggregation可以很好的协作,至少比openvswitch和linux bonding的效果好。

linux/openvswitch bonding的链路倒换测试

经测试,openvswitch bonding和linux bonding的线路切换要丢7个包。


1. 对于1G nic来说,netqueue用途不大,开启该功能却会导致esxi网络连接丢失,请参考 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035701