有个项目需要对互联网出口流量进行DPI,我的想法是将防火墙的出口流量镜像到一台VM中进行分析,由于出口流量达到了3G,所以需要在物理服务器中用10G口接收,然而如何将接收到的流量送给VM?可以采用sr-iov,也可以采用pass-through,sr-iov当然更省资源,然而sr-iov后的虚拟网卡无法运行在混杂模式下,所以只能采用pass-through。

以下是两种方式的具体配置,虽然sr-iov无法满足要求项目需求,但记录下来已备不时之需。

  • 物理服务器:HP DL585G7

  • 万兆网卡:HP 560 dual port SFP+,芯片为intel 82599

sr-iov

kvm host

  1. 添加内核amd_iommu=on

    $ sudo vim /boot/grub/menu.lst
    ...
    title CentOS (2.6.32-504.1.3.el6.x86_64)
            root (hd0,1)
            kernel /vmlinuz-2.6.32-504.1.3.el6.x86_64 ro root=/dev/mapper/centos-root rd_NO_LUKS rd_LVM_LV=pve/swap rd_LVM_LV=centos/root LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb console=tty1 console=ttyS0,115200n8 amd_iommu=on
    ...
  2. 配置max_vfs

    $ sudo vim /etc/modprobe.d/ixge.conf
    options ixgbe max_vfs=8

    intel 82599支持的最大vfs为64,max_vfs将平均分配至每个物理网口,譬如本例的物理网口有2个,max_vfs为8,则每个物理网口的vf为4。如下所示:

    $ sudo ip link show eth4
    38: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
        link/ether 38:ea:a7:36:e4:cc brd ff:ff:ff:ff:ff:ff
        vf 0 MAC 38:ea:a7:36:e4:01
        vf 1 MAC a6:fc:71:47:e0:6e
        vf 2 MAC 32:f4:c3:18:d6:ae
        vf 3 MAC 4e:65:17:83:65:28

    10G网卡的另外一个端口已经被pass-through给了一台VM,所以在host里面已经看不到该端口,当然更不会有VF。

    $ sudo lspci | grep 82599
    47:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    47:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    47:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    同一个端口下,47:10.0、47:10.2、47:10.4、47:10.6为eth4的Virtual Function,后缀要么同为奇数,要么同为偶数。
  3. 将一个vf分配给vm

    $ sudo vim /etc/libvirt/qemu/c6-64-probe.xml
    ...
        <interface type='hostdev' managed='yes'>
          <source>
            <address type='pci' domain='0x0000' bus='0x47' slot='0x10' function='0x1'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
        </interface>
  4. 分配另一个vf给第二台vm

    $ sudo vim /etc/libvirt/qemu/u1404-64-ntopng.xml
    ...
        <interface type='hostdev' managed='yes'>
          <source>
            <address type='pci' domain='0x0000' bus='0x47' slot='0x10' function='0x3'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
        </interface>

    如果一切正常,vm启动后将自动加载ixgbevf这个驱动:

    $ sudo lsmod | grep ixgbe
    ixgbevf
  5. 测试

    分别在这两台vm中,为网卡分配相同网段的ip地址,可互ping

    然而,根据intel官方的说法,vf并不支持混杂模式,也就是说无法捕获目标mac为非本网卡的数据帧。这从某种程度上来说限制了vf的应用,特别是无法作为ntopng,不过这也容易理解,如果同一个pf下的vf,都能获得整个物理网口的数据帧,在管理和安全上将是一场噩梦。

    所以作为如果想对镜像的流量进行DPI,只能将整个pf,也就是物理网口passthrough给VM。

pass-through

kvm host

  1. 修改vm的配置文件

    $ sudo vim /etc/libvirt/qemu/c6-64-probe.xml
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x47' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
  2. 检查网卡

    $ sudo lsmod | grep ixgbe
    ixgbevf                43812  0
    ixgbe                 275825  0
    dca                     7101  1 ixgbe
    mdio                    4769  1 ixgbe
    ptp                     9614  2 ixgbe,tg3
    
    [root@master mapper]# lspci | grep 82599
    47:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    47:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    47:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    47:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
    
    $ sudo ip link show eth4
    6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
       	link/ether 38:ea:a7:36:e4:cc brd ff:ff:ff:ff:ff:ff
       	vf 0 MAC e6:4d:47:33:80:39
       	vf 1 MAC 7e:0c:59:a9:b7:94
       	vf 2 MAC aa:32:97:a3:3b:dc
       	vf 3 MAC 1e:31:c5:36:ef:dd
       	vf 4 MAC 8e:96:dc:fb:b4:d3
       	vf 5 MAC 3e:e5:54:52:d5:f0
       	vf 6 MAC 82:91:ef:a2:d9:e2
       	vf 7 MAC fa:d5:42:0e:53:ee
    
    $ sudo ip link show eth5
    Device "eth5" does not exist.

    说明host已经将一个物理口(eth5)pass-through给vm了,ip命令无法查看到该物理口的状态。这个例子也说明,同一块网卡的不同端口,可以分别做pass-through和sr-iov。

kvm vm

检查是否成功pass-through

$ sudo lsmod | grep ixgbe
ixgbe                 275921  0
dca                     7101  1 ixgbe
ptp                     9614  1 ixgbe
mdio                    4769  1 ixgbe

$ sudo lspci | grep 82599
00:08.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)