Mellanox IB/Ehternet 模式切换

使用Mellanox 100G网卡做虚拟化的时候发现，安装好Proxmox VE后，在网络中没有发现Mellanox的网卡。

首先检查下系统能否识别到该网卡：

root@localhost:~# lspci | grep -i mellanox
86:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

因为在Ubuntu 22.04 Server系统中默认能够直接使用Mellanox网卡的，现在通过ip a找不到这张网卡，那么原因有以下几个：

• 网卡坏了
• 网卡模式不对

接下来先安装上驱动，驱动安装教程可查看往期的文章，下面主要讲如何切换Mellanox网卡的模式，因为该网卡接的是RoCE交换机，因此需要将网卡模式切换为Ethernet。

启动mst服务：

root@localhost:~# systemctl start mst.service
root@localhost:~# systemctl status mst.service 
● mst.service - LSB: mst
     Loaded: loaded (/etc/init.d/mst; generated)
     Active: active (exited) since Wed 2024-09-11 09:12:24 UTC; 2s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 2837 ExecStart=/etc/init.d/mst start (code=exited, status=0/SUCCESS)
        CPU: 2.278s

Sep 11 09:12:22 localhost systemd[1]: Starting LSB: mst...
Sep 11 09:12:23 localhost mst[2837]: Starting MST (Mellanox Software Tools) driver set
Sep 11 09:12:23 localhost mst[2837]: Loading MST PCI module - Success
Sep 11 09:12:23 localhost mst[2837]: Loading MST PCI configuration module - Success
Sep 11 09:12:23 localhost mst[2837]: Create devices
Sep 11 09:12:24 localhost mst[2837]: Unloading MST PCI module (unused) - Success
Sep 11 09:12:24 localhost systemd[1]: Started LSB: mst.

查看网卡配置文件路径：

root@localhost:~# mst status 
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4115_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:86:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00

查看LINK_TYPE的名称：

root@localhost:~# mlxconfig -d /dev/mst/mt4115_pciconf0 query | grep -i link_type
        LINK_TYPE_P1                                IB(1)  

切换网卡协议从IB模式切换到Ethernet：

root@localhost:~# mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2

Device #1:
----------

Device type:        ConnectX4           
Name:               N/A                 
Description:        N/A                 
Device:             /dev/mst/mt4115_pciconf0

Configurations:                                          Next Boot       New
        LINK_TYPE_P1                                IB(1)                ETH(2)              

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
root@localhost:~# mlxconfig -d /dev/mst/mt4115_pciconf0 query | grep -i link_type
        LINK_TYPE_P1                                ETH(2)  

重启服务器并查看网卡情况：

root@localhost:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:3d:1a:28:15:3c brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
    inet 172.17.1.104/24 brd 172.17.1.255 scope global eno3
       valid_lft forever preferred_lft forever
    inet6 fe80::e63d:1aff:fe28:153c/64 scope link 
       valid_lft forever preferred_lft forever
3: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:3d:1a:28:15:3e brd ff:ff:ff:ff:ff:ff
    altname enp25s0f0np0
4: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:3d:1a:28:15:3f brd ff:ff:ff:ff:ff:ff
    altname enp25s0f1np1
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:3d:1a:28:15:3d brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
6: enp94s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether bc:97:e1:ef:f8:c0 brd ff:ff:ff:ff:ff:ff
7: enp134s0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 24:8a:07:ad:3a:22 brd ff:ff:ff:ff:ff:ff
8: enp94s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether bc:97:e1:ef:f8:c1 brd ff:ff:ff:ff:ff:ff

从输出结果可以看到一个名为 enp134s0np0 的网卡。