Introduction

The VAST NFS package provides a modified version of the Linux NFS client and server kernel code stacks designed to be built for a large range of kernel versions.

It contains backported upstream NFS stack code from Linux v5.15.x LTS kernel branch with. This allows older kernels to receive the full functionality of newer NFS stack code. Only kernels from 4.15.x and above receive the backports and the fully added functionality. The package applies also to older kernels but with less functionality (legacy VAST NFS).

Features

NFS stack improvements and fixes
Multipath support for NFSv3
Multipath support for NFSv4.1 (not for legacy kernels)
Nvidia GDS integration

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Package download

It is possible to download the latest version of the package with the following command:

curl -sSf https://vast-nfs.s3.amazonaws.com/download.sh | bash -s --

Or a specific version with:

curl -sSf https://vast-nfs.s3.amazonaws.com/download.sh | bash -s -- --version <version>

For building from source, see Source build.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Source build

There are various ways possible to build this package from source.

Packaged build output (RPM or Deb package)
Build without packaging, for the purpose loading directly from a development tree

This section provides information about how to accomplish it.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Supported kernel versions

This package is supported on the following Linux distribution kernels:

Red Hat Enterprise Linux 8.1 to 8.9, 9.0 to 9.6 and derivatives (CentOS, Rocky, Alma)
SUSE Enterprise Linux 15
Ubuntu 24 LTS (up to kernel 6.8)
Ubuntu 22 LTS
Ubuntu 20 LTS
Ubuntu 18 LTS

Generic upstream kernel support ranges are the following:

Linux v4.15 to v6.8.

Attempting to build for an unsupported kernel may produce compilation errors.

NOTE: Some older distribution kernels (such as RH 8.1) may require a recent Mellanox OFED being installed, as it provides a more recent IB layer along with newer kernel APIs.

Legacy VAST NFS 3.x fallback

The source package for VAST NFS 4.x includes the older VAST NFS 3.x branch code targeting older kernels that belong to:

Red Hat Enterprise Linux 7.x and derivatives (CentOS 7.x)
SUSE Enterprise Linux 12

Building the 4.x source package will result in a package marked as 3.x due to the less supported functionality. It currently has less functionality than 4.x (e.g. for NFSv4.1 mounts, no remoteports-based multipath).

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Mellanox OFED support

All versions of Mellanox OFED that successfully install on a Linux system should be supported by this package. Newer OFED versions are preferred.

If the Mellanox OFED is installed on a system, this package can also be built in such a way that NFS RDMA is functional and working using the Mellanox RDMA stack instead of the inbox kernel RDMA stack, which the installation of the Mellanox OFED supersedes.

The state where the Mellanox OFED is installed is detected automatically by the build scripts, and the resultant package will only work with that particular Mellanox OFED version.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Building an installable package

The following command will build the source package. It detects the currently running kernel, distribution, and optional Mellanox OFED installation.

./build.sh bin

Following a successful build, the resultant package should reside in a dist/ sub-directory, for example:

$ ls -1 dist/vastnfs* | grep -v debug

vastnfs-4.0-kernel_4.18.0_348.20.1.el8_5.x86_64.rpm

For older kernel, the build relies on the legacy VAST NFS 3.x sources that are included in the source package.

See Installation for installation instructions.

Arguments

The build.sh bin command can receive the following optional arguments:

--without-rpcrdma: Don't build RDMA support
--dkms: For Debian-based system, generate a DKMS-based package even if no DKMS-based Mellanox OFED is installed.
--no-ofed: Don't detect Mellanox OFED installation, and instead build a driver targeting the inbox kernel instead.

Regarding DKMS

DKMS is a source-based package for which the binary driver is built on-demand. This is useful for minor kernel updates.

If Mellanox OFED installation is detected, and if that installation is DKMS-based, then the resultant package will also be DKMS-based by default. This is currently supported only on Debian-based system. Otherwise, the built package will install binaries that depend on the current kernel version.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Development build

The following command should build the driver for the current kernel, assuming that a compiler and the kernel headers are installed for the local machine. Further development package of the relevant Linux distro may be needed.

make

When this command is issued, no package is being generated and instead the kernel modules reside within the built source tree.

To load the newly built development version, the following command can be issued (sudo is being used by the script):

./scripts/control.sh dev-reload

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Signing kernel modules for secure boot

Some systems have Secure Boot enabled, meaning that unauthorized kernel code cannot be loaded.

$ dmesg | grep secure
[    0.000000] secureboot: Secure boot enabled

Only trusted private keys can sign kernel modules that can be loaded.

At package build time

If you are building the package yourself, likely that you already have a key that that is used to sign its kernel modules.

For building a binary package where the module are signed, provide two environment variables, MODULE_SIGN_PRIV_KEY and MODULE_SIGN_PUB_KEY pointing to the key pair:

export MODULE_SIGN_PRIV_KEY=<path to private key>
export MODULE_SIGN_PUB_KEY=<path to public key>
./build.sh bin

The produced package will contain signed kernel modules.

Expect to see "Found Sign tool at" in the output to confirm that signing took place.

If you don't have a key

If you are new to secure boot, it is possible to create your own key, and enroll it in the BIOS. This is a general procedure is not specific to the VAST NFS driver.

Create a key

openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -nodes -days 3650 -subj "/CN=My Custom Key/"

Enroll the key

sudo mokutil --import MOK.der

You will be prompted to set a password. Make sure to remember this password, as you'll need it during the next boot.

Reboot your system

After enrolling the key, reboot your system. During the boot process, you'll see a blue screen with the title "MOK management." This is where you'll use the password you set earlier.

Enroll the key in MOK

Follow these steps in the MOK management screen:

Choose "Enroll MOK."
Select "Continue."
Choose "Yes" to confirm that you want to enroll the key.
Enter the password you set earlier.
Select "OK" to complete the enrollment process.
Your system should now boot normally, and your custom key is enrolled in MOK.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Installation

This section deals with installation of the binary package. The goal is for the installed NFS stack to supersede the inbox kernel NFS stack.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

RHEL 8.x/9.x

Install:

sudo yum install ./dist/vastnfs-[1-9]*.x86_64.rpm

sudo dracut -f

Following install, rebooting is required for the newly installed drivers to load.

Verify that the package is indeed installed (versions should match):

rpm -qa | grep vastnfs

To further validate installation, verify that rpcrdma in the right path exists and get loaded from it. All the added kernel modules go under updates/bundle:

rpm -qif /lib/modules/`uname -r`/extra/vastnfs/bundle/net/sunrpc/xprtrdma/rpcrdma.ko

$ modinfo rpcrdma | grep filename:
filename:       /lib/modules/..../extra/vastnfs/bundle/net/sunrpc/xprtrdma/rpcrdma.ko

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Installing under Ubuntu/Debian

For Ubuntu/Debian, either a DKMS-based package or kernel module binary package was generated by the build command (see build section)

For binary packages:

sudo dpkg -i ./dist/vastnfs-modules_*-generic_amd64.deb
sudo update-initramfs -u -k `uname -r`

Alternatively,

For DKMS packages:

sudo dpkg -i ./dist/vastnfs-dkms_*.deb

This should result in the generation of kernel module binaries by the DKMS subsystem.

Verify that the package is indeed installed (versions should match):

dpkg -l | grep vastnfs-modules

To further validate installation, verify that rpcrdma in the right path exists and get loaded from it. All the added kernel modules go under updates/bundle:

dpkg -S /lib/modules/`uname -r`/updates/bundle/net/sunrpc/xprtrdma/rpcrdma.ko

Verification

Reboot, to ensure that this version is loaded correctly.

Once rebooted, and to verify fully once it is loaded, for each kernel module, loaded version should match installed version. For example:

$ cat /sys/module/sunrpc/srcversion
4CC8389C7889F82F5A59269

$ modinfo sunrpc | grep srcversion
srcversion:     4CC8389C7889F82F5A59269

(This is only an example - srcversion is different for each build and kernel module).

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Configuration

This section deals with configurations for the provided NFS stack in this package.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Multipath

To make an efficient load balanced used of single NFS mount, multipath should be enabled.

Under the multipath scheme used in this package, all participating local and remote ports have distinct network IP addresses. This section deals with the configuration that enable this mode.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

RHEL 9.x/8.x

This package is supported on RHEL, and the RHEL-based distributions such as CentOS Linux, Rocky Linux, and Alma Linux.

NOTE: On RHEL 7.x-based kernels the package installs in backward-compatibility mode: it only provides multipath on NFSv3 mounts.

Here we demonstrate multipath with two local interfaces on the same subnet.

First we need to install NetworkManager-config-routing-rules.

yum install NetworkManager-config-routing-rules

Configuring for multipath with `nmcli`

Suppose we have a host with two network cards, ib0 and ib1. The network cards are directly connected to the server's switch and belong to the same IP subnet. We need to define source-based routing to the server (with IP 192.168.40.11) to take advantage of multipath mode.

$ nmcli -f 'GENERAL.DEVICE,IP4.ADDRESS' device show
GENERAL.DEVICE:                         enp1s0
IP4.ADDRESS[1]:                         192.168.122.235/24

GENERAL.DEVICE:                         ib0
IP4.ADDRESS[1]:                         192.168.40.1/24

GENERAL.DEVICE:                         ib1
IP4.ADDRESS[1]:                         192.168.40.2/24

GENERAL.DEVICE:                         lo
IP4.ADDRESS[1]:                         127.0.0.1/8

Verify that routing tables 101 and 102 are empty:

ip route show table 101
ip route show table 102

Add symbolic names for custom routing tables:

echo '101 101' >> /etc/iproute2/rt_tables
echo '102 102' >> /etc/iproute2/rt_tables

Add custom routing tables for ib0 and ib1:

nmcli device modify ib0 ipv4.routes "192.168.40.0/24 src=192.168.40.1 table=101"
nmcli device modify ib1 ipv4.routes "192.168.40.0/24 src=192.168.40.2 table=102"

Add custom routing rules with priority 100:

nmcli device modify ib0 +ipv4.routing-rules "priority 100 from 192.168.40.1 table 101"
nmcli device modify ib1 +ipv4.routing-rules "priority 100 from 192.168.40.2 table 102"

Reload configuration and verify it:

$ nmcli connection reload
$ ip route get 192.168.40.11 from 192.168.40.1
192.168.40.11 from 192.168.40.1 dev ib0 table 101
    cache
$ ip route get 192.168.40.11 from 192.168.40.2
192.168.40.11 from 192.168.40.2 dev ib1 table 102
    cache

If needed, troubleshoot with commands:

ip route show table 101
ip route show table 102
ip rule show

Configuring for multipath using old style `network-scripts` interface

We configure new source routing tables:

echo '101 101' >> /etc/iproute2/rt_tables
echo '102 102' >> /etc/iproute2/rt_tables

For this example we assume two interfaces are configured via ifcfg- scripts:

$ grep IPADDR /etc/sysconfig/network-scripts/ifcfg-ib0
IPADDR=192.168.40.1
$ grep IPADDR /etc/sysconfig/network-scripts/ifcfg-ib1
IPADDR=192.168.40.2

For each interface we need to add a route- and rule- files:

$ cat /etc/sysconfig/network-scripts/route-ib0
192.168.40.0/24 via 192.168.40.1 table 101
$ cat /etc/sysconfig/network-scripts/rule-ib0
from 192.168.40.1/32 table 101

NOTE: This is only an example. The IP addresses you pick depend on your network configuration.

After reloading with nmcli connection reload, the ip command with regard to routing should look like this (notice the two added lookup lines):

$ ip rule
0:      from all lookup local
32764:  from 192.168.40.1 lookup 101
32765:  from 192.168.40.2 lookup 102
32766:  from all lookup main
32767:  from all lookup default

$ ip route show table 101
192.168.40.0/24 via 192.168.40.1 dev ib0

$ ip route show table 102
192.168.40.0/24 via 192.168.40.2 dev ib1

Verify that the IPs and routing tables appear correctly. Below are examples which will vary based on environment:

IP addresses:

$ ip a s | grep 192.168.40

    inet 192.168.40.1/24 brd 192.168.40.255 scope global ib0
    inet 192.168.40.2/24 brd 192.168.40.255 scope global ib1

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Ubuntu

Ubuntu installations can use netplan to specify the network configuration. For each port, we need to have one IP address in the subnet. For each IP address, source routing needs to be enabled.

For example, in the updated 01-netcfg.yml for local port configuration (ib0, ib1, ib2, ib3), append routes and routing-policy rules:

    ib0:
        dhcp4: no
        addresses: [172.25.1.101/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.101
           table: 101
        routing-policy:
         - from: 172.25.1.101
           table: 101
    ib1:
        dhcp4: no
        addresses: [172.25.1.102/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.102
           table: 102
        routing-policy:
         - from: 172.25.1.102
           table: 102
    ib2:
        dhcp4: no
        addresses: [172.25.1.103/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.103
           table: 103
        routing-policy:
         - from: 172.25.1.103
           table: 103
    ib3:
        dhcp4: no
        addresses: [172.25.1.104/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.104
           table: 104
        routing-policy:
         - from: 172.25.1.104
           table: 104

NOTE: This is only an example. The IP addresses you pick depend on your network configuration.

The apply the new configuration, run:

sudo netplan apply

Verify that the IPs and routing tables appear correctly. Below are examples which will vary based on environment:

IP addresses:

$ ip a s | grep 172.25.1

    inet 172.25.1.101/20 brd 172.25.1.255 scope global ib0
    inet 172.25.1.102/20 brd 172.25.1.255 scope global ib1
    inet 172.25.1.103/20 brd 172.25.1.255 scope global ib2
    inet 172.25.1.104/20 brd 172.25.1.255 scope global ib3

Routing tables:

$ for i in 101 103 102 104; do ip route show table $i; done;

172.25.1.0/20 via 172.25.1.101 dev ib0 proto static
172.25.1.0/20 via 172.25.1.102 dev ib1 proto static
172.25.1.0/20 via 172.25.1.103 dev ib2 proto static
172.25.1.0/20 via 172.25.1.104 dev ib3 proto static

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Uninstalling the package

This section deals with removal of the driver package.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

RHEL 8.x/9.0

Uninstalling the driver involves removal of RPM package and making sure the boot image is regenerated:

sudo yum remove -y vastnfs
sudo dracut -f

To unload the driver, a reboot is recommended.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Ubuntu

Uninstalling the driver involves removal of the Deb package and making sure the boot image is regenerated:

sudo apt remove -y vastnfs-modules
sudo update-initramfs -u -k `uname -r`

To unload the driver, a reboot is recommended.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Usage

There are extra features that this stack enables and these are described in this section.

If no extra features are used, the usage and behavior of the provided NFS stack matches the upstream or distribution NFS stack versions.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Mount parameters

This package adds to the standard mount parameters that the original Linux NFS stack originally supports.

VAST NFS parameters

The VAST NFS client supports the following additional mount parameters that can be used individually or together.

remoteports=<IP-SPEC | dns> enables multipath allowing a single to spread traffic across multiple endpoint IPs. The parameter takes a list of addresses.

The IP specification allows addresses to be specified using an inclusive range operator with -: <FIRST-IP>-<LAST-IP>.

Multiple ranges and individual IPs can be combined using the ~ operator: <FIRST-IP>-<LAST-IP>~<IP>.

Example usage:
```
mount -o vers=3,remoteports=172.25.1.1-172.25.1.32 172.25.1.1:/
```
With a special mode remoteports=dns, the client will issue one or more DNS queries with the mount target DNS name and will get back an array of addresses and use those as the list of remote addresses as if it was passed explicitly. Example:
```
mount -t nfs -o vers=3,nconnect=16,remoteports=dns vippool3.company.lab:/ /mnt
```
Notes:
- Do not use this option in driver versions prior to 4.0.29.
- This feature expects that no DNS caching done on the DNS server or locally. If caching exist on the DNS server, and the number of DNS addresses returned in the DNS response is lower than the value of nconnect, it may result in unintended reduction of the remoteports array due to repeating (cached) DNS responses. In order to test if DNS caching is enabled in your environment run:
```
$ for i in $(seq 3); do dig <dns-name>.example.com +short; done
10.10.12.7
10.10.12.6
10.10.12.2
```
  If the responses contain the same ip addresess, then caching is enabled.
- If DNS caching is enabled, It is required to increase the number of addresses in the DNS server response in order for the client to correctly distribute connections across different remote ports. See example (4 addresses in the DNS response):
```
$ dig <dns-name>.example.com +short
10.10.12.1
10.10.12.14
10.10.12.9
10.10.12.3
```
localports=<INT-SPEC> controls which client interfaces are used to send traffic. By default the vastnfs driver will send traffic over one network interface. If the client has multiple network interfaces you can add additional interfaces with this parameter.

The interface specification uses the same form as remoteports above with the - and ~ operators and the IPs assigned to the client interfaces.

Interface names can be used instead of the IP. The driver will resolve the IP currently on the interface and use that IP as described above.

Example usage:
```
mount -o vers=3,localports=172.25.1.1-172.25.1.2 172.25.1.1:/

mount -o vers=3,localports=eth0~eth1 172.25.1.1:/
```
The localports parameter is not necessary for multipath to work. However if the local machine has more than one IP address configured for the local subnet, it may be useful to prevent a single local port from trying to surpass its maximum line rate while the other local ports are underused.

Also, for GDS-based setups, it is necessary for making the transport layer to perform GPU-based routing for GPU overload to be utilized.

The localports mount option isn't compatible with IP reverse path validation (RFC 3704). When using this option, make sure that the rp_filter sysctl option is disabled on the corresponding network interfaces using:
```
sysctl -w "net.ipv4.conf.all.rp_filter=0" &&
sysctl -w "net.ipv4.conf.<iface>.rp_filter=0"
```
Or, disable rp_filter globally by placing following in the /etc/sysctl.conf file: net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.all.rp_filter = 0 For some distros it may be additionally required to disable rp_filter in the /etc/sysctl.conf file for each corresponding network interface.
nconnect=<NUM> enables multiple connections for a single mount. This parameter is available on recent Linux kernels and the vastnfs client allows it to be used on Linux kernels for which it was not backported.

The recommended value for this parameter is 4 for RDMA mounts and 8 for TCP mounts.

Example usage:
```
mount -o vers=3,nconnect=8 172.25.1.1:/
```

Multipath mount examples

Often the above parameters are combined and used together.

Consider this multipath mount:

mount -o vers=3,nconnect=8,remoteports=172.25.1.1-172.25.1.32 172.25.1.1:/

This mount command will result in 8 TCP connections, going to a psuedo-random sub-range of 8 addresses under the provided 32 address range.

While multipath can be used with TCP (proto=tcp), using it with RDMA (proto=rdma) will have less CPU utilization.

Here is a more advanced example of an RDMA mount with 4 connections, 8 remote ports and 4 local ports:

mount -o proto=rdma,port=20049,vers=3,nconnect=4,localports=172.25.1.101-172.25.1.104,remoteports=172.25.1.1-172.25.1.8 172.25.1.1:/

Advanced parameters

NFS optimizations

forcerdirplus - Instructs the NFS client to always send an NFSv3 READDIRPLUS requests for servers that support it. On some remote systems this can improve directory listing performance significantly.
relmtime - Don't block stat() calls while there are pending writes. This improves the scenario in which 'ls -l' blocks while the same client is writing to a file under the listed directory.
optlockflush - This is an optimization for applications that use a lock to protect a read-mostly file. The assumption that GETATTR costs less than zapping the read-cache, and that we can use GETATTR to detect whether the file was modified at the server.

NFS operational parameters

noextend - Turns off the default extend-to-page optimization in writes, useful for some specific applications having multiple clients on NFSv3.

Additional transport parameters

These parameters can be used for experimentation and fine-tuning.

noidlexprt - Do not disconnect idle connections.
remoteports_offset=<NUM> - Controls the offset into picking out of remoteports for transports if the number of actual transports via nconnect is smaller than the amount of IPs given here. If not given a pseudo-random number is picked based on source IP address.
localports_failover - Special mode for multipath where failing transports can temporarily move from local addresses that cannot serve connections, for example on a local cable disconnect. For this option to work, noidlexprt needs to be passed too, along with an nconnect value that is at least a multiple of the port count in remoteports over localports. This option is supported for RDMA only.

Example:
```
mount -o rdma,vers=3,noidlexprt,localports_failover,localports=172.25.1.101-172.25.1.104,remoteports=172.25.1.1-172.25.1.8 172.25.1.1:/
```
This is only supported for NFSv3.
spread_reads or spread_writes - Whether a single file's IO should use a single connection or multiple connections. Single connection enjoys a few optimizations on the server side for latency but multiple increases the potential bandwidth.

These are only supported for NFSv3.
mdconnect - Specifies the number of additional transport connections dedicated exclusively to metadata operations. Limited at 8. When enabled, the transports defined by nconnect handle only READ/WRITE requests, while the extra connections defined by mdconnect serve metadata requests exclusively. This option is particularly useful in scenarios where heavy data I/O causes metadata operations to be queued and starved on shared transport channels.

This is only supported for NFSv3.
nosharetransport - This option causes the client to establish its own isolated transport connections. The client will not share the transport connections with any other mount done before or after.

This is only supported for NFSv3.
sharetransport=N - N is positive number that identifies mounts sharing the same transport connections. If two or more mounts to a particular NFS server have a different value of sharetransport, these mounts will use different connections. If you don't specify the option value for mounts to a particular NFS server, all the mounts will shate one transport connections.

This is only supported for NFSv4.x

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

vastnfs-ctl

This helper script is installed with the package, and can assist in various situations.

The usage of this script is optional in most cases.

Status

Observe the status of the loaded NFS services and kernel modules

vastnfs-ctl status

Example output:

version: v4.0-pre4-11-ge520dd8eb9cb
kernel modules: sunrpc rpcrdma compat_nfs_ssc lockd nfs_acl auth_rpcgss nfsd rpcsec_gss_krb5 nfs nfsv3 nfsv4
services: rpcbind.socket rpcbind
rpc_pipefs: /var/lib/nfs/rpc_pipefs

Reload

The following command will try to reload the NFS stack. It will first try to unmount all NFS file systems, and then reload all the necessary kernel modules.

vastnfs-ctl reload

Example output:

vastnfs-ctl: stopping service rpcbind.socket
vastnfs-ctl: umounting fs /var/lib/nfs/rpc_pipefs
vastnfs-ctl: unloading kmod nfsv4
vastnfs-ctl: unloading kmod nfsv3
vastnfs-ctl: unloading kmod nfs
vastnfs-ctl: unloading kmod rpcsec_gss_krb5
vastnfs-ctl: unloading kmod nfsd
vastnfs-ctl: unloading kmod auth_rpcgss
vastnfs-ctl: unloading kmod nfs_acl
vastnfs-ctl: unloading kmod lockd
vastnfs-ctl: unloading kmod compat_nfs_ssc
vastnfs-ctl: unloading kmod rpcrdma
vastnfs-ctl: unloading kmod sunrpc
vastnfs-ctl: loading kmod sunrpc
vastnfs-ctl: loading kmod rpcsec_gss_krb5
vastnfs-ctl: loading kmod rpcrdma
vastnfs-ctl: loading kmod nfsv4
vastnfs-ctl: loading kmod nfsv3
vastnfs-ctl: loading kmod nfsd
vastnfs-ctl: loading kmod nfs_acl
vastnfs-ctl: loading kmod nfs
vastnfs-ctl: loading kmod lockd
vastnfs-ctl: loading kmod compat_nfs_ssc
vastnfs-ctl: loading kmod auth_rpcgss
vastnfs-ctl: mounting fs /var/lib/nfs/rpc_pipefs
vastnfs-ctl: starting service rpcbind.socket

Tracing

The vastnfs-ctl utility comes with a helper command for tracing the NFS stack. Please see a short tutorial on using vastnfs-ctl trace.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Monitoring and diagnosis

This section contains resources that may help in diagnosing NFS mount issues.

There are specific VAST NFS extensions such as multipath that we should consider when debugging. Also, there can be issue outside the scope of differences between this implementation and the upstream or distribution-based implementation.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Detecting running version

To verify that the driver is loaded, check if the /sys/module/sunrpc/parameters/nfs_bundle_version exists. The content of the file can be obtained via:

cat /sys/module/sunrpc/parameters/nfs_bundle_version

The content of the file indicates the loaded version. If it does not exist, then either it was not loaded yet, or that the NFS layer that is provided with the installed kernel is loaded instead - in that case the upper directory /sys/module/sunrpc exists.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Mount and transport state

Normally, the mountstats command shows NFS mount status. We have extended the interface it uses, /proc/self/mountstats, with our own extra state reporting via /sys. The extended information contains the IP addresses related to the transport, and a string that indicates its state flags.

The extended form of it is obtainable via /sys/kernel/debug. For the association to an NFS mount, we need to obtain the related sunrpc-id of the mount point.

But first, we need to enable sunrpc-id reporting. This can be done using the following command after boot:

echo 1 | sudo tee /sys/module/sunrpc/parameters/expose_id_in_mountstats

Then, identify the sunrpc-id relevant to the mount point by looking into /proc/self/mountstats:

$ cat /proc/self/mountstats | grep -E 'fstype nfs|sunrpc-id'
device 192.168.40.7:/opt/export mounted on /mnt/export with fstype nfs statvers=1.1
        sunrpc-id:      4

Now we are ready to fetch the full mountstats via the following command:

    sudo cat /sys/kernel/debug/sunrpc/rpc_clnt/4/stats

The added information contains multipath IP address information per xprt (transport) and xprt state in string format. The numbers in the first xprt: line are counters and other numbers related that transports.

For example:

        xprt:   rdma 0 0 1 0 24 3 3 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.101 -> 172.25.1.1, state: CONNECTED BOUND
        xprt:   rdma 0 0 1 0 24 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.102 -> 172.25.1.2, state: CONNECTED BOUND
        xprt:   rdma 0 0 1 0 23 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.103 -> 172.25.1.3, state: CONNECTED BOUND
        xprt:   rdma 0 0 1 0 22 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.104 -> 172.25.1.4, state: CONNECTED BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.101 -> 172.25.1.5, state: BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.102 -> 172.25.1.6, state: BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.103 -> 172.25.1.7, state: BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.104 -> 172.25.1.8, state: BOUND

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

External resources

Official upstream guide for admins: Linux v5.15 NFS admin guide.

On the linux-nfs.org site: General troubleshooting recommendations - Linux NFS

Linux distribution pages

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Tracing

The supplied vastnfs-ctl helper script allows tracing the driver. The underlying mechanism uses trace-cmd in order to accomplish that.

Prerequisites

`trace-cmd`

Make sure that the trace-cmd command is available.

Red Hat-based: dnf install -y trace-cmd Debian-based: apt install trace-cmd

It is invoked by the script using sudo.

Loaded kernel modules

The vastnfs-ctl status command should indicate that the modules are already loaded prior to starting trace.

# vastnfs-ctl status | grep modules
kernel modules: sunrpc rpcrdma lockd nfs_acl auth_rpcgss rpcsec_gss_krb5 nfs nfsv3 nfsv4

Performing trace

For trace collections there are 3 run modes:

Live - traces are emitted as they happen.
Collect to RAM.
Collect to RAM and files.

Live traces

This is useful when there are not a lot of events, such as in the mount stage.

There are present commands for this, for example:

vastnfs-ctl trace cmd meta

Collect to RAM

Manages cyclic buffers in RAM - execution goes to background. Later, the traces can be saved to a file.

In the following example, we use 1GB of buffers for data-based IO and 100MB for transports, connection management and error handling.

vastnfs-ctl trace cmd collect \
    buffer meta 100 class meta \
    buffer io 1000 class all

Output should be as followed:

Events:
     class meta 149 events bufsize 100 MB
     class all 493 events bufsize 1000 MB

Instructing trace-cmd to start background collection to RAM...

Saving the buffers to a file

$ vastnfs-ctl trace cmd save
Saving tracing data to nfs-traces-20221020-122203.tar.gz... (via /tmp/vastnfs-tmp-BzoxUUyTPp)

(note: the tracing RAM buffer is pruged and 'collect' will be
       needed to be used again to collect new data)

-rw-r--r-- 1 root root 2202606 Oct 20 12:22 /home/user/workdir/nfs-traces-20221020-122203.tar.gz

Done. To start a new session use 'collect' again.

Observing a saved trace report

The developers can use the extracted nfs-traces-<timestamp>.tar.gz tarball and trace-cmd report to see meaningful output of the saved traces.

Collect to RAM and files

Manages cyclic buffers both in RAM and files, where the RAM is immediately flushed to files. Execution stays in foreground. Useful for large volume of traces.

Append an outdir <path> to the collect command. The directory needs to exist before execution.

For example:

$ vastnfs-ctl trace cmd collect buffer meta 100 class meta   buffer io 1000 class all outdir saved-traces-20221001
Checking tracepoints...

Output directory: saved-traces-20221001
Output FS type: ext4
FS storage for traces: 696192 MB

Events:
     class meta 149 events bufsize 100 MB
     class all 493 events bufsize 1000 MB

Saving logfile: /home/dan/vd/kernel/vastnfs/saved-traces-20221001/collect.log
Running trace-cmd. Wait for it to prompt readiness.
Hit Ctrl^C to stop recording
...
...
Collection done, saved-traces-20221001 can be packed and sent.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Troubleshooting

This section includes some troubleshooting tips when building or installing the driver.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

Installation problems

`dnf` errors about conflicting prerequisites

If you have a custom-built kernel you might encounter an error like this:

sudo dnf install dist/vastnfs-4.0.23-kernel_5.15.147.el9.x86_64.rpm
...
Error:
 Problem: conflicting requests
  - nothing provides kernel(PDE_DATA) = 0x0e0f5bd2 needed by vastnfs-4.0.23-kernel_5.15.147.el9.x86_64 from @commandline
  - nothing provides kernel(__alloc_pages) = 0xc7233469 needed by vastnfs-4.0.23-kernel_5.15.147.el9.x86_64 from @commandline
  - nothing provides kernel(__bitmap_weight) = 0x63c4d61f needed by vastnfs-4.0.23-kernel_5.15.147.el9.x86_64 from @commandline
  - nothing provides kernel(__cpu_online_mask) = 0x564405cb needed by vastnfs-4.0.23-kernel_5.15.147.el9.x86_64 from @commandline
...

The problem is that the kernel-core package for your kernel is not reporting features that the vastnfs package requires. This could either be due to:

The target kernel is installed from a custom kernel RPM, but that RPM was not properly built to include kabichk 'provides' specifications (standard RHEL-based kernels always have kabichk).
The target kernel wasn't installed with rpm/dnf/yum but rather installed using some custom setup.

Confirm that your kernel is installed as a package by checking with rpm:

$ rpm -qa | grep kernel-core

You can see what kernel() features that kernel-core package says it provides with:

$ rpm -q --provides kernel-core-$(uname -r) | grep kernel

If the kernel was installed with a package but the package doesn't say it provides the symbols vastnfs is looking and you suspect it's a packaging issue with your custom kernel RPM you can force the install with --nodeps to bypass the dependency check.

$ sudo rpm -i --nodeps ./dist/vastnfs-vast*.x86_64.rpm

A reboot is required for the new drivers to load.

If the vastnfs driver works in this configuration consider updating your custom RPM kernel builds to specify the kernel features it provides to avoid this in the future.

`sunrpc: Unknown symbol mlx_backport_dependency_symbol (err -2)`

This means VAST NFS tried to load but the MOFED kernel layer is not loaded and cannot load.

The reason is that by default a VAST NFS build depends directly on the MOFED kernel modules in case the MOFED installation exists.

The usual scenario for this is following a kernel upgrade. After a kernel upgrade, if the MOFED kernel modules are not prepared for the new kernel, they are not loading, and system administrators sometimes don't notice it because the inbox Infiniband layers are loaded instead and cover for that. However, VAST NFS directly depends on the Infiniband layer it was compiled against.

The most trivial way is to reinstall MOFED (or at minimum, its kernel module subpackage), followed by VAST NFS rebuild and install.

Another option, suppose that MOFED installation is no longer desired, is to run ./build.sh bin --no-ofed to create a VAST NFS build that does not depend on it, and reinstall it instead.

Document generated on 2025.06.13 19:31 for v4.0.34, branch 4.0

4.0.34

Source: vastnfs-4.0.34.tar.xz

Fix a regression with spread_reads and spread_writes performance.
Fix a issues related to mdconnect implementation.
Fix a crash with buffered_internal_writeback related to writev calls.

4.0.33

Source: vastnfs-4.0.33.tar.xz

Introduce mdconnect mount option to isolate metadata RPC connections for NFSv3 workloads
For RHEL9.5 and newer kernels, update compatibility layers
Add Debian installation dependency on nfs-common and run weak-modules on RHEL8/9 to ensure kernel modules are properly integrated
Support separate transport channels for NFS mounts via new nosharetransport (NFSv3) and sharetransport=N (NFSv4) mount options
Enhance Makefile: centralize NFS config definitions; fix GCC version parsing; improve RPM-based distribution error messaging
Improve scripts: add missing trace events and revert an unintended change
Fix sunrpc crash with backchannels when rpc_task_state_transition is enabled
Correct tracepoint error in trace_nfs3proc_access

4.0.32

Source: vastnfs-4.0.32.tar.xz

Fix a backporting bug for older kernels (5.2 and below) that caused FD leak in rpc-gssd, relevant only for setups where Kerberos is used on those kernels.

4.0.31

Source: vastnfs-4.0.31.tar.xz

Fix a build issue where source tarball is unpack in a path that has symlink components
Fix module signing on Ubuntu
Backport 'sunrpc: fix NFSACL RPC retry on soft mount' from upstream
For KRB5 setups using PID-based credentials cache, add fixes for NFSv4 IO recovery modes
Don't ignore localports= mount option with NFSv4.x
Don't ignore noidlexprt mount param with NFSv4.x
Extend vastnfs-ctl with status commands for transports and clients
For NFS4.x show remoteports in mountstats
Documentation fixes

4.0.30

Source: vastnfs-4.0.30.tar.xz

Fix a rare case of stuck IOs in NFS v4.1 remoteports= multipath
Fix documentation regarding remoteports=dns
Fix mount failure when remoteports=dns is passed and nconnect is not
Remove unwanted caching of DNS results when remoteports=dns is used
Fix an issue with buffered_internal_writeback and unaligned user buffers
Show the extra mount options in the output of mount for existing mounts (except: NFSv4.1 remoteports mode)

4.0.29

Source: vastnfs-4.0.29.tar.xz

Support up to Linux 6.8
Fix stability issues regard remoteports=dns feature
Fix stability issues regard passing network device names in localports=
Backport fix for idmapper upcall (upstream commit 51fd2eb52c0ca827)
Further fixes for recovery states when using RDMA bonding devices

4.0.28

Source: vastnfs-4.0.28.tar.xz

Introduced a new buffered_internal_writeback module parameter, which enhances write performance on older kernels. When enabled, it applies to synchronous and large page-aligned writes. These writes are buffered but behave similarly to O_DIRECT writes by bypassing the overhead of the Linux writeback.
Added support for Oracle Linux 8.9.

4.0.27

Source: vastnfs-4.0.27.tar.xz

Bugfix for Linux kernel 4.19.x LTS tree changes
Fix for remoteports= ranges on IPv6 addresess
Support remoteports=dns mode
Fix for recovery states when using RDMA bonding devices
Automatic readlink retry via lookup. This allows clients to replace symlinks in directories using rename without causing other clients to fail when using them for path traversal. This improvement is pending upstream NFS client contribution.
When debugging with all trace-events enabled, fix for stability
Documentation changes

4.0.26

Source: vastnfs-4.0.26.tar.xz

Support for Linux kernels up to Linux 6.6
Import stability patches from Linux LTS 5.15.x tree up to 5.15.147
Add an optlockflush mount option (see documentation)
Add a noextend mount option (see documentation)
Debian packaging: add missing depmod call on install
Debian packaging: fix build error on compressed kernel modules
Fix backporting issue regarding srcaddr access in sysfs
build.sh: fix Oracle Linux detection
Various documentation changes

4.0.25

Source: vastnfs-4.0.25.tar.xz

Add relmtime mount option (see documentation).

4.0.24

Source: vastnfs-4.0.24.tar.xz

Bugfix for rare cases of lost writeback in backported code from Linux 5.18 to 6.7.

4.0.23

Source: vastnfs-4.0.23.tar.xz

Support RHEL 9.3.
Support up to upstream kernels Linux 6.2.
Fix the localports_failover implementation which was broken due to a trivial bug.
Allow interop with builds of Lustre (only ones that have Kerberos disabled).

4.0.22

Source: vastnfs-4.0.22.tar.xz

Support passing network device names in localports=.

4.0.21

Source: vastnfs-4.0.21.tar.xz

Fix crash when using xattrs under NFS protocol version 4.2.

4.0.20

Source: vastnfs-4.0.20.tar.xz

Improve kernel header detection on Debian
Linux 5.15.x LTS sync to v5.15.126.
nfsd: fix support for xattrs on various kernels
3.9.x fallback sync: v3.9.30.

4.0.19

Source: vastnfs-4.0.19.tar.xz

Fix for upcoming RHEL 8.9 kernels.
NFSv4.1 support for NFS4_RESULT_PRESERVER_UNLINKED

4.0.18

Source: vastnfs-4.0.18.tar.xz

Fix for Ubuntu kernels 5.19.x.

4.0.17

Source: vastnfs-4.0.17.tar.xz

Support for Linux kernels up to 5.19, including.
Support for RHEL 9.2.
Tracing improvements.

4.0.16

Source: vastnfs-4.0.16.tar.xz

Fix nfsd (the NFS server service) for various platforms, instead of loading a stub. Platforms include later RHEL 8.x and above ; Linux 5.4.0 kernels and above.
Add the spread_reads and spread_writes mount parameters. These allow remoteports=-based NFSv3 multipath spread of IOs on single files, in contrast to the default where file handles pin to remote IP address to obtain cache locality in remote servers. NFSv4.1 remoteports= multipath is unchanged.

4.0.15

Source: vastnfs-4.0.15.tar.xz

Build fixes for OFED-5.9.
Fix stuck IOs in NFS v4.1 remoteports= multipath.
Packaging sync of legacy 3.x branch to 3.9.27.

4.0.14

Source: vastnfs-4.0.14.tar.xz

Packaging sync of legacy 3.x branch to 3.9.26.

4.0.13

Source: vastnfs-4.0.13.tar.xz

Build fix for Scientific Linux detection.
Support RHEL 9.1.

4.0.12

Source: vastnfs-4.0.12.tar.xz

Fix read_ahead_kb default for kernel targets to be between 60 KB and 128 KB.

4.0.11

Source: vastnfs-4.0.11.tar.xz

Package meta-data fix.

4.0.10

Source: vastnfs-4.0.10.tar.xz

An additional compatibility fix was required to prevent a crash on sysctl access. The crash only happened with rpcrdma loaded under kernels older than Linux v5.7 which don't contain a backport of upstream commit 32927393dc1.