# Job-ID Tagging

VastNFS supports tagging NFS I/O and metadata operations with a job identifier
read from the calling process's environment.  This is designed for workload
managers such as **SLURM** that assign a unique job identifier to every
running task.  When the feature is enabled, the client appends the job-ID
string to the `auth_unix` machine name field on every RPC it sends, allowing
the VAST server to associate each I/O or metadata request with the
originating job and apply per-job quality-of-service or accounting policies.
The feature adds no overhead when it is disabled (no environment variable
names are configured).

## Module parameters

The feature is controlled through two module parameters on the **sunrpc**
kernel module.

### `job_id_var_names`

A comma-separated list of environment variable names the client should look
for in the calling process's environment.  When a matching variable is found
its value is captured and appended to the machine name in the format
`<hostname>#VastMD=<value>#`, where the `#VastMD=` prefix and the trailing
`#` separator are hardcoded.  Only one variable is evaluated at a time (the first
match wins).  Environment variable values that contain the `#` character are
skipped to avoid ambiguity.

```bash
# Enable job-ID tagging using the SLURM job ID variable
echo "SLURM_JOB_ID" | sudo tee /sys/module/sunrpc/parameters/job_id_var_names
```

Setting the parameter to an empty string disables the feature immediately and
subsequent RPCs carry only the plain machine name.

```bash
# Disable job-ID tagging
echo "" | sudo tee /sys/module/sunrpc/parameters/job_id_var_names
```

### `jobid_cache_max_entries`

The client maintains a kernel-side cache that maps each process (identified by
its global TGID and start time) to its captured job-ID string, avoiding a
repeated full environment scan on every RPC.  This parameter controls the
maximum number of entries the cache may hold.  The default is **1024**.
Writing a new value immediately resizes the limit and **flushes the entire
cache**, so all subsequent calls will perform a fresh environment scan.

```bash
# Change the cache limit to 4096 entries
echo 4096 | sudo tee /sys/module/sunrpc/parameters/jobid_cache_max_entries
```

## DebugFS interface

The following entries are available under
`/sys/kernel/debug/sunrpc/` when the sunrpc module is loaded.

| Entry | Access | Description |
|---|---|---|
| `jobid_cache_stats` | read | Cache hit/miss counters, hit-ratio percentage, and current/maximum entry count. |
| `jobid_cache_dump` | read | Per-entry table showing global TGID, process start time, last access time, age in milliseconds, and the captured job-ID string. |
| `jobid_cache_flush` | write | Write any value to immediately evict all cache entries.  This forces the next RPC from every process to re-scan its environment. |

```bash
# Show cache statistics
cat /sys/kernel/debug/sunrpc/jobid_cache_stats

# Dump all cached entries
cat /sys/kernel/debug/sunrpc/jobid_cache_dump

# Flush the cache manually
echo 1 | sudo tee /sys/kernel/debug/sunrpc/jobid_cache_flush
```

## Example: SLURM setup

The steps below show a typical end-to-end setup where SLURM tasks use
job-ID tagging on a VastNFS mount.

1. **Load the module and configure the variable name** (once, at system
   startup or via a configuration-management tool):

   ```bash
   # /etc/modprobe.d/vastnfs-jobid.conf
   options sunrpc job_id_var_names=SLURM_JOB_ID
   ```

   If the module is already loaded, the parameter can be set at runtime:

   ```bash
   echo "SLURM_JOB_ID" | sudo tee /sys/module/sunrpc/parameters/job_id_var_names
   ```

2. **Mount the share** as usual (no additional mount options are required):

   ```bash
   mount -t nfs -o vers=3 192.0.2.10:/vastexport /mnt/vast
   ```

3. **Run a SLURM job** that writes to the mount.  When `sbatch` accepts a
   job, the scheduler assigns it a numeric ID (e.g. `471823`) and injects
   `SLURM_JOB_ID=471823` into the environment of every task that runs under
   that job.  No changes to job scripts are needed:

   ```bash
   sbatch --wrap="dd if=/dev/zero of=/mnt/vast/testfile bs=1M count=100"
   ```

4. **Verify tagging** by checking the cache or the tracing output:

   ```bash
   cat /sys/kernel/debug/sunrpc/jobid_cache_dump
   ```

   Each row in the dump whose `Env Data` column contains
   `#VastMD=<SLURM_JOB_ID_VALUE>#` confirms that the tagging is active for
   that process.

## Example: manual environment variable

This example shows how to tag I/O from an arbitrary process without a
workload manager, which is useful for testing or for custom scripting.

1. **Configure the variable name** to look for:

   ```bash
   echo "MY_JOB_ID" | sudo tee /sys/module/sunrpc/parameters/job_id_var_names
   ```

2. **Launch the process with the variable set** in its environment.
   Using `env` ensures the variable is present only for that process and
   does not pollute the shell session:

   ```bash
   env MY_JOB_ID=test-id-1 dd if=/dev/zero of=/mnt/vast/testfile bs=1M count=100
   ```

3. **Confirm the tag was captured**:

   ```bash
   cat /sys/kernel/debug/sunrpc/jobid_cache_dump
   ```

   The `Env Data` column for the `dd` process should show
   `#VastMD=batch-42#`, confirming that all RPCs issued by that process
   carried the tag while it was running.
