Slurm

To have Slurm ask LicOps for a seat before a licensed job enters the queue, install the OpenLM Slurm adapters. Users keep requesting licenses the way Slurm already expects, with --licenses=matlab:1. A job that LicOps denies is never queued, and the user is told at once which license was denied. The reason is written to /var/log/openlm/hpc-adapter.log.

How the adapters fit into Slurm

The following table lists the requests, the adapter that makes each, where it runs, and when:

Request	Adapter	Runs on	When
Admit	ol-sbatch, a wrapper around sbatch	Each submit and login node, as the user	At submission, before the job reaches Slurm
Dispatch	ol-prolog, as PrologSlurmctld	The Slurm controller	When the job is allocated
Release	ol-epilog, as EpilogSlurmctld	The Slurm controller	When the job ends, however it ends
Checkout routing (optional)	ol-taskprolog, as TaskProlog	Each compute node	When the job's tasks start

The admit request runs in the user's own sbatch call, not inside the Slurm controller, so a slow or unreachable LicOps never stalls scheduling for other jobs. ol-sbatch stores the session ID in the job's comment, as olsid=<session-id>, and the controller hooks read it back from there.

A job can ask for several licenses at once, such as --licenses=matlab:1,abaqus:2. The adapter admits each one, and if any is denied, it releases the others and rejects the whole submission.

Before you begin

Confirm these prerequisites:

• LicOps is running and sees your pools, as described in Configure OpenLM LicOps.
• The controller and every submit and login node can reach LicOps on port 8100.
• curl and jq are installed on the controller and on every submit and login node.
• You can edit slurm.conf and run scontrol reconfigure as root.
• You have the OpenLM HPC adapter package and the integration kit, from OpenLM.

Install the adapters

To install the adapters, check LicOps from the controller, map the license names, configure the adapters, and install them on the controller and the submit nodes.

Check LicOps from the controller

Unpack the adapter package on the controller. Then run the verifier, verify-router.sh from the integration kit, against a pool from GET /api/v1/stats. It sends one admit, dispatch, and release, and changes nothing on Slurm. Run this command:

./slurm/verify-router.sh --rest-only \
  --base-url http://<licops-host>:8100/api/v1/admission-controller \
  --vendor MLM --feature MATLAB

Fix anything it reports before you continue. Fixing the address or the names is easier now than after jobs depend on them.

Map Slurm license names to LicOps pools

Slurm names licenses with short names such as matlab. LicOps names them VENDOR:FEATURE, such as MLM:MATLAB. The adapters translate with a license map, OL_LICENSE_MAP. Do not enter it by hand: generate it from the pools LicOps has, with make-license-map.sh from the integration kit:

./make-license-map.sh --router http://<licops-host>:8100

# 1) /etc/openlm/hpc-adapter.conf  -- single-quoted, so $ and %20 stay literal:
OL_LICENSE_MAP='MATLAB=MLM:MATLAB Acrobat_Pro=adskflex:Acrobat%20Pro 86508IDSS_2016_0F=ansyslmd:86508IDSS_2016_0F'
#
# 2) slurm.conf, then: scontrol reconfigure
Licenses=MATLAB:100,Acrobat_Pro:25,86508IDSS_2016_0F:10

The names before each = are yours to choose: they are what users enter in --licenses. Rename them to what your users already know, such as matlab, and use the same names in Licenses=. Never edit the VENDOR:FEATURE values on the right. A pool served by several license servers appears once, with its seats added up.

A Slurm name with no entry in the map is sent as both vendor and feature, so matlab becomes matlab:matlab, which matches no pool. Every name your users submit needs an entry.

Set up the configuration file

To set up the configuration file, copy the template from the adapter package and edit the copy:

cp config/hpc-adapter.conf.example hpc-adapter.conf

Set at least these values:

Setting	Value
OL_BASE_URL	http://<licops-host>:8100/api/v1/admission-controller
OL_LICENSE_MAP	The line the generator printed, in single quotes.
OL_FAIL_MODE_ADMIT, OL_FAIL_MODE_DISPATCH	open or closed, as described in Decide what happens when LicOps is unreachable.
OL_TIMEOUT	Seconds before the adapter gives up on LicOps. Keep the default, 2, unless you use Throttle rules with a longer hold.
OL_CLUSTER_NAME	Optional. A name for this cluster in the LicOps records.

The same file drives the installer and becomes the adapters' runtime configuration, /etc/openlm/hpc-adapter.conf.

Install on the controller and the submit nodes

To install on the controller, run this command as root:

sudo ./slurm/install-slurm-adapters.sh --settings hpc-adapter.conf --role controller

The installer copies the adapters to /opt/openlm/hpc, backs up slurm.conf, adds PrologSlurmctld and EpilogSlurmctld, and runs scontrol reconfigure. It refuses to overwrite a prolog or epilog that already points at another script. In that case the installer reports it. Call the OpenLM hook from your existing one. Run it with --dry-run first to see every change it would make. It is safe to run again.

To install on every submit and login node, run this command:

sudo ./slurm/install-slurm-adapters.sh --settings hpc-adapter.conf --role submit

This installs ol-sbatch in /usr/local/bin. To install the submit nodes from the controller in one step, add --submit-nodes login1,login2 to the controller command. That needs passwordless root SSH to those nodes.

Install both roles

The controller role alone leaves the submit nodes without ol-sbatch, so jobs are submitted without asking LicOps. The submit role alone admits jobs that nothing ever dispatches or releases, so their seats wait for the session timeout.

Declare the license names to Slurm

--licenses is a native Slurm option, and Slurm rejects a license name it does not know. Add the generated Licenses= line to slurm.conf, merged with any licenses you already declare, and run sudo scontrol reconfigure. Alternatively, run the controller install with --register-licenses, which adds every name in the map with a count of --license-count, 100 by default. Set --license-count at least as high as your largest pool's total.

Slurm also counts these licenses itself and holds jobs in the queue past its count. Set each count no lower than the pool's real total, which is what the generator writes, and let LicOps enforce the real limit.

Keep the license map current

A generated map is a snapshot. A license added to OpenLM Platform later is missing from it, and jobs that ask for it are denied as NoServerAvailable. To keep the map current, copy refresh-license-map.sh and make-license-map.sh from the integration kit to the same folder on each submit node, such as /opt/openlm/, and run the refresh on a timer:

# /etc/cron.d/openlm-license-map
0 * * * * root /opt/openlm/refresh-license-map.sh --router http://<licops-host>:8100 --quiet

Then add this line at the end of /etc/openlm/hpc-adapter.conf, so the refreshed map wins over the earlier one:

[ -r /etc/openlm/license-map.generated ] && . /etc/openlm/license-map.generated

The refresh replaces the file in one step, keeps the previous map if LicOps cannot be reached, and never installs an empty map. To give a pool a name the generator would not choose, add a name=VENDOR:FEATURE line to /etc/openlm/license-map.overrides. The refresh keeps it.

The refresh keeps the adapters' map current, not slurm.conf. Before users can request a new license, also add its name to Licenses= and run scontrol reconfigure.

Submit licensed jobs

Users submit with ol-sbatch and the options they already use:

ol-sbatch --licenses=matlab:1 my-simulation.sh

A #SBATCH --licenses=matlab:1 line in the job script works too. A denied job is not queued, and the user sees OpenLM: licence admission denied for MLM:MATLAB. A job without licenses passes straight through.

To let users keep typing sbatch, alias it to the wrapper on the submit nodes, for example in /etc/profile.d/openlm.sh:

alias sbatch='/usr/local/bin/ol-sbatch'

Keep the real sbatch on the PATH as well: the wrapper calls it.

A user who runs the real sbatch directly skips LicOps. To stop that, add a check in job_submit.lua that rejects a job requesting licenses whose comment has no olsid=. Keep that check to the comment only: job_submit.lua runs inside the controller's scheduling loop, so it must never call LicOps itself.

Route checkouts through LicOps

By default, the adapters gate when a job starts, and the application checks out from the license servers in its own settings. To make the application check out through LicOps, so that its checkout goes to the server LicOps chose, add --lb-endpoint to the controller install:

sudo ./slurm/install-slurm-adapters.sh --settings hpc-adapter.conf --role controller \
  --lb-endpoint 30001@<licops-host>

The installer adds ol-taskprolog as the Slurm TaskProlog and sets OL_LB_ENDPOINT in a configuration file it creates. If /etc/openlm/hpc-adapter.conf already exists, the installer leaves it alone, so add OL_LB_ENDPOINT="30001@<licops-host>" to it yourself.

Slurm runs the TaskProlog on the compute nodes for every task, including jobs that were not submitted through ol-sbatch, so the change is node-wide. The installer does not touch the compute nodes. Copy /opt/openlm/hpc/slurm/bin/ol-taskprolog and /etc/openlm/hpc-adapter.conf to every compute node, at the same paths, or keep them on a shared file system. On each node, the script puts 30001@<licops-host> in front of the job's LM_LICENSE_FILE, keeping the existing value behind it as a fallback. See Decide whether LicOps also chooses the server.

Many applications read their own variable before LM_LICENSE_FILE. MATLAB reads MLM_LICENSE_FILE, for example. Where such a variable is set, it wins, and the application checks out without going through LicOps.

Verify

To test the adapters, run these commands from a submit node:

ol-sbatch --licenses=matlab:1 --wrap 'sleep 30'
scontrol show job <job-id> | grep -i comment

The comment reads olsid=<session-id>;olvendor=MLM;olfeature=MATLAB. While the job runs, the pool's reserved count in GET /api/v1/stats rises by one until the job checks out or ends. After the job ends, it is back where it started.

The adapters log every request to /var/log/openlm/hpc-adapter.log: the admit on the submit node, and the dispatch and release on the controller. Look in both places when you trace a job.

When something goes wrong

The following table lists symptoms, their causes, and fixes:

Symptom	Cause	Fix
Every licensed job is rejected at submission.	LicOps is unreachable and OL_FAIL_MODE_ADMIT=closed, or OL_BASE_URL is wrong.	Run verify-router.sh from that submit node.
OpenLM: admission controller warming up; please resubmit shortly.	LicOps restarted a few seconds ago and has not caught up with OpenLM Platform.	Resubmit.
One license name is always denied as NoServerAvailable. Others work.	The name is not in the license map, or the map is older than the license.	Regenerate the map, and schedule the refresh.
The hooks never run.	slurm.conf was not reloaded, or the node-side Prolog and Epilog were used instead of PrologSlurmctld and EpilogSlurmctld.	Check scontrol show config | grep -i slurmctld.
Jobs have no olsid in their comment.	They were submitted with the real sbatch.	Use ol-sbatch, or the alias.
Admit lines appear in the log, but no dispatch or release lines.	The log directory is not writable by SlurmUser, which the controller hooks run as.	chown /var/log/openlm to SlurmUser. The hooks still run. Only their logging was lost.
Jobs are held after submission.	LicOps is unreachable at dispatch and OL_FAIL_MODE_DISPATCH=closed.	Restore LicOps, or switch dispatch to open.

The adapter stores the session ID in the job's comment. A job that sets its own --comment conflicts with it, so keep comments out of licensed jobs.

Remove the adapters

To preview the removal and then remove the adapters, run these commands:

sudo ./slurm/uninstall-slurm-adapters.sh --dry-run
sudo ./slurm/uninstall-slurm-adapters.sh

The uninstaller reads the record the installer kept, /etc/openlm/hpc-adapter.install, and reverses exactly that installation: it removes the wrapper and only the hook lines it added, then runs scontrol reconfigure. Add --purge to remove the configuration file and logs as well.