--- title: Slurm description: Install the OpenLM Slurm adapters so that licensed jobs get a LicOps seat from OpenLM Platform before they enter the queue. product: OpenLM Platform --- To have Slurm ask LicOps for a seat before a licensed job enters the queue, install the OpenLM Slurm adapters. Users keep requesting licenses the way Slurm already expects, with `--licenses=matlab:1`. A job that LicOps denies is never queued, and the user is told at once which license was denied. The reason is written to `/var/log/openlm/hpc-adapter.log`. ## How the adapters fit into Slurm The following table lists the requests, the adapter that makes each, where it runs, and when: | Request | Adapter | Runs on | When | | --- | --- | --- | --- | | Admit | `ol-sbatch`, a wrapper around `sbatch` | Each submit and login node, as the user | At submission, before the job reaches Slurm | | Dispatch | `ol-prolog`, as `PrologSlurmctld` | The Slurm controller | When the job is allocated | | Release | `ol-epilog`, as `EpilogSlurmctld` | The Slurm controller | When the job ends, however it ends | | Checkout routing (optional) | `ol-taskprolog`, as `TaskProlog` | Each compute node | When the job's tasks start | The admit request runs in the user's own `sbatch` call, not inside the Slurm controller, so a slow or unreachable LicOps never stalls scheduling for other jobs. `ol-sbatch` stores the session ID in the job's comment, as `olsid=`, and the controller hooks read it back from there. A job can ask for several licenses at once, such as `--licenses=matlab:1,abaqus:2`. The adapter admits each one, and if any is denied, it releases the others and rejects the whole submission. ## Before you begin Confirm these prerequisites: - LicOps is running and sees your pools, as described in [Configure OpenLM LicOps](https://www.openlm.com/documentation/cloud/deployment-operations/licops/configuration). - The controller and every submit and login node can reach LicOps on port `8100`. - `curl` and `jq` are installed on the controller and on every submit and login node. - You can edit `slurm.conf` and run `scontrol reconfigure` as root. - You have the OpenLM HPC adapter package and the integration kit, from OpenLM. ## Install the adapters To install the adapters, check LicOps from the controller, map the license names, configure the adapters, and install them on the controller and the submit nodes. ### Check LicOps from the controller Unpack the adapter package on the controller. Then run the verifier, `verify-router.sh` from the integration kit, against a pool from `GET /api/v1/stats`. It sends one admit, dispatch, and release, and changes nothing on Slurm. Run this command: ```bash ./slurm/verify-router.sh --rest-only \ --base-url http://:8100/api/v1/admission-controller \ --vendor MLM --feature MATLAB ``` Fix anything it reports before you continue. Fixing the address or the names is easier now than after jobs depend on them. ### Map Slurm license names to LicOps pools Slurm names licenses with short names such as `matlab`. LicOps names them `VENDOR:FEATURE`, such as `MLM:MATLAB`. The adapters translate with a license map, `OL_LICENSE_MAP`. Do not enter it by hand: generate it from the pools LicOps has, with `make-license-map.sh` from the integration kit: ```bash ./make-license-map.sh --router http://:8100 ``` ```text # 1) /etc/openlm/hpc-adapter.conf -- single-quoted, so $ and %20 stay literal: OL_LICENSE_MAP='MATLAB=MLM:MATLAB Acrobat_Pro=adskflex:Acrobat%20Pro 86508IDSS_2016_0F=ansyslmd:86508IDSS_2016_0F' # # 2) slurm.conf, then: scontrol reconfigure Licenses=MATLAB:100,Acrobat_Pro:25,86508IDSS_2016_0F:10 ``` The names before each `=` are yours to choose: they are what users enter in `--licenses`. Rename them to what your users already know, such as `matlab`, and use the same names in `Licenses=`. Never edit the `VENDOR:FEATURE` values on the right. A pool served by several license servers appears once, with its seats added up. A Slurm name with no entry in the map is sent as both vendor and feature, so `matlab` becomes `matlab:matlab`, which matches no pool. Every name your users submit needs an entry. ### Set up the configuration file To set up the configuration file, copy the template from the adapter package and edit the copy: ```bash cp config/hpc-adapter.conf.example hpc-adapter.conf ``` Set at least these values: | Setting | Value | | --- | --- | | `OL_BASE_URL` | `http://:8100/api/v1/admission-controller` | | `OL_LICENSE_MAP` | The line the generator printed, in single quotes. | | `OL_FAIL_MODE_ADMIT`, `OL_FAIL_MODE_DISPATCH` | `open` or `closed`, as described in [Decide what happens when LicOps is unreachable](https://www.openlm.com/documentation/cloud/deployment-operations/licops/integrations/index#decide-what-happens-when-licops-is-unreachable). | | `OL_TIMEOUT` | Seconds before the adapter gives up on LicOps. Keep the default, `2`, unless you use Throttle rules with a longer hold. | | `OL_CLUSTER_NAME` | Optional. A name for this cluster in the LicOps records. | The same file drives the installer and becomes the adapters' runtime configuration, `/etc/openlm/hpc-adapter.conf`. ### Install on the controller and the submit nodes To install on the controller, run this command as root: ```bash sudo ./slurm/install-slurm-adapters.sh --settings hpc-adapter.conf --role controller ``` The installer copies the adapters to `/opt/openlm/hpc`, backs up `slurm.conf`, adds `PrologSlurmctld` and `EpilogSlurmctld`, and runs `scontrol reconfigure`. It refuses to overwrite a prolog or epilog that already points at another script. In that case the installer reports it. Call the OpenLM hook from your existing one. Run it with `--dry-run` first to see every change it would make. It is safe to run again. To install on every submit and login node, run this command: ```bash sudo ./slurm/install-slurm-adapters.sh --settings hpc-adapter.conf --role submit ``` This installs `ol-sbatch` in `/usr/local/bin`. To install the submit nodes from the controller in one step, add `--submit-nodes login1,login2` to the controller command. That needs passwordless root SSH to those nodes. :::warning[Install both roles] The controller role alone leaves the submit nodes without `ol-sbatch`, so jobs are submitted without asking LicOps. The submit role alone admits jobs that nothing ever dispatches or releases, so their seats wait for the session timeout. ::: ### Declare the license names to Slurm `--licenses` is a native Slurm option, and Slurm rejects a license name it does not know. Add the generated `Licenses=` line to `slurm.conf`, merged with any licenses you already declare, and run `sudo scontrol reconfigure`. Alternatively, run the controller install with `--register-licenses`, which adds every name in the map with a count of `--license-count`, 100 by default. Set `--license-count` at least as high as your largest pool's total. Slurm also counts these licenses itself and holds jobs in the queue past its count. Set each count no lower than the pool's real total, which is what the generator writes, and let LicOps enforce the real limit. ### Keep the license map current A generated map is a snapshot. A license added to OpenLM Platform later is missing from it, and jobs that ask for it are denied as `NoServerAvailable`. To keep the map current, copy `refresh-license-map.sh` and `make-license-map.sh` from the integration kit to the same folder on each submit node, such as `/opt/openlm/`, and run the refresh on a timer: ```bash # /etc/cron.d/openlm-license-map 0 * * * * root /opt/openlm/refresh-license-map.sh --router http://:8100 --quiet ``` Then add this line at the end of `/etc/openlm/hpc-adapter.conf`, so the refreshed map wins over the earlier one: ```bash [ -r /etc/openlm/license-map.generated ] && . /etc/openlm/license-map.generated ``` The refresh replaces the file in one step, keeps the previous map if LicOps cannot be reached, and never installs an empty map. To give a pool a name the generator would not choose, add a `name=VENDOR:FEATURE` line to `/etc/openlm/license-map.overrides`. The refresh keeps it. The refresh keeps the adapters' map current, not `slurm.conf`. Before users can request a new license, also add its name to `Licenses=` and run `scontrol reconfigure`. ## Submit licensed jobs Users submit with `ol-sbatch` and the options they already use: ```bash ol-sbatch --licenses=matlab:1 my-simulation.sh ``` A `#SBATCH --licenses=matlab:1` line in the job script works too. A denied job is not queued, and the user sees `OpenLM: licence admission denied for MLM:MATLAB.` A job without licenses passes straight through. To let users keep typing `sbatch`, alias it to the wrapper on the submit nodes, for example in `/etc/profile.d/openlm.sh`: ```sh alias sbatch='/usr/local/bin/ol-sbatch' ``` Keep the real `sbatch` on the `PATH` as well: the wrapper calls it. A user who runs the real `sbatch` directly skips LicOps. To stop that, add a check in `job_submit.lua` that rejects a job requesting licenses whose comment has no `olsid=`. Keep that check to the comment only: `job_submit.lua` runs inside the controller's scheduling loop, so it must never call LicOps itself. ## Route checkouts through LicOps By default, the adapters gate when a job starts, and the application checks out from the license servers in its own settings. To make the application check out through LicOps, so that its checkout goes to the server LicOps chose, add `--lb-endpoint` to the controller install: ```bash sudo ./slurm/install-slurm-adapters.sh --settings hpc-adapter.conf --role controller \ --lb-endpoint 30001@ ``` The installer adds `ol-taskprolog` as the Slurm `TaskProlog` and sets `OL_LB_ENDPOINT` in a configuration file it creates. If `/etc/openlm/hpc-adapter.conf` already exists, the installer leaves it alone, so add `OL_LB_ENDPOINT="30001@"` to it yourself. Slurm runs the `TaskProlog` on the compute nodes for every task, including jobs that were not submitted through `ol-sbatch`, so the change is node-wide. The installer does not touch the compute nodes. Copy `/opt/openlm/hpc/slurm/bin/ol-taskprolog` and `/etc/openlm/hpc-adapter.conf` to every compute node, at the same paths, or keep them on a shared file system. On each node, the script puts `30001@` in front of the job's `LM_LICENSE_FILE`, keeping the existing value behind it as a fallback. See [Decide whether LicOps also chooses the server](https://www.openlm.com/documentation/cloud/deployment-operations/licops/integrations/index#decide-whether-licops-also-chooses-the-server). Many applications read their own variable before `LM_LICENSE_FILE`. MATLAB reads `MLM_LICENSE_FILE`, for example. Where such a variable is set, it wins, and the application checks out without going through LicOps. ## Verify To test the adapters, run these commands from a submit node: ```bash ol-sbatch --licenses=matlab:1 --wrap 'sleep 30' scontrol show job | grep -i comment ``` The comment reads `olsid=;olvendor=MLM;olfeature=MATLAB`. While the job runs, the pool's `reserved` count in `GET /api/v1/stats` rises by one until the job checks out or ends. After the job ends, it is back where it started. The adapters log every request to `/var/log/openlm/hpc-adapter.log`: the admit on the submit node, and the dispatch and release on the controller. Look in both places when you trace a job. ## When something goes wrong The following table lists symptoms, their causes, and fixes: | Symptom | Cause | Fix | | --- | --- | --- | | Every licensed job is rejected at submission. | LicOps is unreachable and `OL_FAIL_MODE_ADMIT=closed`, or `OL_BASE_URL` is wrong. | Run `verify-router.sh` from that submit node. | | `OpenLM: admission controller warming up; please resubmit shortly.` | LicOps restarted a few seconds ago and has not caught up with OpenLM Platform. | Resubmit. | | One license name is always denied as `NoServerAvailable`. Others work. | The name is not in the license map, or the map is older than the license. | Regenerate the map, and schedule the refresh. | | The hooks never run. | `slurm.conf` was not reloaded, or the node-side `Prolog` and `Epilog` were used instead of `PrologSlurmctld` and `EpilogSlurmctld`. | Check `scontrol show config \| grep -i slurmctld`. | | Jobs have no `olsid` in their comment. | They were submitted with the real `sbatch`. | Use `ol-sbatch`, or the alias. | | Admit lines appear in the log, but no dispatch or release lines. | The log directory is not writable by `SlurmUser`, which the controller hooks run as. | `chown` `/var/log/openlm` to `SlurmUser`. The hooks still run. Only their logging was lost. | | Jobs are held after submission. | LicOps is unreachable at dispatch and `OL_FAIL_MODE_DISPATCH=closed`. | Restore LicOps, or switch dispatch to `open`. | The adapter stores the session ID in the job's comment. A job that sets its own `--comment` conflicts with it, so keep comments out of licensed jobs. ## Remove the adapters To preview the removal and then remove the adapters, run these commands: ```bash sudo ./slurm/uninstall-slurm-adapters.sh --dry-run sudo ./slurm/uninstall-slurm-adapters.sh ``` The uninstaller reads the record the installer kept, `/etc/openlm/hpc-adapter.install`, and reverses exactly that installation: it removes the wrapper and only the hook lines it added, then runs `scontrol reconfigure`. Add `--purge` to remove the configuration file and logs as well.