IBM Spectrum LSF

To have LSF ask LicOps for a seat before a licensed job is queued, install the OpenLM LSF adapters. Users add an openlm[...] block to the job's resource request, and LSF does the rest: a job that LicOps denies is never queued.

Test on your cluster first

LSF support is in early access. Validate the adapters on a non-production queue, with the checks in Verify, before you rely on them.

How the adapters fit into LSF

The following table lists the three requests, the adapter that makes each, where it runs, and when:

Request	Adapter	Runs on	When
Admit	esub.openlm, an LSF esub	Each submit host	At submission, before the job reaches mbatchd
Dispatch	ol-pre-exec, as the queue's PRE_EXEC	The job's first execution host	When the job starts
Release	ol-post-exec, as the queue's POST_EXEC	The job's first execution host	When the job ends, however it ends

esub.openlm stores the session ID in the job's environment as OL_SESSION_ID, and the queue hooks read it back. Some LSF configurations strip that variable before POST_EXEC, so ol-pre-exec also writes the session ID to a small file on the execution host, and ol-post-exec falls back to it.

Unlike Slurm, LSF needs no license map: the vendor and feature travel in the job's own resource request.

Before you begin

Confirm these prerequisites:

• LicOps is running and sees your pools, as described in Configure OpenLM LicOps.
• The LSF management host, the execution hosts, and every submit host can reach LicOps on port 8100.
• curl and jq are installed on the management host, the execution hosts, and every submit host.
• You can edit lsb.queues and run badmin reconfig on the management host, and you can write to $LSF_SERVERDIR on each submit host.
• You have the OpenLM HPC adapter package for LSF and the integration kit, from OpenLM.

Install the adapters

To install the adapters:

1. Unpack the adapter package on the LSF management host.

2. Check LicOps from the management host with verify-router.sh from the integration kit:

   ./lsf/verify-router.sh --base-url http://<licops-host>:8100/api/v1/admission-controller

3. On the management host, wire the dispatch and release hooks into a queue:

   sudo ./lsf/install-lsf-adapters.sh --role master --queue normal \
     --base-url http://<licops-host>:8100/api/v1/admission-controller

   The installer copies the adapters to /opt/openlm/hpc, creates /etc/openlm/hpc-adapter.conf, backs up lsb.queues, adds PRE_EXEC and POST_EXEC to the queue you name, and runs badmin reconfig. It refuses to overwrite a PRE_EXEC or POST_EXEC that already points at another script. It wires one queue per run: run it again with another --queue for every queue that runs licensed jobs.

4. On every submit host, install the esub:

   sudo ./lsf/install-lsf-adapters.sh --role submit \
     --base-url http://<licops-host>:8100/api/v1/admission-controller

   To install the submit hosts from the management host in one step, add --submit-nodes host1,host2 to the --role master command. That needs passwordless root SSH to those hosts.

5. On every host, open /etc/openlm/hpc-adapter.conf and set these values. The installer creates the file from the template the first time, with OL_BASE_URL filled in, and never overwrites it afterwards:

   Setting	Value
   OL_FAIL_MODE_ADMIT, OL_FAIL_MODE_DISPATCH	open or closed, as described in Decide what happens when LicOps is unreachable.
   OL_CLUSTER_NAME	Optional. The only way LicOps learns the name of an LSF cluster.
   OL_STATE_DIR	Optional. A directory that PRE_EXEC and POST_EXEC both reach on the execution host. See When something goes wrong.

Run either command with --dry-run first to see every change it would make. Both are safe to run again.

Install both roles

Without the esub on the submit hosts, jobs are queued without asking LicOps. Without the queue hooks, jobs are admitted but never dispatched or released, so their seats wait for the session timeout.

Submit licensed jobs

Users select the esub with -a openlm and name the pool in an openlm[...] block:

bsub -a openlm -R "openlm[vendor=MLM:feature=MATLAB:count=1]" ./my-simulation.sh

The vendor and feature must match a pool exactly, including case. Copy them from GET /api/v1/stats. Write a space in a feature name as %20, such as feature=Acrobat%20Pro. count is the number of seats and defaults to 1. The same block works as a #BSUB -R line at the top of the job script.

To make admission non-optional, configure openlm as a mandatory esub in lsf.conf instead of relying on users to pass -a openlm. A job without an openlm[...] block passes straight through.

Each job requests one vendor and feature. A job that needs two different products cannot ask for both in one submission.

Route checkouts through LicOps

By default, the adapters gate when a job starts, and the application checks out from the license servers in its own settings. To make the application check out through LicOps, so that its checkout goes to the server LicOps chose, add --lb-endpoint 30001@<licops-host> to both install commands. If /etc/openlm/hpc-adapter.conf already exists, the installer leaves it alone, so add OL_LB_ENDPOINT="30001@<licops-host>" to it yourself on each submit host.

The esub then puts 30001@<licops-host> in front of the job's LM_LICENSE_FILE, keeping the existing value behind it as a fallback. See Decide whether LicOps also chooses the server.

Many applications read their own variable before LM_LICENSE_FILE. MATLAB reads MLM_LICENSE_FILE, for example. Where such a variable is set, it wins, and the application checks out without going through LicOps.

Verify

To test the adapters, run these commands from a submit host:

bsub -a openlm -R "openlm[vendor=MLM:feature=MATLAB:count=1]" sleep 30
tail -f /var/log/openlm/hpc-adapter.log

Check that the log records the admit when you submit, the dispatch when the job starts, and the release when it ends. While the job is admitted or running, the pool's reserved count in GET /api/v1/stats is one higher.

Also check that the release happens when POST_EXEC runs, not only after the session times out: the reserved count falls as soon as the job ends.

When something goes wrong

The following table lists symptoms, their causes, and fixes:

Symptom	Cause	Fix
Nothing is admitted.	No esub.openlm in $LSF_SERVERDIR, or the job was submitted without -a openlm.	Install the submit role, or make the esub mandatory.
ol-common.sh not found	The adapter folders were moved apart, or the esub in $LSF_SERVERDIR was replaced by a symbolic link.	Keep the installed layout under /opt/openlm/hpc, and keep the one-line esub wrapper the installer wrote.
The hooks never run.	badmin reconfig was not run, or the job ran in a queue without the hooks.	Check bqueues -l <queue> for PRE_EXEC and POST_EXEC.
Seats return only after the session timeout.	POST_EXEC lost OL_SESSION_ID, and its fallback file was not readable.	Set OL_STATE_DIR to a directory that PRE_EXEC and POST_EXEC both reach on the execution host.
Every licensed job is rejected at submission.	LicOps is unreachable and OL_FAIL_MODE_ADMIT=closed, or OL_BASE_URL is wrong.	Run verify-router.sh from that submit host.
Jobs are requeued after they start.	LicOps is unreachable at dispatch and OL_FAIL_MODE_DISPATCH=closed.	Restore LicOps, or switch dispatch to open.

Remove the adapters

To preview the removal and then remove the adapters, run these commands:

sudo ./lsf/uninstall-lsf-adapters.sh --dry-run
sudo ./lsf/uninstall-lsf-adapters.sh

The uninstaller reads the record the installer kept, /etc/openlm/hpc-adapter.install, and removes only what that installation added.