Operate OpenLM LicOps

To check what LicOps holds, find out why a job was denied, return a seat that was never released, or take a server out of placement, use the tasks on this page. Day to day, LicOps needs no attention: it keeps its seat counts in step with OpenLM Platform and returns seats when jobs end.

Run the commands from the LicOps host, or from a machine on your administration network. On Kubernetes, start a kubectl port-forward to the openlm-router-api Service first.

See what LicOps is holding

To see every pool and server with its live seat counts:

curl -s http://localhost:8100/api/v1/stats \
  | jq -r '.[] | [.vendor, .feature, .host, .total, .used, .reserved, .free, .isHealthy] | @tsv'

MLM       MATLAB  lic01.example.com  50  31  2  17  true
MLM       MATLAB  lic02.example.com  50  28  0  22  true
adskflex  86000ACD_2024_0F  lic03.example.com  25  25  0  0  true

The reserved column is what LicOps adds to the license servers' own figures: seats it has admitted that are not checked out yet. It rises when jobs are admitted and falls as they check out or end. A reserved count that only ever rises means jobs are not releasing their seats. See Return a seat that was never released.

used comes from OpenLM Platform, so it is as fresh as the last reading of that license server by OpenLM Platform.

Find out why a request was denied

A denied admit carries a reason. Start there. The following table lists the reasons and where to look for each:

Reason	Where to look
NoServerAvailable	The vendor or feature name, first. A name that matches no pool reads exactly like an outage. Compare it with /api/v1/stats. Then check whether every server in the pool is unhealthy.
NoCapacity	A real shortage: no eligible server has enough free seats. If seats look free, a rule such as Keep off a server might be narrowing the choice.
ConstraintBlocked	A reservation: the servers with free seats have reached their Maximum seats, or are blocked during the current period.
PolicyDenied	An admission rule with Deny the request matched.
StaleSnapshot	LicOps has stopped receiving fresh figures from OpenLM Platform. Every request is denied until the sync recovers.

For the full picture behind a denial, read the availability report. It shows every pool and every server, including those that are down, with the exact figures admission uses. Run this command:

curl -s http://localhost:8100/api/v1/admission-controller/availability | jq .

Read the report in this order:

1. coldStart and staleForAdmission: If either is true, the figures of LicOps are not reliable, and nothing else in the report matters until it recovers. See Troubleshooting.
2. The pool: Find the vendor and feature the job asked for. If the pool is missing, the job's names do not match any pool.
3. largestAdmittableRequest: For the pool, it is the largest request LicOps could admit right now. A job asking for more seats than this is denied even if the pool's total free looks large enough, because one job's seats must come from one server.
4. Each server: isHealthy: false takes a server out. A free below zero means LicOps admitted more than the server had, usually seats that were never released. A constraint whose headroom is 0 is a reservation cap that is full.

Return a seat that was never released

When a job ends without its release, such as after a node crash or an epilog that did not run, LicOps keeps counting its seat until the session times out: 15 minutes for a job that never started, and 2 hours for one that did, unless you changed the timeouts. To return the seat sooner, release the session yourself:

curl -X POST http://localhost:8100/api/v1/admission-controller/release \
  -H 'Content-Type: application/json' \
  -d '{"sessionId": "<session-id>"}'

Release always succeeds. "state": "Released" means it returned a seat. AlreadyClosed or NotFound mean LicOps had no seat to return.

To find a job's session ID:

Scheduler	Where the session ID is
Slurm	The job's comment: scontrol show job <job-id> | grep -i comment shows olsid=<session-id>. A job with several licenses carries one session ID per license, separated by |. Release each one.
LSF	The job's environment, as OL_SESSION_ID.
Both	The adapter log, /var/log/openlm/hpc-adapter.log, which records every admit with its session ID. Slurm writes the admit on the submit host and the dispatch and release on the controller.

If seats leak regularly, fix the release path rather than releasing by hand. See Connect workloads.

Take a license server out of placement

When a license server goes down, OpenLM Platform reports it unhealthy. Within about 10 seconds of that report, LicOps stops placing seats on the server. You do not need to do anything. Jobs admitted to it that have not started yet are moved to a healthy server in the same pool when they start.

To take a healthy server out on purpose, before maintenance or to drain it, add a rule with Keep off a server, as described in Take a server out of placement. Remove the rule when the work is done.

Check the OpenLM Platform sync

LicOps depends on fresh figures from OpenLM Platform. To check that they keep arriving, run this command:

curl -s http://localhost:8100/api/v1/admission-controller/availability \
  | jq '{lastPlatformSyncTime, lastReconcileTime, coldStart, staleForAdmission}'

Both times are normally a few seconds old. If they stop advancing, LicOps starts denying every request with StaleSnapshot after about 30 seconds. The configuration sync is reported separately, in configurationManagement.lastSync of GET /api/v1/status, as for a failover Router.

Restart or upgrade LicOps

Open sessions live in Redis and the configuration lives in OpenLM Platform, so a restart loses neither. While LicOps is down, the scheduler adapters apply the fail mode you chose. For the first few seconds after it starts, admit answers 503 while LicOps catches up with OpenLM Platform. To upgrade, see Deploy OpenLM LicOps.