Troubleshooting

Work from the symptom you can see. Most LicOps problems are one of four things: LicOps is not syncing with OpenLM Platform, the Router is not in LicOps mode, a request names a pool that does not exist, or jobs are not going through LicOps.

Why does the container not start?

LicOps runs on the OpenLM Router, so it fails at startup for the same reasons a failover Router does: a misnamed connection-string variable, a missing correlation setting, or an encryption key encoded twice. See The container does not start in the Router troubleshooting. The following table lists the symptoms, causes, and fixes:

Symptom	Cause	Fix
The Router starts, but every admit fails, and the logs show Redis errors on the admit.	Redis is older than 7.0.	Upgrade Redis to 7.0 or higher. See System requirements.
On Kubernetes, the Router refuses to start and its log names TrustedCidrs.	The trusted range is empty.	Set routerapi.admissionController.proxyProtocol.trustedCidrs to your pod CIDR. See Deploy on Kubernetes.
On Kubernetes, an image pull fails with a reference ending in a colon.	A chart tag was not set.	Set every tag explicitly.
On Kubernetes, the webhook's pods fail with ImagePullBackOff.	An image name or tag is wrong.	Use the charts from the OpenLM bundle, which name olm-pod-sidecar, olm-dispatch-init, and olm-iptables-init.

Why is LicOps not syncing with OpenLM Platform?

Until LicOps syncs with OpenLM Platform, it has no configuration, no rules, and no pools, and it denies or refuses every request. The Router UI shows a red notice under Platform Connection that names the reason, and GET /api/v1/status reports the same reason in configurationManagement.lastSync.lastFailure. The causes and fixes are the same as for a failover Router. See The Router is not syncing with the Platform.

LicOps also needs fresh seat counts and server health, which it collects separately from its configuration. To check them, read the availability report:

curl -s http://localhost:8100/api/v1/admission-controller/availability \
  | jq '{lastPlatformSyncTime, lastReconcileTime, coldStart, staleForAdmission}'

The following table lists the symptoms, causes, and fixes:

Symptom	Cause	Fix
coldStart stays true, and admit keeps answering 503.	LicOps has never received seat counts from OpenLM Platform.	Check that GET /api/v1/platform/servers lists your license servers. If it answers 503, follow the Router troubleshooting linked above: the usual cause is a credential whose client type is not Router.
Admit was working and now denies everything with StaleSnapshot, and lastReconcileTime has stopped advancing.	The OpenLM Platform sync stopped. If Redis was restarted empty, the Router lost its job schedule, and the sync does not restart by itself.	Restart the Router, then make Redis persistent so it cannot recur. See System requirements.
staleForAdmission flips between true and false.	OpenLM Platform answers slowly or intermittently.	Check the network path from the LicOps host to OpenLM Platform, and the log of the Router for failed sync attempts.

Why does LicOps Configuration show Port Mappings?

If LicOps Configuration shows Port Mappings instead of LicOps and Rules, it is working with a failover Router. It shows the pages for the mode of the Router it is working with: Port Mappings for a failover Router, and LicOps and Rules for a LicOps Router. The following table lists the symptoms, causes, and fixes:

Symptom	Cause	Fix
LicOps Configuration shows Port Mappings for your LicOps deployment.	The Router was started without ROUTER_CONFIG_BOOTSTRAP_MODE=loadBalancer, or with basic.	Set the variable as in Deploy OpenLM LicOps. The mode is fixed when the Router first starts. Deploy a new Router with its own empty MongoDB database, and stop the old one.
The pages switch between Port Mappings and LicOps from one reload to the next.	Two Routers report to the same OpenLM account, such as a failover Router and a LicOps Router. LicOps Configuration works with the one that reported most recently.	Run one Router per OpenLM account. Stop the Router you do not use. See Change, upgrade, or remove OpenLM Router.
The page says that no Router has reported itself yet.	LicOps has not synced with OpenLM Platform, or the page was opened before it did. The page does not refresh by itself.	Confirm the OpenLM Platform connection works, then reload the page.
Each page shows LicOps License is not available with current openLM License, and saves are refused.	Your OpenLM license does not include LicOps.	Contact OpenLM to have the license reissued. LicOps keeps running with the configuration it already has. See System requirements.

Why does LicOps have no pools?

GET /api/v1/stats answers [], and every admit is denied. The following table lists the symptoms, causes, and fixes:

Symptom	Cause	Fix
The list is empty within a minute of connecting LicOps.	LicOps has not received license-server data from OpenLM Platform.	Follow Why is LicOps not syncing with OpenLM Platform?.
A pool, or one server in a pool, is missing.	OpenLM Platform does not monitor that license server: the OpenLM Broker is not installed on it, not approved in Broker Hub, or the server is not approved under License Servers.	Follow the Broker steps in System requirements, then check that Live Servers Statistics shows the server as Up.

Why are requests denied?

Start from the deny reason, then read the availability report as described in Find out why a request was denied. The following table lists the symptoms, causes, and fixes:

Symptom	Cause	Fix
NoServerAvailable, while the pool looks full of free seats.	The vendor or feature in the request does not match the pool, almost always in case or spelling.	Compare with GET /api/v1/stats. For Slurm, regenerate the license map rather than editing it by hand.
One Slurm license name started being denied with NoServerAvailable. The others still work.	The license was added or renamed in OpenLM Platform after the license map was generated.	Regenerate the map and schedule the refresh, as described in Slurm.
NoCapacity, while free shows seats on some servers.	A rule such as Keep off a server narrowed the servers, or the request asks for more seats than any one server has free.	Check the rules, and largestAdmittableRequest for the pool.
ConstraintBlocked.	A reservation caps or blocks every server that has free seats.	See Cap seats on a schedule.
PolicyDenied.	A Deny the request rule matched.	Check the rules under LicOps Configuration.
A rule seems to have no effect.	It was saved less than a minute ago, its Target server does not match a server, or a condition compares names with a different case.	Wait a minute and check GET /api/v1/rules/router-admission. Pick targets with the picker, and copy names from GET /api/v1/stats.
A time-window rule or a reservation period starts hours early or late.	LicOps runs in UTC.	Set TZ on the routerapi container. See Make the schedule follow your time zone.

Why are seats never returned?

The pool's reserved count in GET /api/v1/stats keeps rising, and requests start being denied with NoCapacity although few jobs run. The following table lists the causes and fixes:

Cause	Fix
Jobs do not release their seats on every ending.	Check that the release runs from a hook or trap that always runs. For Slurm and LSF, confirm both adapter roles are installed and the hooks run.
LSF's POST_EXEC cannot find the session ID.	Set OL_STATE_DIR to a directory both PRE_EXEC and POST_EXEC can reach. See IBM Spectrum LSF.
Jobs wait in the queue longer than the admit timeout and lose their seats, then start and check out anyway.	Raise the admit timeout above your typical queue wait. See Set the session timeouts.

Seats that were never released return when their sessions time out. To return one at once, see Return a seat that was never released.

Why are jobs not going through LicOps?

The following table lists the symptoms, causes, and fixes:

Symptom	Cause	Fix
Licensed jobs start without any admit in the adapter log.	Jobs were submitted with the real sbatch, without -a openlm on LSF, or from a submit host without the adapter.	See Slurm or IBM Spectrum LSF.
Kubernetes pods start without the LicOps containers.	The namespace label or the openlm.io/needs-licensing annotation is missing. Either one missing makes the webhook skip the pod without any error.	See Kubernetes workloads.
No pod can be created in an opted-in namespace.	The admission webhook is unavailable, and its failure policy is Fail.	Restore the webhook, or reinstall it with webhook.failurePolicy=Ignore. See Deploy on Kubernetes.
Jobs are admitted, but checkouts do not go to the server LicOps chose.	Without checkout routing, applications check out from their own license settings. With it, a vendor-specific variable such as MLM_LICENSE_FILE can still take precedence.	See Decide whether LicOps also chooses the server.
With checkout routing, checkouts reach LicOps but are not matched to their jobs.	The compute nodes reach LicOps through address translation, so their checkouts arrive from a different IP address than the one dispatch recorded.	Let the compute nodes reach the LicOps host directly.

Collect diagnostics

To collect diagnostics, run these commands:

docker compose logs routerapi --tail 200
curl http://localhost:8100/api/v1/status
curl http://localhost:8100/api/v1/stats
curl http://localhost:8100/api/v1/admission-controller/availability
curl http://localhost:8100/api/v1/rules/router-admission

On Kubernetes, read the logs with kubectl -n <namespace> logs deployment/openlm-routerapi --tail 200, and reach the endpoints through kubectl port-forward svc/openlm-router-api 8100:8100. For the scheduler side, include /var/log/openlm/hpc-adapter.log from the submit host and from the controller or execution host.