Skip to main content

Troubleshooting

This page lists common problems with a Platform as VM deployment. If your problem isn't listed, use this general approach:

  1. Run kubectl get pods -A to find the failing pod.
  2. Run kubectl logs <pod> -n <namespace> --tail 200 to read its output.
  3. Run kubectl describe pod <pod> -n <namespace> to see the recent events at the bottom.

Installer failures​

This section covers errors and stalls while the installer runs.

"This playbook is only for RedHat-family systems"​

The installer only supports RHEL-family 9.x: RHEL 9, AlmaLinux 9, and Rocky Linux 9. Ubuntu, Debian, and SUSE are not supported. Reprovision the VM with a supported operating system. See Requirements.

This check tests only the OS family, not the version. On RHEL-family 8 or 10 the installer starts normally and can fail later in less obvious ways. Those versions are untested and unsupported.

"Minimum 32 GB RAM required" / "Minimum 4 CPU cores required"​

The VM is undersized. Resize it before continuing. No workaround exists.

The RAM check allows a small tolerance under a literal 32 GB, because hypervisors and cloud providers report slightly less than the nominal figure. A VM advertised as 32 GB passes the check. A VM advertised as 16 GB does not.

"Authentication failure" on the sudo password prompt​

The password you entered was wrong, or your user does not have sudo privileges. Verify it with this command:

sudo -v

If that fails, ask whoever provisioned the VM to grant you sudo access.

Why does the installer stop responding at "Wait for K3s to be ready"?​

K3s is downloading or its API server is slow to start. Wait 5 minutes. If K3s still isn't ready, open another SSH session and check:

sudo systemctl status k3s
sudo journalctl -u k3s --no-pager --since "5 minutes ago"

The most common cause is the VM not having outbound HTTPS access to get.k3s.io. Fix the proxy or firewall and run the installer again.

Why does the installer fail at "Helm install"?​

Look at the Helm error message in the installer output. The usual causes:

  • Timeout: the chart's pods aren't reaching Ready in time. Check kubectl get pods -n openlm-infrastructure and look at the failing pod's events.
  • ImagePullBackOff: the VM cannot reach the container image registry. Verify outbound HTTPS to public.ecr.aws and docker.io.
  • Insufficient disk space: run df -h and free up space under /var/lib/rancher/k3s.

After you fix the underlying issue, run ./entrypoint.sh again. The playbook re-checks each step and skips work already completed, so running it again is safe.

Post-install issues​

This section covers problems after the installer finishes.

Why is a pod stuck in Pending?​

Run this command:

kubectl describe pod <pod> -n openlm | tail -20

Look at the Events section. Typical causes:

  • Insufficient memory or CPU: the VM is oversubscribed. Resize the VM or scale non-critical services down.
  • PVC stuck Pending: kubectl get pvc -n openlm-infrastructure. If a PVC is Pending, the storage class can't provision. Check that local-path-provisioner is running in kube-system.
  • Disk pressure: the node is low on disk and the scheduler will not place new pods. Check df -h and see How do I free disk space on the VM?.
  • Unsatisfiable node affinity: if affinity_label_main_workload or affinity_label_reporting_workload is set in config.yaml, pods require a node label that a single-VM deployment does not have. Clear both fields and run the installer again.

Why is a pod stuck in ImagePullBackOff?​

Run this command:

kubectl describe pod <pod> -n openlm | grep -i image

Check that the image tag listed there exists in public.ecr.aws/r3q3q2f4/. If it looks right, the cause is a network problem. Run this command:

curl -I https://public.ecr.aws/r3q3q2f4/olm-server/manifests/latest 2>&1 | head -5

Any TLS or timeout error here means the VM cannot reach the registry. Fix the proxy or firewall.

If the tag looks wrong or unknown, the deployment package might be corrupted. Download it from OpenLM again.

Why is a pod stuck in CrashLoopBackOff?​

Run this command:

kubectl logs <pod> -n openlm --previous --tail 100

The --previous flag is critical: it shows the logs from the last crashed container, not the empty new one. Common patterns:

  • Database connection failure: the service cannot reach its database. Check that mariadb-0, postgres-postgresql-0, and mongodb-0 are Running, and that the passwords in passwords.yaml match what the databases were initialized with.
  • Kafka topic missing: the service expects a topic that wasn't created. Run the topic creation script again (see Why are Kafka topics missing?).
  • Schema mismatch: the database schema is out of date. The AllDbUpgradeAPI service migrates schemas on startup. Verify that it is Running and check its logs.

Why does the OpenLM Platform URL return 404 or Bad Gateway?​

Run these commands:

# Is Traefik routing the request?
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik --tail 50

# Does the ingress exist?
kubectl get ingress -n openlm | grep <path-prefix>

If the expected ingress doesn't exist, the OpenLM Platform Helm release didn't install fully. Run ./entrypoint.sh again.

Why does the browser show a certificate error?​

The browser shows one of these errors:

  • NET::ERR_CERT_AUTHORITY_INVALID: the certificate is self-signed or from a CA the browser doesn't trust. Expected for non-production setups.
  • NET::ERR_CERT_COMMON_NAME_INVALID: the certificate doesn't cover the FQDN you're using. Re-issue the certificate for the right name and run the installer again. It re-reads the certificate files and updates the Kubernetes secret.
  • "certificate has expired": re-issue the certificate and run the installer again, as for the previous error. The installer always overwrites the certificate secret with the current file contents.

To verify that the certificate and key match, run these commands:

openssl x509 -noout -modulus -in /etc/openlm/certs/tls.crt | md5sum
openssl rsa  -noout -modulus -in /etc/openlm/certs/tls.key | md5sum

The two outputs must be identical.

DNS issues​

This section covers name resolution problems inside and outside the cluster.

Why do services crash with "host not found" for the OpenLM Platform domain?​

  • Symptom: Services that call back to OpenLM Platform (for example, the identity service issuing tokens) fail with host not found.
  • Cause: The cluster's internal DNS can't resolve openlm_system_domain. This typically happens when the VM is in a network where the FQDN isn't published.

To fix it, set the CoreDNS hosts entry in config.yaml and run the installer again:

add_coredns_hosts_entry: true
coredns_hosts_entry_ip:  "<vm-ip-address>"

Replace <vm-ip-address> with the IP address of the VM.

See Air-gapped network without DNS for the full pattern.

Why can't end users reach OpenLM Platform?​

Run dig <openlm_system_domain> +short on the user's machine, not on the VM. If it doesn't resolve to the VM's IP, fix your DNS A record. If it does resolve but the connection still fails, check the external firewall. Port 443 must be open from the user's network.

Database issues​

This section covers resetting passwords and connecting to the bundled databases.

How do I reset the PostgreSQL or MariaDB password?​

The bundled database passwords are set once at install time. After that, passwords.yaml is no longer the source of truth. To reset a password, run the command for your database.

PostgreSQL​

kubectl exec -n openlm-infrastructure -it postgres-postgresql-0 -- \
  psql -U postgres -c "ALTER USER postgres WITH PASSWORD '<new-password>';"

MariaDB​

kubectl exec -n openlm-infrastructure -it mariadb-0 -- \
  mysql -uroot -p'<old-password>' -e \
  "ALTER USER 'root'@'%' IDENTIFIED BY '<new-password>'; FLUSH PRIVILEGES;"

In the commands, replace <new-password> with the new password and <old-password> with the current password.

Then update passwords.yaml and run the installer again so the OpenLM Platform services use the new password.

How do I connect to a bundled database directly?​

Run the command for your database:

# PostgreSQL
kubectl port-forward -n openlm-infrastructure svc/postgres-postgresql 5432:5432

# MariaDB
kubectl port-forward -n openlm-infrastructure svc/mariadb 3306:3306

# MongoDB
kubectl port-forward -n openlm-infrastructure svc/mongodb 27017:27017

Then run psql, mysql, or mongosh from another terminal pointing at localhost.

Kafka issues​

This section covers missing topics and how to inspect them.

Why are Kafka topics missing?​

If a service log shows topic does not exist or Unknown topic or partition, run the topic creation script again:

kubectl cp installation_files/kafka_topics/full_topic_list_create.sh \
  openlm-infrastructure/kafka-controller-0:/tmp/create-topics.sh
kubectl exec -n openlm-infrastructure kafka-controller-0 -- \
  bash /tmp/create-topics.sh

You can run the script again safely. It skips topics that already exist.

How do I list the existing Kafka topics?​

Run this command:

kubectl exec -n openlm-infrastructure kafka-controller-0 -- \
  /opt/bitnami/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --list

How do I inspect a single Kafka topic?​

To check a topic's partition count, replication factor, and configuration:

kubectl exec -n openlm-infrastructure kafka-controller-0 -- \
  /opt/bitnami/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --describe --topic <topic-name>

How do I free disk space on the VM?​

K3s, container images, and database data all live on the VM's primary disk. If pods start failing with eviction warnings, check disk usage:

df -h
sudo du -sh /var/lib/rancher/k3s/*
sudo du -sh /var/lib/rancher/k3s/storage/*   # persistent volume data

If /var/lib/rancher/k3s is full, the simplest fix is to extend the disk. Pruning container images is only a stopgap, because the images are pulled again the next time a pod is rescheduled or upgraded. To prune them, run these commands:

sudo k3s crictl images        # list
sudo k3s crictl rmi --prune   # remove unused

Never delete anything under /var/lib/rancher/k3s/storage/. That is live persistent volume data, including your databases.

Get help​

When you contact OpenLM support, include:

  1. Output of kubectl get pods -A.
  2. Output of helm list -A.
  3. Logs of the failing pod: kubectl logs <pod> -n <namespace> --tail 500.
  4. kubectl describe pod <pod> -n <namespace> if the pod isn't running.
  5. Your config.yaml (redact passwords) and the version of the deployment package.

The documentation team might draft this page using a large language model. A technical writer reviews every page and verifies it against the product before publishing it. A human makes the final decision.