DumbMonit
DE
GitHub Live-Demo

Monitore

Hypervisoren und Cluster

Proxmox, vSphere, XCP-ng, Nutanix und Kubernetes: Hosts, Gäste, Speicher und Cluster-Zustand.

Beschreibungen der Monitore und Texte der Alarmregeln sind wie im Produkt auf Englisch. Die Beschriftungen der Seite folgen Ihrer Sprache.

Proxmox VE

proxmox

Nodes, virtual machines, containers, storage and backups of a Proxmox server or cluster.

Beispiele

  • Proxmox server
  • Proxmox cluster

Was Sie brauchen

  • API token (recommended)No expiry and no session opened: the right choice for monitoring.
  • Username / passwordOpens a two-hour session, renewed automatically. Use it only if tokens are not an option.

Standardport: 8006

Mitgelieferte Alarmregeln 32

  • HinweisBackup too oldNo successful Proxmox VE backup for more than seven days.
  • HinweisVM or container stoppedThe guest was running within the last two hours and has been stopped for five minutes == 1).
  • HinweisVM or container CPU highA guest has used more than 90 % of its allocated cores for fifteen minutes. The notification names the guest, nextcloud (202).
  • HinweisVM or container memory highA guest has used more than 95 % of its memory for ten minutes.
  • HinweisVM or container disk almost fullThe root disk of a guest is more than 90 % full: containers always, VMs only when the QEMU guest agent reports usage.
  • HinweisGuest lockedA VM or container has been locked for six hours: an interrupted backup or migration left the lock behind and no operation can run on the guest, not even the next backup.
26 weitere anzeigen
  • InfoOld snapshotThe oldest snapshot of a guest is more than thirty days old.
  • WarnungHA resource in errorA Proxmox HA resource is in error or fenced.
  • WarnungHA manager not reportingThe HA local resource manager of a node has stopped writing its state: that node's HA guests will be neither restarted nor relocated.
  • WarnungCluster lost quorumThe Proxmox cluster is not quorate: guests can no longer be started.
  • WarnungProxmox node offlineA cluster node is reported offline.
  • WarnungProxmox service downA core Proxmox daemon is not running on a node.
  • HinweisNode network interface downAn interface set to start at boot is not up on a node. An interface brought up on demand is never reported.
  • HinweisProxmox storage almost fullA Proxmox storage is more than 85% full; "Disk almost full" takes over at 90%.
  • WarnungThin pool almost fullAn LVM thin pool is more than 90 % full: every guest stored on it goes read-only when it fills; clears below 88 %.
  • WarnungThin pool metadata almost fullThe metadata volume of an LVM thin pool is more than 80 % full. It is far smaller than the data volume and often saturates first, with the same consequence.
  • HinweisProxmox disk wearing outAn SSD or NVMe of a node has used more than 90 % of its rated life. The notification reads pve1 · /dev/nvme0n1.
  • WarnungProxmox disk SMART failureA disk of a node reports SMART health FAILED.
  • WarnungProxmox ZFS pool degradedA ZFS pool of a node is not ONLINE. The notification reads pve2 · tank.
  • WarnungBackup job failedThe last vzdump job on a node, or a scheduled backup job, failed.
  • HinweisBackup job excludes a diskA scheduled backup job skips a disk of one of the guests it covers: the job succeeds every night with that data missing. CD-ROM drives and cloudinit images are excluded by design and never counted.
  • WarnungReplication failedA replication job reports an error or a non-zero fail count: the standby copy is stale.
  • WarnungCeph health errorCeph reports HEALTH_ERR.
  • HinweisCeph health warningCeph reports HEALTH_WARN for more than fifteen minutes.
  • WarnungCeph OSD downA Ceph OSD has been down for five minutes. The notification names the OSD and its host.
  • HinweisCeph OSD nearly fullA single Ceph OSD is more than 85 % full: writes stop when it reaches its limit, whatever the cluster average says; clears below 82 %.
  • HinweisCeph pool almost fullA Ceph pool is more than 85 % full; clears below 82 %.
  • InfoCeph noout flag left onThe Ceph noout flag has been set for two hours: rebalancing is disabled, and the cluster reads healthy while its redundancy shrinks.
  • InfoProxmox updates pendingMore than twenty package updates are pending on a node.
  • HinweisProxmox security updates pendingAt least one pending update comes from a security archive.
  • InfoProxmox packages changedThe installed version of a Proxmox package changed between two probes — someone ran apt upgrade. Resolves by itself an hour later.
  • HinweisNode certificate expiringA Proxmox node certificate expires in less than fourteen days.
Einrichtungsschritte: Create a read-only user and token in Proxmox VE

  1. Open Datacenter → Permissions → Users → Add, enter the user name dumbmonit, choose the realm Proxmox VE authentication server, then click Add.

    dumbmonit@pve

  2. Open Datacenter → Permissions → Add → User Permission: path /, user dumbmonit@pve, role PVEAuditor, then click Add.

  3. Open Datacenter → Permissions → API Tokens → Add: user dumbmonit@pve, Token ID monitor, untick Privilege Separation, then click Add.

    dumbmonit@pve!monitor

  4. Copy the Token ID and the Secret into DumbMonit's Token ID and Secret fields. The secret is shown only once.

  5. Optional, for the pending-updates count: Datacenter → Permissions → Roles → Create a role with only Sys.Modify, then add it as a User Permission on path /nodes.

  6. In DumbMonit, enter the address of any node (port 8006 by default).

Hinweis. Proxmox VE uses a self-signed certificate by default: tick "Accept an unverifiable certificate" in the options if the connection is refused. On Proxmox VE 8, guest agent details also need VM.Monitor (see the guide).

Dokumentation

Proxmox Datacenter Manager

pdm

The console over several clusters: which instances it reaches, their failed tasks, its health.

Beispiele

  • Proxmox Datacenter Manager

Was Sie brauchen

  • API token (recommended)No expiry and no session opened: the right choice for monitoring.

Standardport: 8443

Mitgelieferte Alarmregeln 6

  • WarnungFederated instance unreachableThe datacenter console cannot reach one of the Proxmox VE clusters or backup servers it federates. The notification names the instance and its product.
  • InfoFederated instance behindAn instance runs an older version than another instance of the same product in the same estate. Compared between instances, never against what Proxmox publishes.
  • HinweisTask failed on a federated instanceAt least one task failed on a federated instance in the review window.
  • HinweisDatacenter console disk almost fullThe root filesystem of the datacenter console is more than 90 % full: full, the console stops recording what it collects.
  • HinweisDatacenter console certificate expiringA certificate of the datacenter console expires in less than fourteen days.
  • InfoDatacenter console updates pendingMore than twenty package updates are pending on the datacenter console.
Einrichtungsschritte: Create a read-only user and token in Proxmox Datacenter Manager

  1. Open Configuration → Access Control → Users → Add, enter the user name dumbmonit with the realm pdm and no password, then click Add.

    dumbmonit@pdm

  2. Open Configuration → Access Control → API Tokens → Add, pick that user and name the token monitor. Copy the Secret now, it is shown only once.

    dumbmonit@pdm!monitor

  3. Open Configuration → Access Control → Permissions → Add → User Permission: path /, user dumbmonit@pdm, role Auditor.

  4. Add a second permission with Add → API Token Permission: path /, token dumbmonit@pdm!monitor, role Auditor. A token has its own permissions, so both are needed.

  5. In DumbMonit, enter the console address (for example dc.lan:8443), the Token ID and the Secret.

Hinweis. The console uses a self-signed certificate by default: tick "Accept an unverifiable certificate" in the options if the connection is refused.

Dokumentation

VMware vSphere

vsphere

vCenter and ESXi hosts: hosts not responding or left in maintenance, VMs and their VMware Tools, datastores filling up, and the alarms vSphere raised.

Beispiele

  • vCenter Server
  • ESXi (standalone host)

Was Sie brauchen

  • User name / passwordThe dedicated user with the Read-only role. On a vCenter, include the domain.

Standardport: 443

Mitgelieferte Alarmregeln 8

  • WarnungvSphere host not respondingA vSphere host is not responding to vCenter. A host disconnected on purpose (1) does not fire.
  • WarnungvSphere host health redvSphere reports a host's overall health as red. Gray means unknown and does not fire.
  • InfovSphere host left in maintenanceA vSphere host has been in maintenance mode for a day: its VMs were moved away and it may have been forgotten there.
  • HinweisvSphere datastore almost fullA vSphere datastore is 90% full or more; clears below 85 %.
  • WarnungvSphere datastore inaccessibleA vSphere datastore is inaccessible: the VMs stored on it cannot run.
  • InfovSphere VM tools not runningVMware Tools are installed but not running in a powered-on VM: the guest may be hung. VMs without Tools do not fire.
2 weitere anzeigen
  • HinweisvSphere red alarmvSphere raised a red alarm that nobody acknowledged.
  • WarnungMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Einrichtungsschritte: Create a read-only user in vSphere

  1. On a vCenter, open Administration → Single Sign On → Users and Groups → Users, choose the domain vsphere.local, click Add and enter this user name with a long random password.

    dumbmonit

  2. Open Administration → Access Control → Global Permissions → Add: pick that user, role Read-only, tick Propagate to children.

  3. On a standalone ESXi host instead: in the host client, Manage → Security & users → Users → Add user dumbmonit, then Host → Actions → Permissions → Add user with role Read-only.

  4. In DumbMonit, enter the vCenter or host address, with the user [email protected] on a vCenter or dumbmonit on a host. Add only the vCenter when one manages the hosts.

Hinweis. vCenter and ESXi use a self-signed certificate by default: enable "Accept an unverifiable certificate" unless the DumbMonit host trusts their certificate authority.

Dokumentation

XCP-ng / XenServer

xcpng

An XCP-ng or XenServer pool through XAPI: hosts disabled or offline, VMs and their power state, storage repositories filling up, and a high-availability pool that can no longer survive the host failures it is set to tolerate.

Beispiele

  • XCP-ng 8.2 and 8.3
  • XenServer / Citrix Hypervisor

Was Sie brauchen

  • User name and passwordThe dumbmonit account, registered on the pool as a subject with the read-only role.

Standardport: 443

Mitgelieferte Alarmregeln 5

  • WarnungXCP-ng host offlineA host of the XCP-ng pool no longer answers the coordinator.
  • HinweisXCP-ng SR almost fullAn XCP-ng storage repository is more than 90% full.
  • WarnungXCP-ng HA-protected VM downA VM protected by the pool's high availability is halted.
  • HinweisXCP-ng pool HA overcommittedThe pool no longer has a plan to survive the host failures it is set to tolerate.
  • WarnungMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Einrichtungsschritte: Give DumbMonit a read-only XAPI account

  1. On every host of the pool, create a local account with a long random password.

    useradd -M -s /sbin/nologin dumbmonit
passwd dumbmonit

  2. On the pool coordinator, turn on PAM authentication once.

    xe pool-enable-external-auth auth-type=PAM service-name=pam

  3. Register the account as a subject and note the UUID it prints.

    xe subject-add subject-name=dumbmonit

  4. Give that subject the built-in read-only role, with the UUID.

    xe subject-role-add uuid=SUBJECT_UUID role-name=read-only

  5. In DumbMonit, enter the address of the pool coordinator (for example xcp01.lan) with that user name and password.

Hinweis. If the pool already uses Active Directory, create the account there and skip steps 1 and 2. XCP-ng uses a self-signed certificate: enable "Accept an unverifiable certificate" unless the DumbMonit host trusts it.

Dokumentation

Nutanix AHV (Prism Central)

nutanix

Nutanix clusters through the v4 API of Prism Central: hosts degraded or in maintenance, AHV VMs and their power state, storage containers filling up, and unresolved alerts by severity.

Beispiele

  • Prism Central pc.2024.3 or later
  • AHV clusters

Was Sie brauchen

  • User name and passwordThe dumbmonit local user of Prism Central, with the Prism Viewer role.

Standardport: 9440

Mitgelieferte Alarmregeln 4

  • WarnungNutanix host degradedPrism reports a Nutanix host as degraded.
  • HinweisNutanix storage container almost fullA Nutanix storage container is more than 90% full.
  • WarnungNutanix critical alertsPrism Central reports one or more unresolved critical alerts.
  • WarnungMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Einrichtungsschritte: Create a read-only user in Prism Central

  1. In Prism Central, open Admin Center → IAM → Identities and add a local user with a long random password.

    dumbmonit

  2. Open Admin Center → IAM → Authorization Policies and create a policy with the Prism Viewer role, for all entities, assigned to that user.

  3. In DumbMonit, enter the Prism Central address (for example prism-central.lan) with that user name and password.

Hinweis. Needs Prism Central pc.2024.3 or later (v4 API); Prism Element alone is not supported. Prism Central uses a self-signed certificate: enable "Accept an unverifiable certificate" unless DumbMonit trusts it.

Dokumentation

Kubernetes / k3s

kubernetes

A cluster through its API: nodes not ready or under pressure, pods crash looping or stuck pending, workloads missing replicas, volume claims never bound.

Beispiele

  • k3s
  • kubeadm cluster
  • k0s
  • MicroK8s
  • Talos

Was Sie brauchen

  • Service account tokenThe token of the dumbmonit service account, read from its dumbmonit-token Secret.

Standardport: 6443

Mitgelieferte Alarmregeln 8

  • WarnungKubernetes node not readyA node is not Ready: its kubelet no longer reports, and its pods will be evicted.
  • HinweisKubernetes node under pressureA node reports memory, disk or PID pressure.
  • WarnungKubernetes pod crash loopingA container is in CrashLoopBackOff and still restarting: at least two restarts in 15 minutes (max_over_time(dumbmonit_k8s_pod_crashlooping[10m]) and on (target, namespace, pod) increase_prometheus(dumbmonit_k8s_pod_restarts[15m]) >= 2; the state disappears for a moment at each restart attempt). Containers killed by a node or runtime restart (exit 137, 143 or 255, not OOMKilled) do not count, so a reboot does not look like a crash loop.
  • HinweisKubernetes pod stuck pendingA pod has been Pending for fifteen minutes: no node can take it, or a volume or an image it needs does not come.
  • HinweisKubernetes pod restartingA pod restarted more than five times in the last hour without being in CrashLoopBackOff (increase_prometheus(dumbmonit_k8s_pod_restarts[1h]) unless on (target, namespace, pod) max_over_time(dumbmonit_k8s_pod_crashlooping[10m]) > 0).
  • HinweisKubernetes workload missing replicasA Deployment, StatefulSet or DaemonSet has fewer available replicas than it asks for.
2 weitere anzeigen
  • HinweisKubernetes volume claim pendingA PersistentVolumeClaim that a pod needs, or that a provisioning error concerns, is still not bound.
  • WarnungMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Einrichtungsschritte: Create a read-only service account for DumbMonit

  1. Create the service account.

    kubectl create serviceaccount dumbmonit -n kube-system

  2. Create a ClusterRole that can only get and list what DumbMonit reads.

    kubectl create clusterrole dumbmonit-read --verb=get,list --resource=nodes,pods,persistentvolumeclaims,events,deployments.apps,statefulsets.apps,daemonsets.apps

  3. Bind the role to the service account.

    kubectl create clusterrolebinding dumbmonit-read --clusterrole=dumbmonit-read --serviceaccount=kube-system:dumbmonit

  4. Create a token that does not expire, then print the token and the cluster CA in base64.

    echo '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"dumbmonit-token","namespace":"kube-system","annotations":{"kubernetes.io/service-account.name":"dumbmonit"}},"type":"kubernetes.io/service-account-token"}' | kubectl apply -f -
kubectl -n kube-system get secret dumbmonit-token -o jsonpath='{.data.token}' | base64 -d
kubectl -n kube-system get secret dumbmonit-token -o jsonpath='{.data.ca\.crt}'

  5. In DumbMonit, enter the API server address (for example https://k3s.lan:6443), paste the token as Service account token and the CA as Cluster CA certificate.

Hinweis. On k3s, run each command as sudo k3s kubectl. The API server certificate only lists its own names and addresses: use one it lists, or tick Accept an unverifiable certificate.

Dokumentation

Weitere Familien