DumbMonit
IT
GitHub Demo dal vivo

Monitor

Hypervisor e cluster

Proxmox, vSphere, XCP-ng, Nutanix e Kubernetes: host, guest, storage e stato del cluster.

Le descrizioni dei monitor e i testi delle regole di avviso sono in inglese, come nel prodotto. Le etichette della pagina seguono la tua lingua.

Proxmox VE

proxmox

Nodes, virtual machines, containers, storage and backups of a Proxmox server or cluster.

Esempi

  • Proxmox server
  • Proxmox cluster

Cosa serve

  • API token (recommended)No expiry and no session opened: the right choice for monitoring.
  • Username / passwordOpens a two-hour session, renewed automatically. Use it only if tokens are not an option.

Porta predefinita: 8006

Regole di avviso incluse 32

  • AvvisoBackup too oldNo successful Proxmox VE backup for more than seven days.
  • AvvisoVM or container stoppedThe guest was running within the last two hours and has been stopped for five minutes == 1).
  • AvvisoVM or container CPU highA guest has used more than 90 % of its allocated cores for fifteen minutes. The notification names the guest, nextcloud (202).
  • AvvisoVM or container memory highA guest has used more than 95 % of its memory for ten minutes.
  • AvvisoVM or container disk almost fullThe root disk of a guest is more than 90 % full: containers always, VMs only when the QEMU guest agent reports usage.
  • AvvisoGuest lockedA VM or container has been locked for six hours: an interrupted backup or migration left the lock behind and no operation can run on the guest, not even the next backup.
Mostra altre 26
  • InfoOld snapshotThe oldest snapshot of a guest is more than thirty days old.
  • AllarmeHA resource in errorA Proxmox HA resource is in error or fenced.
  • AllarmeHA manager not reportingThe HA local resource manager of a node has stopped writing its state: that node's HA guests will be neither restarted nor relocated.
  • AllarmeCluster lost quorumThe Proxmox cluster is not quorate: guests can no longer be started.
  • AllarmeProxmox node offlineA cluster node is reported offline.
  • AllarmeProxmox service downA core Proxmox daemon is not running on a node.
  • AvvisoNode network interface downAn interface set to start at boot is not up on a node. An interface brought up on demand is never reported.
  • AvvisoProxmox storage almost fullA Proxmox storage is more than 85% full; "Disk almost full" takes over at 90%.
  • AllarmeThin pool almost fullAn LVM thin pool is more than 90 % full: every guest stored on it goes read-only when it fills; clears below 88 %.
  • AllarmeThin pool metadata almost fullThe metadata volume of an LVM thin pool is more than 80 % full. It is far smaller than the data volume and often saturates first, with the same consequence.
  • AvvisoProxmox disk wearing outAn SSD or NVMe of a node has used more than 90 % of its rated life. The notification reads pve1 · /dev/nvme0n1.
  • AllarmeProxmox disk SMART failureA disk of a node reports SMART health FAILED.
  • AllarmeProxmox ZFS pool degradedA ZFS pool of a node is not ONLINE. The notification reads pve2 · tank.
  • AllarmeBackup job failedThe last vzdump job on a node, or a scheduled backup job, failed.
  • AvvisoBackup job excludes a diskA scheduled backup job skips a disk of one of the guests it covers: the job succeeds every night with that data missing. CD-ROM drives and cloudinit images are excluded by design and never counted.
  • AllarmeReplication failedA replication job reports an error or a non-zero fail count: the standby copy is stale.
  • AllarmeCeph health errorCeph reports HEALTH_ERR.
  • AvvisoCeph health warningCeph reports HEALTH_WARN for more than fifteen minutes.
  • AllarmeCeph OSD downA Ceph OSD has been down for five minutes. The notification names the OSD and its host.
  • AvvisoCeph OSD nearly fullA single Ceph OSD is more than 85 % full: writes stop when it reaches its limit, whatever the cluster average says; clears below 82 %.
  • AvvisoCeph pool almost fullA Ceph pool is more than 85 % full; clears below 82 %.
  • InfoCeph noout flag left onThe Ceph noout flag has been set for two hours: rebalancing is disabled, and the cluster reads healthy while its redundancy shrinks.
  • InfoProxmox updates pendingMore than twenty package updates are pending on a node.
  • AvvisoProxmox security updates pendingAt least one pending update comes from a security archive.
  • InfoProxmox packages changedThe installed version of a Proxmox package changed between two probes — someone ran apt upgrade. Resolves by itself an hour later.
  • AvvisoNode certificate expiringA Proxmox node certificate expires in less than fourteen days.
Passaggi di configurazione: Create a read-only user and token in Proxmox VE

  1. Open Datacenter → Permissions → Users → Add, enter the user name dumbmonit, choose the realm Proxmox VE authentication server, then click Add.

    dumbmonit@pve

  2. Open Datacenter → Permissions → Add → User Permission: path /, user dumbmonit@pve, role PVEAuditor, then click Add.

  3. Open Datacenter → Permissions → API Tokens → Add: user dumbmonit@pve, Token ID monitor, untick Privilege Separation, then click Add.

    dumbmonit@pve!monitor

  4. Copy the Token ID and the Secret into DumbMonit's Token ID and Secret fields. The secret is shown only once.

  5. Optional, for the pending-updates count: Datacenter → Permissions → Roles → Create a role with only Sys.Modify, then add it as a User Permission on path /nodes.

  6. In DumbMonit, enter the address of any node (port 8006 by default).

Attenzione. Proxmox VE uses a self-signed certificate by default: tick "Accept an unverifiable certificate" in the options if the connection is refused. On Proxmox VE 8, guest agent details also need VM.Monitor (see the guide).

Documentazione

Proxmox Datacenter Manager

pdm

The console over several clusters: which instances it reaches, their failed tasks, its health.

Esempi

  • Proxmox Datacenter Manager

Cosa serve

  • API token (recommended)No expiry and no session opened: the right choice for monitoring.

Porta predefinita: 8443

Regole di avviso incluse 6

  • AllarmeFederated instance unreachableThe datacenter console cannot reach one of the Proxmox VE clusters or backup servers it federates. The notification names the instance and its product.
  • InfoFederated instance behindAn instance runs an older version than another instance of the same product in the same estate. Compared between instances, never against what Proxmox publishes.
  • AvvisoTask failed on a federated instanceAt least one task failed on a federated instance in the review window.
  • AvvisoDatacenter console disk almost fullThe root filesystem of the datacenter console is more than 90 % full: full, the console stops recording what it collects.
  • AvvisoDatacenter console certificate expiringA certificate of the datacenter console expires in less than fourteen days.
  • InfoDatacenter console updates pendingMore than twenty package updates are pending on the datacenter console.
Passaggi di configurazione: Create a read-only user and token in Proxmox Datacenter Manager

  1. Open Configuration → Access Control → Users → Add, enter the user name dumbmonit with the realm pdm and no password, then click Add.

    dumbmonit@pdm

  2. Open Configuration → Access Control → API Tokens → Add, pick that user and name the token monitor. Copy the Secret now, it is shown only once.

    dumbmonit@pdm!monitor

  3. Open Configuration → Access Control → Permissions → Add → User Permission: path /, user dumbmonit@pdm, role Auditor.

  4. Add a second permission with Add → API Token Permission: path /, token dumbmonit@pdm!monitor, role Auditor. A token has its own permissions, so both are needed.

  5. In DumbMonit, enter the console address (for example dc.lan:8443), the Token ID and the Secret.

Attenzione. The console uses a self-signed certificate by default: tick "Accept an unverifiable certificate" in the options if the connection is refused.

Documentazione

VMware vSphere

vsphere

vCenter and ESXi hosts: hosts not responding or left in maintenance, VMs and their VMware Tools, datastores filling up, and the alarms vSphere raised.

Esempi

  • vCenter Server
  • ESXi (standalone host)

Cosa serve

  • User name / passwordThe dedicated user with the Read-only role. On a vCenter, include the domain.

Porta predefinita: 443

Regole di avviso incluse 8

  • AllarmevSphere host not respondingA vSphere host is not responding to vCenter. A host disconnected on purpose (1) does not fire.
  • AllarmevSphere host health redvSphere reports a host's overall health as red. Gray means unknown and does not fire.
  • InfovSphere host left in maintenanceA vSphere host has been in maintenance mode for a day: its VMs were moved away and it may have been forgotten there.
  • AvvisovSphere datastore almost fullA vSphere datastore is 90% full or more; clears below 85 %.
  • AllarmevSphere datastore inaccessibleA vSphere datastore is inaccessible: the VMs stored on it cannot run.
  • InfovSphere VM tools not runningVMware Tools are installed but not running in a powered-on VM: the guest may be hung. VMs without Tools do not fire.
Mostra altre 2
  • AvvisovSphere red alarmvSphere raised a red alarm that nobody acknowledged.
  • AllarmeMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Passaggi di configurazione: Create a read-only user in vSphere

  1. On a vCenter, open Administration → Single Sign On → Users and Groups → Users, choose the domain vsphere.local, click Add and enter this user name with a long random password.

    dumbmonit

  2. Open Administration → Access Control → Global Permissions → Add: pick that user, role Read-only, tick Propagate to children.

  3. On a standalone ESXi host instead: in the host client, Manage → Security & users → Users → Add user dumbmonit, then Host → Actions → Permissions → Add user with role Read-only.

  4. In DumbMonit, enter the vCenter or host address, with the user [email protected] on a vCenter or dumbmonit on a host. Add only the vCenter when one manages the hosts.

Attenzione. vCenter and ESXi use a self-signed certificate by default: enable "Accept an unverifiable certificate" unless the DumbMonit host trusts their certificate authority.

Documentazione

XCP-ng / XenServer

xcpng

An XCP-ng or XenServer pool through XAPI: hosts disabled or offline, VMs and their power state, storage repositories filling up, and a high-availability pool that can no longer survive the host failures it is set to tolerate.

Esempi

  • XCP-ng 8.2 and 8.3
  • XenServer / Citrix Hypervisor

Cosa serve

  • User name and passwordThe dumbmonit account, registered on the pool as a subject with the read-only role.

Porta predefinita: 443

Regole di avviso incluse 5

  • AllarmeXCP-ng host offlineA host of the XCP-ng pool no longer answers the coordinator.
  • AvvisoXCP-ng SR almost fullAn XCP-ng storage repository is more than 90% full.
  • AllarmeXCP-ng HA-protected VM downA VM protected by the pool's high availability is halted.
  • AvvisoXCP-ng pool HA overcommittedThe pool no longer has a plan to survive the host failures it is set to tolerate.
  • AllarmeMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Passaggi di configurazione: Give DumbMonit a read-only XAPI account

  1. On every host of the pool, create a local account with a long random password.

    useradd -M -s /sbin/nologin dumbmonit
passwd dumbmonit

  2. On the pool coordinator, turn on PAM authentication once.

    xe pool-enable-external-auth auth-type=PAM service-name=pam

  3. Register the account as a subject and note the UUID it prints.

    xe subject-add subject-name=dumbmonit

  4. Give that subject the built-in read-only role, with the UUID.

    xe subject-role-add uuid=SUBJECT_UUID role-name=read-only

  5. In DumbMonit, enter the address of the pool coordinator (for example xcp01.lan) with that user name and password.

Attenzione. If the pool already uses Active Directory, create the account there and skip steps 1 and 2. XCP-ng uses a self-signed certificate: enable "Accept an unverifiable certificate" unless the DumbMonit host trusts it.

Documentazione

Nutanix AHV (Prism Central)

nutanix

Nutanix clusters through the v4 API of Prism Central: hosts degraded or in maintenance, AHV VMs and their power state, storage containers filling up, and unresolved alerts by severity.

Esempi

  • Prism Central pc.2024.3 or later
  • AHV clusters

Cosa serve

  • User name and passwordThe dumbmonit local user of Prism Central, with the Prism Viewer role.

Porta predefinita: 9440

Regole di avviso incluse 4

  • AllarmeNutanix host degradedPrism reports a Nutanix host as degraded.
  • AvvisoNutanix storage container almost fullA Nutanix storage container is more than 90% full.
  • AllarmeNutanix critical alertsPrism Central reports one or more unresolved critical alerts.
  • AllarmeMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Passaggi di configurazione: Create a read-only user in Prism Central

  1. In Prism Central, open Admin Center → IAM → Identities and add a local user with a long random password.

    dumbmonit

  2. Open Admin Center → IAM → Authorization Policies and create a policy with the Prism Viewer role, for all entities, assigned to that user.

  3. In DumbMonit, enter the Prism Central address (for example prism-central.lan) with that user name and password.

Attenzione. Needs Prism Central pc.2024.3 or later (v4 API); Prism Element alone is not supported. Prism Central uses a self-signed certificate: enable "Accept an unverifiable certificate" unless DumbMonit trusts it.

Documentazione

Kubernetes / k3s

kubernetes

A cluster through its API: nodes not ready or under pressure, pods crash looping or stuck pending, workloads missing replicas, volume claims never bound.

Esempi

  • k3s
  • kubeadm cluster
  • k0s
  • MicroK8s
  • Talos

Cosa serve

  • Service account tokenThe token of the dumbmonit service account, read from its dumbmonit-token Secret.

Porta predefinita: 6443

Regole di avviso incluse 8

  • AllarmeKubernetes node not readyA node is not Ready: its kubelet no longer reports, and its pods will be evicted.
  • AvvisoKubernetes node under pressureA node reports memory, disk or PID pressure.
  • AllarmeKubernetes pod crash loopingA container is in CrashLoopBackOff and still restarting: at least two restarts in 15 minutes (max_over_time(dumbmonit_k8s_pod_crashlooping[10m]) and on (target, namespace, pod) increase_prometheus(dumbmonit_k8s_pod_restarts[15m]) >= 2; the state disappears for a moment at each restart attempt). Containers killed by a node or runtime restart (exit 137, 143 or 255, not OOMKilled) do not count, so a reboot does not look like a crash loop.
  • AvvisoKubernetes pod stuck pendingA pod has been Pending for fifteen minutes: no node can take it, or a volume or an image it needs does not come.
  • AvvisoKubernetes pod restartingA pod restarted more than five times in the last hour without being in CrashLoopBackOff (increase_prometheus(dumbmonit_k8s_pod_restarts[1h]) unless on (target, namespace, pod) max_over_time(dumbmonit_k8s_pod_crashlooping[10m]) > 0).
  • AvvisoKubernetes workload missing replicasA Deployment, StatefulSet or DaemonSet has fewer available replicas than it asks for.
Mostra altre 2
  • AvvisoKubernetes volume claim pendingA PersistentVolumeClaim that a pod needs, or that a provisioning error concerns, is still not bound.
  • AllarmeMonitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
Passaggi di configurazione: Create a read-only service account for DumbMonit

  1. Create the service account.

    kubectl create serviceaccount dumbmonit -n kube-system

  2. Create a ClusterRole that can only get and list what DumbMonit reads.

    kubectl create clusterrole dumbmonit-read --verb=get,list --resource=nodes,pods,persistentvolumeclaims,events,deployments.apps,statefulsets.apps,daemonsets.apps

  3. Bind the role to the service account.

    kubectl create clusterrolebinding dumbmonit-read --clusterrole=dumbmonit-read --serviceaccount=kube-system:dumbmonit

  4. Create a token that does not expire, then print the token and the cluster CA in base64.

    echo '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"dumbmonit-token","namespace":"kube-system","annotations":{"kubernetes.io/service-account.name":"dumbmonit"}},"type":"kubernetes.io/service-account-token"}' | kubectl apply -f -
kubectl -n kube-system get secret dumbmonit-token -o jsonpath='{.data.token}' | base64 -d
kubectl -n kube-system get secret dumbmonit-token -o jsonpath='{.data.ca\.crt}'

  5. In DumbMonit, enter the API server address (for example https://k3s.lan:6443), paste the token as Service account token and the CA as Cluster CA certificate.

Attenzione. On k3s, run each command as sudo k3s kubectl. The API server certificate only lists its own names and addresses: use one it lists, or tick Accept an unverifiable certificate.

Documentazione

Altre famiglie