监控项
虚拟化平台与集群
Proxmox、vSphere、XCP-ng、Nutanix 和 Kubernetes:主机、客户机、存储和集群健康。
监控项描述和告警规则文本为英文,与产品一致。页面标签使用您的语言。
Nodes, virtual machines, containers, storage and backups of a Proxmox server or cluster.
示例
- Proxmox server
- Proxmox cluster
您需要准备
- API token (recommended)No expiry and no session opened: the right choice for monitoring.
- Username / passwordOpens a two-hour session, renewed automatically. Use it only if tokens are not an option.
默认端口: 8006
内置告警规则 32
- 提示Backup too oldNo successful Proxmox VE backup for more than seven days.
- 提示VM or container stoppedThe guest was running within the last two hours and has been stopped for five minutes == 1).
- 提示VM or container CPU highA guest has used more than 90 % of its allocated cores for fifteen minutes. The notification names the guest, nextcloud (202).
- 提示VM or container memory highA guest has used more than 95 % of its memory for ten minutes.
- 提示VM or container disk almost fullThe root disk of a guest is more than 90 % full: containers always, VMs only when the QEMU guest agent reports usage.
- 提示Guest lockedA VM or container has been locked for six hours: an interrupted backup or migration left the lock behind and no operation can run on the guest, not even the next backup.
再显示 26 条
- 信息Old snapshotThe oldest snapshot of a guest is more than thirty days old.
- 警告HA resource in errorA Proxmox HA resource is in error or fenced.
- 警告HA manager not reportingThe HA local resource manager of a node has stopped writing its state: that node's HA guests will be neither restarted nor relocated.
- 警告Cluster lost quorumThe Proxmox cluster is not quorate: guests can no longer be started.
- 警告Proxmox node offlineA cluster node is reported offline.
- 警告Proxmox service downA core Proxmox daemon is not running on a node.
- 提示Node network interface downAn interface set to start at boot is not up on a node. An interface brought up on demand is never reported.
- 提示Proxmox storage almost fullA Proxmox storage is more than 85% full; "Disk almost full" takes over at 90%.
- 警告Thin pool almost fullAn LVM thin pool is more than 90 % full: every guest stored on it goes read-only when it fills; clears below 88 %.
- 警告Thin pool metadata almost fullThe metadata volume of an LVM thin pool is more than 80 % full. It is far smaller than the data volume and often saturates first, with the same consequence.
- 提示Proxmox disk wearing outAn SSD or NVMe of a node has used more than 90 % of its rated life. The notification reads pve1 · /dev/nvme0n1.
- 警告Proxmox disk SMART failureA disk of a node reports SMART health FAILED.
- 警告Proxmox ZFS pool degradedA ZFS pool of a node is not ONLINE. The notification reads pve2 · tank.
- 警告Backup job failedThe last vzdump job on a node, or a scheduled backup job, failed.
- 提示Backup job excludes a diskA scheduled backup job skips a disk of one of the guests it covers: the job succeeds every night with that data missing. CD-ROM drives and cloudinit images are excluded by design and never counted.
- 警告Replication failedA replication job reports an error or a non-zero fail count: the standby copy is stale.
- 警告Ceph health errorCeph reports HEALTH_ERR.
- 提示Ceph health warningCeph reports HEALTH_WARN for more than fifteen minutes.
- 警告Ceph OSD downA Ceph OSD has been down for five minutes. The notification names the OSD and its host.
- 提示Ceph OSD nearly fullA single Ceph OSD is more than 85 % full: writes stop when it reaches its limit, whatever the cluster average says; clears below 82 %.
- 提示Ceph pool almost fullA Ceph pool is more than 85 % full; clears below 82 %.
- 信息Ceph noout flag left onThe Ceph noout flag has been set for two hours: rebalancing is disabled, and the cluster reads healthy while its redundancy shrinks.
- 信息Proxmox updates pendingMore than twenty package updates are pending on a node.
- 提示Proxmox security updates pendingAt least one pending update comes from a security archive.
- 信息Proxmox packages changedThe installed version of a Proxmox package changed between two probes — someone ran apt upgrade. Resolves by itself an hour later.
- 提示Node certificate expiringA Proxmox node certificate expires in less than fourteen days.
设置步骤: Create a read-only user and token in Proxmox VE
Open Datacenter → Permissions → Users → Add, enter the user name dumbmonit, choose the realm Proxmox VE authentication server, then click Add.
dumbmonit@pve
Open Datacenter → Permissions → Add → User Permission: path /, user dumbmonit@pve, role PVEAuditor, then click Add.
Open Datacenter → Permissions → API Tokens → Add: user dumbmonit@pve, Token ID monitor, untick Privilege Separation, then click Add.
dumbmonit@pve!monitor
Copy the Token ID and the Secret into DumbMonit's Token ID and Secret fields. The secret is shown only once.
Optional, for the pending-updates count: Datacenter → Permissions → Roles → Create a role with only Sys.Modify, then add it as a User Permission on path /nodes.
In DumbMonit, enter the address of any node (port 8006 by default).
注意. Proxmox VE uses a self-signed certificate by default: tick "Accept an unverifiable certificate" in the options if the connection is refused. On Proxmox VE 8, guest agent details also need VM.Monitor (see the guide).
文档
Proxmox Datacenter Manager
pdm
The console over several clusters: which instances it reaches, their failed tasks, its health.
示例
- Proxmox Datacenter Manager
您需要准备
- API token (recommended)No expiry and no session opened: the right choice for monitoring.
默认端口: 8443
内置告警规则 6
- 警告Federated instance unreachableThe datacenter console cannot reach one of the Proxmox VE clusters or backup servers it federates. The notification names the instance and its product.
- 信息Federated instance behindAn instance runs an older version than another instance of the same product in the same estate. Compared between instances, never against what Proxmox publishes.
- 提示Task failed on a federated instanceAt least one task failed on a federated instance in the review window.
- 提示Datacenter console disk almost fullThe root filesystem of the datacenter console is more than 90 % full: full, the console stops recording what it collects.
- 提示Datacenter console certificate expiringA certificate of the datacenter console expires in less than fourteen days.
- 信息Datacenter console updates pendingMore than twenty package updates are pending on the datacenter console.
设置步骤: Create a read-only user and token in Proxmox Datacenter Manager
Open Configuration → Access Control → Users → Add, enter the user name dumbmonit with the realm pdm and no password, then click Add.
dumbmonit@pdm
Open Configuration → Access Control → API Tokens → Add, pick that user and name the token monitor. Copy the Secret now, it is shown only once.
dumbmonit@pdm!monitor
Open Configuration → Access Control → Permissions → Add → User Permission: path /, user dumbmonit@pdm, role Auditor.
Add a second permission with Add → API Token Permission: path /, token dumbmonit@pdm!monitor, role Auditor. A token has its own permissions, so both are needed.
In DumbMonit, enter the console address (for example dc.lan:8443), the Token ID and the Secret.
注意. The console uses a self-signed certificate by default: tick "Accept an unverifiable certificate" in the options if the connection is refused.
文档
vCenter and ESXi hosts: hosts not responding or left in maintenance, VMs and their VMware Tools, datastores filling up, and the alarms vSphere raised.
示例
- vCenter Server
- ESXi (standalone host)
您需要准备
- User name / passwordThe dedicated user with the Read-only role. On a vCenter, include the domain.
默认端口: 443
内置告警规则 8
- 警告vSphere host not respondingA vSphere host is not responding to vCenter. A host disconnected on purpose (1) does not fire.
- 警告vSphere host health redvSphere reports a host's overall health as red. Gray means unknown and does not fire.
- 信息vSphere host left in maintenanceA vSphere host has been in maintenance mode for a day: its VMs were moved away and it may have been forgotten there.
- 提示vSphere datastore almost fullA vSphere datastore is 90% full or more; clears below 85 %.
- 警告vSphere datastore inaccessibleA vSphere datastore is inaccessible: the VMs stored on it cannot run.
- 信息vSphere VM tools not runningVMware Tools are installed but not running in a powered-on VM: the guest may be hung. VMs without Tools do not fire.
再显示 2 条
- 提示vSphere red alarmvSphere raised a red alarm that nobody acknowledged.
- 警告Monitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
设置步骤: Create a read-only user in vSphere
On a vCenter, open Administration → Single Sign On → Users and Groups → Users, choose the domain vsphere.local, click Add and enter this user name with a long random password.
dumbmonit
Open Administration → Access Control → Global Permissions → Add: pick that user, role Read-only, tick Propagate to children.
On a standalone ESXi host instead: in the host client, Manage → Security & users → Users → Add user dumbmonit, then Host → Actions → Permissions → Add user with role Read-only.
In DumbMonit, enter the vCenter or host address, with the user [email protected] on a vCenter or dumbmonit on a host. Add only the vCenter when one manages the hosts.
注意. vCenter and ESXi use a self-signed certificate by default: enable "Accept an unverifiable certificate" unless the DumbMonit host trusts their certificate authority.
文档
An XCP-ng or XenServer pool through XAPI: hosts disabled or offline, VMs and their power state, storage repositories filling up, and a high-availability pool that can no longer survive the host failures it is set to tolerate.
示例
- XCP-ng 8.2 and 8.3
- XenServer / Citrix Hypervisor
您需要准备
- User name and passwordThe dumbmonit account, registered on the pool as a subject with the read-only role.
默认端口: 443
内置告警规则 5
- 警告XCP-ng host offlineA host of the XCP-ng pool no longer answers the coordinator.
- 提示XCP-ng SR almost fullAn XCP-ng storage repository is more than 90% full.
- 警告XCP-ng HA-protected VM downA VM protected by the pool's high availability is halted.
- 提示XCP-ng pool HA overcommittedThe pool no longer has a plan to survive the host failures it is set to tolerate.
- 警告Monitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
设置步骤: Give DumbMonit a read-only XAPI account
On every host of the pool, create a local account with a long random password.
useradd -M -s /sbin/nologin dumbmonit
passwd dumbmonit
On the pool coordinator, turn on PAM authentication once.
xe pool-enable-external-auth auth-type=PAM service-name=pam
Register the account as a subject and note the UUID it prints.
xe subject-add subject-name=dumbmonit
Give that subject the built-in read-only role, with the UUID.
xe subject-role-add uuid=SUBJECT_UUID role-name=read-only
In DumbMonit, enter the address of the pool coordinator (for example xcp01.lan) with that user name and password.
注意. If the pool already uses Active Directory, create the account there and skip steps 1 and 2. XCP-ng uses a self-signed certificate: enable "Accept an unverifiable certificate" unless the DumbMonit host trusts it.
文档
Nutanix AHV (Prism Central)
nutanix
Nutanix clusters through the v4 API of Prism Central: hosts degraded or in maintenance, AHV VMs and their power state, storage containers filling up, and unresolved alerts by severity.
示例
- Prism Central pc.2024.3 or later
- AHV clusters
您需要准备
- User name and passwordThe dumbmonit local user of Prism Central, with the Prism Viewer role.
默认端口: 9440
内置告警规则 4
- 警告Nutanix host degradedPrism reports a Nutanix host as degraded.
- 提示Nutanix storage container almost fullA Nutanix storage container is more than 90% full.
- 警告Nutanix critical alertsPrism Central reports one or more unresolved critical alerts.
- 警告Monitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
设置步骤: Create a read-only user in Prism Central
In Prism Central, open Admin Center → IAM → Identities and add a local user with a long random password.
dumbmonit
Open Admin Center → IAM → Authorization Policies and create a policy with the Prism Viewer role, for all entities, assigned to that user.
In DumbMonit, enter the Prism Central address (for example prism-central.lan) with that user name and password.
注意. Needs Prism Central pc.2024.3 or later (v4 API); Prism Element alone is not supported. Prism Central uses a self-signed certificate: enable "Accept an unverifiable certificate" unless DumbMonit trusts it.
文档
Kubernetes / k3s
kubernetes
A cluster through its API: nodes not ready or under pressure, pods crash looping or stuck pending, workloads missing replicas, volume claims never bound.
示例
- k3s
- kubeadm cluster
- k0s
- MicroK8s
- Talos
您需要准备
- Service account tokenThe token of the dumbmonit service account, read from its dumbmonit-token Secret.
默认端口: 6443
内置告警规则 8
- 警告Kubernetes node not readyA node is not Ready: its kubelet no longer reports, and its pods will be evicted.
- 提示Kubernetes node under pressureA node reports memory, disk or PID pressure.
- 警告Kubernetes pod crash loopingA container is in CrashLoopBackOff and still restarting: at least two restarts in 15 minutes (max_over_time(dumbmonit_k8s_pod_crashlooping[10m]) and on (target, namespace, pod) increase_prometheus(dumbmonit_k8s_pod_restarts[15m]) >= 2; the state disappears for a moment at each restart attempt). Containers killed by a node or runtime restart (exit 137, 143 or 255, not OOMKilled) do not count, so a reboot does not look like a crash loop.
- 提示Kubernetes pod stuck pendingA pod has been Pending for fifteen minutes: no node can take it, or a volume or an image it needs does not come.
- 提示Kubernetes pod restartingA pod restarted more than five times in the last hour without being in CrashLoopBackOff (increase_prometheus(dumbmonit_k8s_pod_restarts[1h]) unless on (target, namespace, pod) max_over_time(dumbmonit_k8s_pod_crashlooping[10m]) > 0).
- 提示Kubernetes workload missing replicasA Deployment, StatefulSet or DaemonSet has fewer available replicas than it asks for.
再显示 2 条
- 提示Kubernetes volume claim pendingA PersistentVolumeClaim that a pod needs, or that a provisioning error concerns, is still not bound.
- 警告Monitor unreachableNo measurement received for more than three minutes. bottomk by (target) (1, time() - tlast_over_time(dumbmonit_up[7d]))
设置步骤: Create a read-only service account for DumbMonit
Create the service account.
kubectl create serviceaccount dumbmonit -n kube-system
Create a ClusterRole that can only get and list what DumbMonit reads.
kubectl create clusterrole dumbmonit-read --verb=get,list --resource=nodes,pods,persistentvolumeclaims,events,deployments.apps,statefulsets.apps,daemonsets.apps
Bind the role to the service account.
kubectl create clusterrolebinding dumbmonit-read --clusterrole=dumbmonit-read --serviceaccount=kube-system:dumbmonit
Create a token that does not expire, then print the token and the cluster CA in base64.
echo '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"dumbmonit-token","namespace":"kube-system","annotations":{"kubernetes.io/service-account.name":"dumbmonit"}},"type":"kubernetes.io/service-account-token"}' | kubectl apply -f -
kubectl -n kube-system get secret dumbmonit-token -o jsonpath='{.data.token}' | base64 -d
kubectl -n kube-system get secret dumbmonit-token -o jsonpath='{.data.ca\.crt}'
In DumbMonit, enter the API server address (for example https://k3s.lan:6443), paste the token as Service account token and the CA as Cluster CA certificate.
注意. On k3s, run each command as sudo k3s kubectl. The API server certificate only lists its own names and addresses: use one it lists, or tick Accept an unverifiable certificate.
文档