Skip to content
slg74Public

About

hmc stats at a glance. prometheus hooks + grafana

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

power_AIX

Scripts for managing and monitoring IBM Power / AIX and PowerVM environments: LPAR provisioning from the HMC, AIX paging-space alerting, and a PowerVM quick-glance CLI with a Prometheus/Grafana integration.

Contents

File What it does
pvm_glance.py Command-line "quick glance" at a PowerVM managed system's health via pypowervm: VIOS/LPAR CPU saturation, memory pressure, and disk I/O. See below.
pvm_glance-rs/ Rust port of pvm_glance.py with the same flags/metrics/output modes, for when the Python interpreter + pypowervm import overhead matters (e.g. a cron job polling every minute). See pvm_glance-rs/README.md for what's verified vs. not.
aix_pgspace_monitor.sh Polls AIX paging space utilization and errpt for low-paging-space kill events, writing Prometheus textfile-collector output for node_exporter. Run via cron every 1-5 min.
make_lpars.sh Bulk-provisions client LPARs on a Power system from the HMC CLI, driven by a lpar_config.csv (LPAR profile, dual-VIOS virtual SCSI adapters). Run from the HMC itself.
grafana-demo/ Self-contained Prometheus + Grafana demo of pvm_glance.py's --prom output, using synthetic data so it runs without real Power hardware. See grafana-demo/README.md.

pvm_glance.py

A pypowervm-based CLI that answers "how is this managed system doing right now" without reaching for topas/nmon/vmstat on each partition individually. It connects to an HMC or NovaLink over the PowerVM REST API and reports, for the managed system, its Virtual I/O Servers, and its client LPARs:

  • CPU saturation -- utilization against entitlement (%entc), computed from PCM processor-cycle deltas the same way topas/nmon derive it, with a configurable saturation threshold.
  • Memory pressure -- RMC-reported real-memory-free % and AIX VMM page-in/page-out rates (the actual paging signal, not just configured vs. assigned memory).
  • Disk I/O -- aggregate read/write throughput per VIOS and LPAR, reconciled across PHYP's and each VIOS's PCM samples (a client LPAR's vSCSI/VFC byte counters only exist on the backing VIOS's own sample, not the LPAR's).
  • Shared processor pool utilization -- the closest hypervisor-level analog to "system load" that PowerVM exposes.
  • WPARs (optional, --wpar) -- AIX Workload Partitions aren't modeled by the PowerVM API at all, so this is a best-effort SSH + lswpar probe against each running AIX LPAR, clearly separated from the PowerVM-native data.

Output modes

  • Default: a human-readable table, with saturated/pressured values highlighted.
  • --json: machine-readable JSON of the same raw metrics.
  • --prom: Prometheus text exposition format, either printed to stdout or, combined with --textfile-dir, atomically written for node_exporter's textfile collector (same pattern as aix_pgspace_monitor.sh) -- see grafana-demo/ for a working Prometheus + Grafana pipeline built on this.

Usage

pip install pypowervm

# Human-readable glance
pvm_glance.py --host hmc.example.com --user hscroot

# A specific managed system, JSON output
pvm_glance.py --host hmc.example.com --user hscroot \
    --system Server-8408 --json

# Include AIX WPARs (requires working SSH key auth to the LPARs)
pvm_glance.py --host hmc.example.com --user hscroot --wpar

# Cron-friendly: Prometheus textfile collector output
pvm_glance.py --host hmc.example.com --user hscroot --prom \
    --textfile-dir /var/lib/node_exporter/textfile_collector

Password comes from --password, then the PVM_PASSWORD environment variable, then an interactive prompt -- unattended runs (cron) fail fast instead of hanging if neither is set and stdin isn't a terminal.

Run pvm_glance.py --help for the full flag list.

Rust port

pvm_glance-rs/ is a from-scratch Rust implementation of the same tool -- same flags, same metrics, same --json/--prom output -- for cron-heavy deployments where Python's interpreter startup and pypowervm's oslo.* import chain (~750ms, ~58MB peak RSS measured here) actually costs something running every minute or two. The Rust build starts in ~10ms and peaks around 2MB RSS. It has not been run against a live HMC/NovaLink -- see its README for exactly what is and isn't verified before trusting it in production.

Seeing it in Grafana

grafana-demo/ stands up Prometheus + Grafana with a dashboard built entirely from pvm_glance.py's own metric names, fed by a synthetic exporter that calls the script's real _render_prometheus() renderer so the data shape matches a real run exactly. Swapping the synthetic exporter for a real pvm_glance.py --prom deployment requires no dashboard changes.

pvm_glance Grafana dashboard

System overview (proc units, memory, shared-pool busy %, WPAR status), CPU saturation vs. entitlement, memory pressure, and disk I/O -- all populated from pvm_glance's Prometheus output. The pool utilization gauge and CPU panels are caught here mid-swing through a synthetic saturation spike, showing the threshold coloring in action.

About

hmc stats at a glance. prometheus hooks + grafana

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages