Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- **Scale-down now actually drops the removed instances' series.**
`RemoveInstanceMetrics` matched every metric on the `process` label, but
`cbox_init_process_up`, `_start_time_seconds` and `_last_exit_code` carry
the process name under `name`. A partial match on a label a metric does
not have deletes nothing, so after scaling php-fpm from 10 to 2,
`cbox_init_process_up{instance="php-fpm-9"}` sat at 0 forever and
`process_up == 0` alerts fired on instances scaled away on purpose. Each
metric is now matched on its own label key. `cbox_init_process_threads`,
`_file_descriptors` and `cbox_init_resource_collection_errors_total` are
now dropped as well; they were never in the list. `RemoveProcessMetrics`
had the same mismatch and missed the health check histogram and counter
and the desired-scale gauge. No label names changed.

### Documentation

- The metrics reference now lists the exact label set of every
`cbox_init_` metric, explains `name` vs `process`, and notes that
Prometheus' default scrape config renames the `instance` label to
`exported_instance`. Examples that queried lifecycle metrics with
`process=`, or used metrics that do not exist
(`cbox_init_manager_uptime_seconds`, `cbox_init_hook_execution_seconds`,
`cbox_init_hook_failures_total`), are corrected.

## [3.7.0] - 2026-09-11

### Changed
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ spec:
# - cbox_init_process_up
# - cbox_init_process_restarts_total
# - cbox_init_health_check_status
# - cbox_init_manager_uptime_seconds
# - cbox_init_manager_start_time_seconds
```

## Resource Planning
Expand Down
6 changes: 3 additions & 3 deletions docs/examples/laravel-complete.md
Original file line number Diff line number Diff line change
Expand Up @@ -732,9 +732,9 @@ CBOX_INIT_PROCESS_QUEUE_DEFAULT_SCALE=10
curl http://localhost:9090/metrics

# Key metrics:
# - cbox_init_process_up{process="php-fpm"}
# - cbox_init_process_restarts_total{process="nginx"}
# - cbox_init_health_check_status{process="horizon"}
# - cbox_init_process_up{name="php-fpm"}
# - cbox_init_process_restarts_total{name="nginx"}
# - cbox_init_health_check_status{name="horizon"}
```

### Health Status
Expand Down
47 changes: 24 additions & 23 deletions docs/examples/laravel-with-monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,13 @@ processes:
curl http://localhost:9090/metrics

# Key metrics:
cbox_init_manager_uptime_seconds # Manager uptime
cbox_init_process_up{process="nginx"} # Process status (1=up, 0=down)
cbox_init_process_restarts_total{process="*"} # Restart count
cbox_init_health_check_status{name="*",type="*"} # Health check status
cbox_init_process_start_time{process="*"} # Process start timestamp
cbox_init_hook_execution_seconds{hook="*"} # Hook execution time
cbox_init_manager_start_time_seconds # Manager start (uptime = time() - this)
cbox_init_process_up{name="nginx"} # Process status (1=up, 0=down), per instance
cbox_init_process_restarts_total{name="nginx"} # Restart count, by reason
cbox_init_health_check_status{name="nginx"} # Health check status, by type
cbox_init_process_start_time_seconds{name="nginx"} # Instance start timestamp
cbox_init_hook_duration_seconds{name="migrate"} # Hook execution time (name = hook name)
cbox_init_process_memory_bytes{process="nginx"} # Resource metrics use process=, not name=
```

### Grafana Dashboard
Expand All @@ -130,21 +131,21 @@ datasources:

**Example Queries:**
```promql
# Process uptime
cbox_init_manager_uptime_seconds
# Manager uptime in seconds
time() - cbox_init_manager_start_time_seconds

# Total restarts (all processes)
sum(cbox_init_process_restarts_total)

# Unhealthy processes
count(cbox_init_health_check_status == 0)

# Processes by state
count by (state) (cbox_init_process_up)
# Running instances per process
sum by (name) (cbox_init_process_up)

# Hook execution duration
rate(cbox_init_hook_execution_seconds_sum[5m]) /
rate(cbox_init_hook_execution_seconds_count[5m])
rate(cbox_init_hook_duration_seconds_sum[5m]) /
rate(cbox_init_hook_duration_seconds_count[5m])
```

### Prometheus Alerts
Expand All @@ -161,8 +162,8 @@ groups:
labels:
severity: critical
annotations:
summary: "Process {{ $labels.process }} is down"
description: "{{ $labels.process }} has been down for 1 minute"
summary: "Process {{ $labels.name }} is down"
description: "{{ $labels.name }} has been down for 1 minute"

# Excessive restarts
- alert: FrequentRestarts
Expand All @@ -171,7 +172,7 @@ groups:
labels:
severity: warning
annotations:
summary: "Process {{ $labels.process }} restarting frequently"
summary: "Process {{ $labels.name }} restarting frequently"

# Unhealthy process
- alert: ProcessUnhealthy
Expand All @@ -180,15 +181,15 @@ groups:
labels:
severity: warning
annotations:
summary: "Process {{ $labels.process }} is unhealthy"
summary: "Process {{ $labels.name }} is unhealthy"

# Hook failures
- alert: HookFailed
expr: cbox_init_hook_failures_total > 0
expr: increase(cbox_init_hook_executions_total{status="failure"}[5m]) > 0
labels:
severity: warning
annotations:
summary: "Hook {{ $labels.hook }} failed"
summary: "Hook {{ $labels.name }} failed"
```

## Management API
Expand Down Expand Up @@ -572,9 +573,9 @@ receivers:
"title": "Cbox Init - Laravel Monitoring",
"panels": [
{
"title": "Process Uptime",
"title": "Manager Uptime",
"targets": [{
"expr": "cbox_init_manager_uptime_seconds"
"expr": "time() - cbox_init_manager_start_time_seconds"
}]
},
{
Expand All @@ -598,7 +599,7 @@ receivers:
{
"title": "Hook Execution Time",
"targets": [{
"expr": "rate(cbox_init_hook_execution_seconds_sum[5m]) / rate(cbox_init_hook_execution_seconds_count[5m])"
"expr": "rate(cbox_init_hook_duration_seconds_sum[5m]) / rate(cbox_init_hook_duration_seconds_count[5m])"
}]
}
]
Expand Down Expand Up @@ -853,11 +854,11 @@ global:
```yaml
# ❌ Bad - scheduled tasks restart by design
- alert: SchedulerRestarted
expr: cbox_init_process_restarts_total{process="scheduler"} > 0
expr: cbox_init_process_restarts_total{name="scheduler"} > 0

# ✅ Good - only alert on unexpected restarts
- alert: UnexpectedRestart
expr: cbox_init_process_restarts_total{process!~"scheduler|.*-task"} > threshold
expr: cbox_init_process_restarts_total{name!~"scheduler|.*-task"} > threshold
```

**Don't expose API publicly:**
Expand Down
15 changes: 8 additions & 7 deletions docs/features/process-scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,15 +323,16 @@ server {

```bash
# Count running instances
count(cbox_init_process_up{process=~"queue-default-.*"})
sum(cbox_init_process_up{name="queue-default"})

# Per-instance uptime
cbox_init_process_up{process="queue-default-1"}
cbox_init_process_up{process="queue-default-2"}
cbox_init_process_up{process="queue-default-3"}
# Per-instance status (instance IDs count from 0; Prometheus stores them as
# exported_instance by default, see the metrics label reference)
cbox_init_process_up{name="queue-default", exported_instance="queue-default-0"}
cbox_init_process_up{name="queue-default", exported_instance="queue-default-1"}
cbox_init_process_up{name="queue-default", exported_instance="queue-default-2"}

# Total restarts across all instances
sum(cbox_init_process_restarts_total{process=~"queue-default-.*"})
# Total restarts (counted per process, not per instance)
sum(cbox_init_process_restarts_total{name="queue-default"})
```

### Management API
Expand Down
8 changes: 4 additions & 4 deletions docs/features/restart-policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,10 +284,10 @@ processes:

```bash
# Total restarts per process
cbox_init_process_restarts_total{process="php-fpm"}
cbox_init_process_restarts_total{name="php-fpm"}

# Restart rate (restarts per second)
rate(cbox_init_process_restarts_total{process="php-fpm"}[5m])
rate(cbox_init_process_restarts_total{name="php-fpm"}[5m])
```

### Alert on Excessive Restarts
Expand All @@ -303,15 +303,15 @@ groups:
labels:
severity: warning
annotations:
summary: "Process {{ $labels.process }} restarting frequently"
summary: "Process {{ $labels.name }} restarting frequently"

- alert: RestartLoop
expr: cbox_init_process_restarts_total > 10
for: 1m
labels:
severity: critical
annotations:
summary: "Process {{ $labels.process }} in restart loop"
summary: "Process {{ $labels.name }} in restart loop"
```

## Troubleshooting
Expand Down
Loading
Loading