Repository navigation
feat(firmware): persistent on-node log and coredump-to-flash - #1878
Merged
Merged
Conversation
The 4MB and display layouts have 1.875MB and 2MB OTA slots, sized for 4MB and 8MB parts. A 16MB C6 (DevKitC-1 N16) has no layout in the tree at all, so building for one meant hand-editing sdkconfig -- and a smaller table written over a deployed 16MB node relocates NVS and destroys its provisioning, which is why this belongs in the repo rather than in a workstation's untracked sdkconfig. Two 4MB OTA slots, plus a 64K coredump partition and an 8000K FAT volume for on-node storage. Rollback is enabled so an OTA'd image that cannot confirm itself is reverted by the bootloader. Layered, not inherited: sdkconfig.defaults.esp32c6 keeps its 4MB layout so the common 4MB devkit is never silently repartitioned. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PVWMiHQifoYXL7uL3bphrZ
A remote fault currently leaves no post-mortem. main.c decodes the reset reason at boot and logs it to the CONSOLE -- which is exactly what a power cycle destroys, and a power cycle is how a wedged node gets recovered. The flash was already provisioned for this and the features were simply off: partitions_16mb.csv has reserved a 64K `coredump` partition and an 8000K `storage` FAT volume since the 16MB layout was written, neither had ever been mounted or written, and the build defaulted to CONFIG_ESP_COREDUMP_ENABLE_TO_NONE. - node_log.c/h: a 128 B-record ring on the storage volume. Boot records (reset reason + the previous session's final uptime + a health snapshot), periodic health samples, and event records. Ordering is boot_id + uptime_s because these boards have no RTC and nothing can be timestamped absolutely. Counters are stored RAW and differenced at read time, so a cadence change never invalidates old records. Retention, not wear, binds: 128 B every 300 s is ~208 days in 8 MB, and the write rate is bounded explicitly. - ap_rssi_dbm, the association link from esp_wifi_sta_get_ap_info. The adaptive controller's rssi_median_dbm is a CSI statistic, not the uplink, so a node with a bad uplink currently shows no signal for it anywhere. - CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH, complementary rather than a substitute: a coredump catches a CRASH, the boot record catches a HANG followed by a watchdog reset. node_log_init failing is non-fatal by design -- the module disables itself and every entry point becomes a no-op. A logging volume that will not mount must never take the fleet down with it. csi_collector gains two gate accessors. This build selects its frame gate at compile time, so both return CSI_GATE_NOT_CONFIGURABLE rather than a plausible number: the log keeps a field of fixed width and meaning either way, and an explicit "not applicable" is stored instead of something a reader would later reason from as a real setting. Built with ESP-IDF v5.4 for esp32c6 using defaults + .esp32c6 + .16mb. Verified from the resulting artifacts: 16MB table with coredump and storage present, COREDUMP_ENABLE_TO_FLASH set, rollback enabled, image 0x10abd0 bytes in a 4M slot. NOT flashed to hardware from this branch -- the equivalent code was exercised on a node-3 pilot, but that is not evidence for this build. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PVWMiHQifoYXL7uL3bphrZ
The log engine lands with no consumer, so the access pattern was open. Records the decision to pull a node's log only when a human asks -- after a hang or a crash, or while troubleshooting that node -- and to keep nothing server-side. The reasoning that decided it: the ring already survives a power cycle on the node (~208 days of retention), so a server copy duplicates a durability property that is already met; and airtime is this fleet's binding constraint, measured at 91% consumed by two slow nodes, so a periodic drain across nine boards would spend exactly the resource whose exhaustion you would open the log to investigate. An instrument must not perturb what it measures. Leaves the decode-where question open on purpose, with the argument for a raw proxy stated: decoding server-side defines the record layout in both C and Rust, and the pilot already paid once for a host/firmware disagreement about a field's meaning. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PVWMiHQifoYXL7uL3bphrZ
# Conflicts: # firmware/esp32-csi-node/sdkconfig.defaults.16mb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A remote fault currently leaves no post-mortem.
main.cdecodes the reset reason at boot and logs it to the console — exactly what a power cycle destroys, and a power cycle is how a wedged node gets recovered.The flash was already provisioned for this and the features were simply off:
partitions_16mb.csvhas reserved a 64Kcoredumppartition and an 8000KstorageFAT volume since the 16MB layout was written; neither had ever been mounted, and the build defaulted toCONFIG_ESP_COREDUMP_ENABLE_TO_NONE.node_log.c/h— a 128 B-record ring on the storage volume: boot records (reset reason + the previous session's final uptime + a health snapshot), periodic health samples, and event records. Ordered byboot_id+uptime_sbecause these boards have no RTC. Counters are stored raw and differenced at read time, so a cadence change never invalidates old records. Retention, not wear, is the binding constraint: 128 B every 300 s is ~208 days in 8 MB, and the write rate is bounded explicitly.ap_rssi_dbm— the association link fromesp_wifi_sta_get_ap_info. The adaptive controller'srssi_median_dbmis a CSI statistic, not the uplink, so a node with a failing uplink currently surfaces that nowhere.CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH— complementary, not a substitute: a coredump catches a crash; the boot record catches a hang followed by a watchdog reset.Failure behaviour
node_log_init()failing is non-fatal by design — the module disables itself and every entry point becomes a no-op. A logging volume that will not mount must never take a node down with it.The two gate accessors
node_log's health record carriesgate_mode/gate_seq_period. This tree selects its frame gate at compile time, so both accessors returnCSI_GATE_NOT_CONFIGURABLErather than a plausible number. The record keeps a field of fixed width and meaning either way, and an explicit "not applicable" is stored instead of a value a reader would later reason from as a real setting. If runtime gate selection lands later, these get real implementations with no change to the record format.Not included here
The
GET /logandGET /coredumpHTTP endpoints (and the host-side decoder) depend on the config API from #1827 and will follow once that merges. Until then the log is written but only readable over USB.Test plan
esp32c6,defaults + .esp32c6 + .16mb, 1434/1434partitions_16mb.csvin force,coredump+storagepresent in the generated table,COREDUMP_ENABLE_TO_FLASH=y,WL_SECTOR_SIZE_4096=y, rollback enabled, image0x10abd0in a 4M slot (74% free)🤖 Generated with claude-flow
https://claude.ai/code/session_01PVWMiHQifoYXL7uL3bphrZ