Skip to content

enhance precheck reachability, fix iDRAC deployment, and add credential cleanup - #5201

Merged
priti-parate merged 4 commits into
dell:issue-4849-omnia-modernizationfrom
Kratika-P:new_issue-4849-omnia-modernization
Sep 10, 2026
Merged

enhance precheck reachability, fix iDRAC deployment, and add credential cleanup#5201
priti-parate merged 4 commits into
dell:issue-4849-omnia-modernizationfrom
Kratika-P:new_issue-4849-omnia-modernization

Conversation

@Kratika-P

@Kratika-P Kratika-P commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Issues Resolved by this Pull Request
Fixes: #4849

PR Description

Description of the Solution

Summary: This PR addresses three key improvements to the telemetry system: (1) enhances precheck validation with comprehensive node reachability checks to prevent late-stage deployment failures, (2) fixes iDRAC deployment issues with CSV path parameters, and (3) implements credential cleanup functionality to securely remove sensitive credential data when corresponding deployments no longer exist.

Changes

Precheck Enhancements

  • Added new check_k8s_node_reachability.yml task to validate SSH reachability for all K8s nodes from cluster inventory
  • Enhanced check_ldms_slurm_nodes.yml with network reachability checks for all Slurm node types (control, compute, login, login_compiler)
  • Updated precheck documentation to reflect new reachability validation steps
  • Integrated K8s node reachability check into precheck playbook and main task file

iDRAC Deployment Fixes

  • Fixed missing csv_path parameter in update_bmc_group_entry module calls in initiate_telemetry_service_cluster.yml
  • Fixed missing csv_path parameter in update_bmc_group_entry module calls in retry_unreachable_bmcs.yml
  • Changed from undefined telemetry_data_path variable to properly defined bmc_group_data_filename

Variable Fixes

  • Fixed undefined variable issues in LDMS validation success messages by adding missing login_count, login_compiler_count, and total_count variables

Credential Cleanup Functionality

  • Added credential file path and vault key path configuration variables to cleanup role
  • Added component-to-credential-field mapping for idrac, ldms, powerscale, ufm, vast
  • Created new cleanup_credentials.yml task file for credential cleanup logic
  • Integrated credential cleanup after each component cleanup in cleanup_sources.yml
  • Added complete credential file deletion in finalize.yml for full cleanup mode
  • Updated role documentation in main.yml

Credential Cleanup Logic:

  • Complete cleanup mode: Deletes credential file and vault key when all components are removed
  • Individual cleanup mode: Decrypts credential file, blanks component-specific fields, re-encrypts
  • Preserves credentials for still-deployed components during individual cleanup
  • Handles missing vault key gracefully with appropriate logging

Files Changed

File Change Type Description
src/telemetry/roles/precheck/tasks/check_k8s_node_reachability.yml Added New task for K8s node SSH reachability validation
src/telemetry/roles/precheck/tasks/check_ldms_slurm_nodes.yml Modified Added Slurm node reachability checks and fixed variable issues
src/telemetry/roles/precheck/tasks/main.yml Modified Integrated K8s node reachability check
src/telemetry/playbooks/precheck/precheck.yml Modified Updated documentation and added K8s reachability check
src/telemetry/roles/deploy_idrac_telemetry/tasks/initiate_telemetry_service_cluster.yml Modified Fixed CSV path parameter in BMC validation
src/telemetry/roles/deploy_idrac_telemetry/tasks/retry_unreachable_bmcs.yml Modified Fixed CSV path parameter in retry BMC validation
src/telemetry/roles/cleanup/vars/main.yml Modified Added credential paths and component field mapping
src/telemetry/roles/cleanup/tasks/cleanup_credentials.yml Added Core credential cleanup logic (decrypt/blank/encrypt)
src/telemetry/roles/cleanup/tasks/cleanup_sources.yml Modified Integrated credential cleanup after each component
src/telemetry/roles/cleanup/tasks/finalize.yml Modified Added complete credential file deletion
src/telemetry/roles/cleanup/tasks/main.yml Modified Updated documentation

Testing

Precheck and iDRAC Fixes:

  • Verified precheck successfully detects unreachable K8s nodes with wrong IPs and fails early with clear error messages
  • Verified precheck passes with correct K8s node IPs
  • Verified Slurm node reachability checks work correctly
  • Verified iDRAC deployment completes successfully with fixed CSV path parameters
  • Verified full telemetry deployment pipeline completes successfully (Kafka, VictoriaMetrics, VictoriaLogs, iDRAC, LDMS)

Credential Cleanup:

  • Verified YAML syntax for all modified files
  • Tested credential field mapping configuration
  • Verified individual component cleanup logic for blanking specific fields
  • Verified complete cleanup logic for file deletion

Backward Compatibility

Precheck and iDRAC Fixes:

  • No breaking changes - all enhancements are additive
  • Existing precheck behavior preserved with additional validation steps
  • Fixed CSV path issues resolve runtime errors without changing module behavior
  • Variable fixes prevent template rendering failures

Credential Cleanup:

  • No breaking changes to existing cleanup functionality
  • Credential cleanup is additive - existing cleanup flows work unchanged
  • Components without stored credentials (OME, Kafka, VictoriaMetrics, VictoriaLogs) are unaffected
  • Missing credential file or vault key is handled gracefully with appropriate logging

Suggested Reviewers

@priti-parate @abhishek-sa1

@Kratika-P
Kratika-P marked this pull request as ready for review September 10, 2026 09:04
@abhishek-sa1

abhishek-sa1 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

@Kratika-P please address the github action failures

@Kratika-P Kratika-P changed the title Add credential cleanup to telemetry cleanup role enhance precheck reachability, fix iDRAC deployment, and add credential cleanup Sep 10, 2026
Kratika-P and others added 4 commits September 10, 2026 15:45
@Kratika-P
Add credential cleanup to telemetry cleanup role
2f454ff 
- Delete credential file during complete cleanup
- Blank component-specific credentials during individual cleanup
- Add credential field mapping for idrac, ldms, powerscale, ufm, vast

Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
@Kratika-P
fixing ansible lint for cleanup credentials
4187f07 
Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
@Kratika-P @devin-ai-integration
Fix telemetry precheck and iDRAC deployment issues
323dd56 
- Add K8s node reachability validation to precheck phase
- Enhance Slurm node reachability checks in precheck
- Fix CSV path parameter in update_bmc_group_entry module calls
- Fix undefined variable issues in LDMS validation messages
- Update precheck documentation to reflect new reachability checks

This prevents late-stage deployment failures by validating node reachability
early in the precheck phase, providing clear error messages and troubleshooting
steps when nodes are unreachable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
@Kratika-P
individual component cleanup code change
ed24062 
Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
@Kratika-P
Kratika-P force-pushed the new_issue-4849-omnia-modernization branch 3 times, most recently from d29e76d to ed24062 Compare September 10, 2026 11:11
@priti-parate
priti-parate merged commit d383e29 into dell:issue-4849-omnia-modernization Sep 10, 2026
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@priti-parate priti-parate priti-parate approved these changes
@balajikumaran-c-s balajikumaran-c-s balajikumaran-c-s approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Kratika-P @abhishek-sa1 @priti-parate @balajikumaran-c-s