CommInsight

CommInsight is an observability and semantic-analysis toolkit for collective communication in distributed AI workloads. It captures communication events inside the training process, preserves raw evidence outside the hot path, and recovers higher-level training semantics such as Steps and communication roles.

The current source release contains five independently usable components:

Component	Purpose
NCCL Profiler	Captures Collective and P2P lifecycle, timing, topology, communicator metadata, Stream IDs, and timeout evidence through the NCCL Profiler ABI.
Agent	Consumes Profiler SHM and FIFO data, exposes runtime APIs and Metrics, runs Hang diagnostics, and archives raw and derived data.
Semantic Engine	Recovers Step boundaries and summaries from communication sequences with template-driven FSMs and resumable checkpoints.
Inject	Passes PyTorch process-group descriptions to NCCL commName without requiring a patched PyTorch build.
Server	Provides workload discovery, monitoring, offline semantic reconstruction, Hang/Failure analysis, Straggler attribution, communication-performance analysis, and Perfetto visualization.

The design keeps training safety as the primary boundary. Components are isolated so that observability failures degrade the affected capability first, while the Profiler avoids waiting for the Agent and preserves raw inspector_logs whenever possible.

Whitepaper

• CommInsight Product Whitepaper v2.0 (Chinese)
• CommInsight Product Whitepaper v2.0

Architecture

Click the image to open the editable draw.io source.

The five components shown in the architecture are included in this source release and can be deployed independently or combined as needed.

Data flow

1. CommInsight Inject is installed before torchrun starts and propagates <group_desc>:<group_name> into NCCL commName when supported.
2. NCCL loads libnccl-profiler-comminsight.so and reports communication lifecycle events through its Profiler ABI.
3. The Profiler writes bounded real-time data to SHM and durable raw records to segmented inspector_logs. It can start the local Agent automatically.
4. The Agent consumes SHM/FIFO data, generates Metrics and Hang evidence, runs the Semantic Engine, and archives sealed files through a Provider.
5. The Semantic Engine writes per-GPU Step summaries and checkpoints, allowing parsing to resume after process or Pod restarts.

Requirements

• Linux
• Go 1.23 or newer
• Python 3.8 or newer
• A C++11 compiler
• CUDA Toolkit headers under /usr/local/cuda, or CUDA_HOME set explicitly
• NCCL with Profiler Plugin ABI v4 or v5 support

CommInsight depends on CUDA Runtime headers at build time and the CUDA Runtime at run time, both of which must be provided by the user's environment. The CUDA Toolkit is not distributed with this repository. Users who enable CUDA functionality must install it separately and comply with the NVIDIA CUDA Toolkit license terms.

Quick start

Run the following commands from the repository root. Both the default installation directory and runtime data directory are /opt/comminsight.

1. Build

The collection components require Linux, G++/make, CUDA Runtime headers, Python/pip, and Go. The default CUDA location is /usr/local/cuda. The Server additionally requires Node.js 20/npm and a Go toolchain that satisfies server/backend/go.mod (Go 1.26 is recommended).

make
# If CUDA is installed elsewhere:
# make CUDA_HOME=/path/to/cuda

The default build produces the Agent, NCCL Profiler, and Inject, but not the Server:

dist/comminsight-agent
dist/comminsight_inject-*.whl
profilers/nccl/libnccl-profiler-comminsight.so

2. Install the Profiler, Agent, and configuration

The minimal collection setup requires just these three files; the Semantic Engine is linked into the Agent:

/opt/comminsight/
├── lib/libnccl-profiler-comminsight.so
├── bin/comminsight-agent
└── etc/comminsight.conf

Install this minimal setup (write permission to the destination is required):

make install-agent install-profiler

These targets install the binaries and configuration into /opt/comminsight by default, without installing Inject or the Server. The configuration comes from config/comminsight.conf. Reinstalling overwrites it, so back up customized settings first.

3. Install Inject before starting the workload

Install Inject into the Python environment used by the workload, before starting Python or torchrun. Installing it in a precheck container or another virtual environment does not affect the workload environment. Already-running Python processes do not automatically acquire a newly installed hook.

python3 -m pip install --no-deps dist/comminsight_inject-*.whl
python3 -m comminsight_inject

If the directory contains multiple wheel versions, select a specific file. Alternatively, make install installs the Agent, Profiler, configuration, license, and Inject together; ensure that the Python selected by the Makefile is the workload's Python environment.

Inject uses a Python .pth file and a deferred import hook to propagate group_desc:group_name to NCCL commName, without modifying training code. Unsupported configurations do not block training, but may limit communication-role and Step identification.

4. Enable collection

For a default single-node deployment, set one environment variable before the existing workload launch command:

export NCCL_PROFILER_PLUGIN=/opt/comminsight/lib/libnccl-profiler-comminsight.so
# Then run the existing torchrun / training launch command.

The Profiler on local rank 0 starts the Agent automatically by default; do not start a duplicate manually. The Agent listens on port 19102 by default. Components read /opt/comminsight/etc/comminsight.conf, with environment variables taking precedence over file settings.

curl --noproxy '*' -fsS http://127.0.0.1:19102/healthz

Runtime data is written under /opt/comminsight in inspector_logs/, metadata/, semantics/step/, hang_detection/, shms/, fifos/, and logs/. The default archive interval is 60 seconds. Step recovery requires the stable window to advance; output is not immediate after startup.

Additional settings depend on the deployment:

Environment variable	When to use it
COMMINSIGHT_AGENT_SERVER_URL=http://server-host:8081	Enable real-time reporting from the Master Agent to the Server; use the machine API port, not the browser-facing port.
COMMINSIGHT_COMMON_ARCHIVE_MOUNT_PATH=/path/to/3fs	Use an actual shared-storage mount. The default is /opt/comminsight/archives; archives reside beneath it in comminsight/02_archives.
COMMINSIGHT_COMMON_ARCHIVE_TYPE=source-name	Select another archive Provider compiled into the binary; the default is the built-in 3fs Provider.
COMMINSIGHT_AGENT_MODE=master / slave	Assign each node a role in a multi-node deployment.
COMMINSIGHT_AGENT_ADVERTISE_URL=http://this-node:19102	Specify an Agent address reachable by other nodes.
COMMINSIGHT_AGENT_MASTER_URL=http://master-host:19102	Point a Slave at the Master Agent.
NODE_IP or HOST_IP	Record the actual Node IP. POD_IP and POD_NAME describe the Pod and must not be confused with Node information.

Set environment variables before starting the Agent or training processes. The deployment environment must mount 3FS; an existing directory does not imply that shared storage is mounted. The default path is suitable for a single-machine demonstration. Production deployments require shared storage or another Provider.

5. Start a local Server with one command

The Server keeps separate frontend and backend source trees. Start both for local development with:

make -C server dev

On the first run, the command installs frontend dependencies and builds the backend, then starts the Backend and Vite together:

• Open http://localhost:5173 in your browser.
• The Backend defaults to 127.0.0.1:8080; the machine API defaults to 127.0.0.1:8081.
• Development data defaults to server/.dev/data, preserving users, permissions, and caches. This directory is excluded from Git.
• The terminal prints the location of the one-time credential file, not its contents.
• Ctrl+C stops both services while preserving development data. Startup failures also clean up child processes launched by this command, without killing existing services.
• Without external Providers, only the built-in 3FS Provider is available. Prepare a directory first to build with custom Providers.

make -C server dev PROVIDERS_DIR=../comminsight-providers
# Or specify a data directory:
COMMINSIGHT_SERVER_DATA_DIR=/path/to/server-data make -C server dev

To change only the Vite port, pass DEV_PORT=5174. Resolve existing port conflicts first; the command does not automatically select another port or replace existing processes.

For production or standalone binary operation:

(cd server/frontend && npm ci)
make build-server
COMMINSIGHT_SERVER_DATA_DIR=/opt/comminsight/server-data ./dist/comminsight-server

Open http://localhost:8080. This builds the frontend and embeds it into the Go binary, so Vite is not needed. For remote access, configure COMMINSIGHT_SERVER_LISTEN and set COMMINSIGHT_SERVER_PUBLIC_URL to the URL used by browsers. Configure COMMINSIGHT_SERVER_AGENT_LISTEN separately for remote Agent reporting.

6. First sign-in and the 3FS path

After starting the Server for the first time, open the page:

1. Read the bootstrap-token file in the Server data directory. For one-command development:

   cat server/.dev/data/bootstrap-token

   With the standalone command above, the path is /opt/comminsight/server-data/bootstrap-token. If you set COMMINSIGHT_SERVER_DATA_DIR, read it from that directory instead. This credential is for initialization only; do not share it.
2. Enter the credential on the initialization page, set a root password, and confirm it. Passwords must be 12–72 bytes long.
3. Initialization invalidates and deletes the credential. Subsequently, sign in as root with your chosen password; initialization is not repeated.
4. Open System Settings → Default 3FS archive path, enter an absolute mount path, and save it. The Server reads <mount-path>/comminsight/02_archives. The setting persists across restarts.
5. Select 3fs in Archive Data to check access. By default, the Task Manager discovers workloads and refreshes their sizes and modification times every 30 seconds.

Changing the Server's path does not modify the Agent or other Providers, nor move or delete data. Configure the Agent's mount path separately. Both must access the same archives for workloads to appear. On one machine they can use the same absolute path; different machines may use different mount paths to the same underlying data.

Without an enterprise identity Provider compiled in, use local sign-in directly. With one available, users can choose enterprise sign-in manually; the local form also provides an SSO link. Root is the initialization account. External identities do not automatically receive administrator privileges.

7. Features and access boundaries

Page	Purpose
Workload list / details	Status, Step/Hang counts, nodes, communicators, and topology. Platform details require a Platform Provider.
Monitoring dashboard	Communication and device metrics independent of Step semantics; requires the corresponding TSDB data.
Semantic recovery	Offline Session pipeline, progress, semantic-data deletion, and StepID calibration with apply/clear actions.
Hang analysis	Cross-Rank evidence and dependency chains to locate where progress first stopped. Determine root causes using framework logs, stacks, and hardware metrics together.
Straggler / Perfetto	Step Duration, composite indices, causal waiting, communication performance, IB correlation, and traces.
System Settings	Background jobs, shared concurrency, and the default 3FS path. Root manages administrators.
About	Repository and whitepapers in English and Chinese.

Root and administrators can view all workloads. Regular users can access only their own workloads or those explicitly shared with them. Use HTTPS for the browser-facing production endpoint. The machine API on port 8081 has no user authentication and must remain on a trusted network, not exposed through the public browser endpoint.

Protocol-derived files under api/gen and generated NCCL SHM definitions are maintained by tools/codegen/generate.rb. After changing a protocol, regenerate and verify them with:

make generate
make verify-generated

Interfaces and protocols

• Architecture and component interfaces
• Protocol registry
• NCCL SHM ABI v15
• NCCL Inspector Log v4.9
• Semantic Engine ABI v1
• Step Summary v5
• Agent Master API
• Agent Master-Slave API

Repository layout

agent/                 runtime Agent and Profiler adapters
inject/                Python import hook
profilers/nccl/        NCCL Profiler Plugin
semantic-engine/step/  Step semantic recovery
protocols/             versioned ABI, Schema, Protobuf, and OpenAPI sources
api/gen/               generated protocol bindings
config/                runtime configuration
tools/codegen/         protocol code generation
draw.io/               architecture source

License

CommInsight is licensed under the Apache License 2.0. Third-party components, including NCCL-derived files and the bundled Perfetto UI, retain their own copyright and license notices. See LICENSE.txt, ThirdPartyNotices.txt, and the Perfetto UI notices for details.