2026SMS InfoComm

NVDebug Auto-Rename

Automated GB200/GB300 fault isolation

Bash
PowerShell
Redfish
IPMI
SSH
jq

At a glance

Automates collecting a fresh diagnostic log from a GB200/GB300 compute tray, then analyzes the bundle to identify which GPU compute board failed.
Renames each log to a self-describing filename (system serial, board serial, fault category) so RMA logs are triageable at a glance instead of anonymous archives.
Cross-platform launchers, a double-click Windows batch + PowerShell and a Mac/Linux Bash script, ship a pure-Bash worker to the management jump host and run it over one-time passwordless SSH.
Multi-route isolation engine: maps management event-log entries to fault categories, then attributes the failure to a specific board via power-rail decode, per-event device mapping, boot-status logs, and power-state asymmetry, printing a numbered step trace so no route is hidden.
Freshness guards refuse to rename on a stale or wrong-unit bundle (timestamp, serial, and completeness checks), preventing a mislabeled board from being pulled for RMA.
Validated end-to-end against a mock harness (synthetic BMC/collector output) and real field bundles across single- and dual-board fault scenarios.

A field automation tool that turns GB200/GB300 diagnostic collection into a one-command, self-labeling workflow: collect a fresh log from a failed compute tray, automatically identify the bad board, and rename the log so it's triageable the moment it lands.

Purpose

When a GB200/GB300 compute tray fails on the repair floor, a technician collects a large diagnostic bundle from the tray's management controller. On its own that bundle is an anonymous archive: someone still has to open it, read the event logs, and reason about which of the tray's GPU compute boards actually failed before an RMA can move. That manual triage is slow and easy to get wrong. I built this tool to collapse the whole loop into a single action: collect a fresh log, analyze it automatically, and rename the file to encode the answer.

Goals

Collect fresh, not stale. Always capture a new log after clearing the management event log and powering the unit on, so the bundle reflects the current fault rather than months-old noise.
Identify the failing board automatically from the telemetry inside the bundle, no human log-reading.
Self-describing output. Rename the archive to encode the system serial, board serial, and fault category so any downstream reviewer knows what it is without opening it.
Zero-friction for technicians. Double-click launcher, pick a rack/slot location, done. No PowerShell, SSH, or IP knowledge required.

Development process

Offline analysis first. I built the fault-isolation logic as a standalone offline analyzer (a sibling project) that reads the JSON captured inside an existing bundle, then ported that same engine into this live tool so collection and analysis share one code path.
Layered, transparent worker. The heavy lifting is a single pure-Bash worker that runs on the management jump host using standard tools (ipmitool, curl, jq, awk, unzip). Every management command and its raw output is echoed to the operator: nothing is hidden behind a spinner.
Multi-route isolation. A single fault can be attributable several ways, so the engine runs an ordered set of routes (power-rail decode, per-event device attribution, boot-status logs, critical-entry scoring, and power-state asymmetry) and prints a numbered step trace reporting each route's result, found or not.
Robustness guards. Real collectors occasionally hand back a previous or incomplete capture, so I added freshness checks (bundle timestamp vs. the live log, serial match, completeness) that refuse to rename and keep the file for manual review rather than risk mislabeling a board.
Cross-platform delivery. Windows technicians get a double-click batch that clears execution-policy and downloaded-file friction; Mac/Linux users get a Bash launcher over a single multiplexed SSH session. Both install a one-time SSH key so every later run is password-free.