2026SMS InfoComm
NVDebug Analyzer
Offline GB200/GB300 log triage engine
- Bash
- jq
- awk
- Redfish
- JSON
At a glance
- Offline engine that reads the telemetry JSON inside a diagnostic archive and identifies which GPU compute board failed, no live hardware needed.
- Classifies faults from management event-log entries into severity-ordered categories, then attributes each to a specific board and renames the archive accordingly.
- Dry-run by default: prints a CSV triage report over a whole directory of logs and changes nothing until explicitly told to apply.
- Batch triage across an archive of historical logs, with filters to target a specific fault or force a category, and a flag to skip clean logs.
- Distinguishes a single-board failure from a shared, tray-level fault that implicates no single board: the same isolation logic later ported into the live collection tool so offline and on-hardware analysis stay consistent.
The offline brain behind the fault-isolation workflow: a Bash engine that reads the JSON buried inside a diagnostic archive and answers “which board failed?” without touching the hardware, then batch-triages a whole folder of logs into a CSV report.
Purpose
Diagnostic archives from failed GB200/GB300 trays pile up: attached to tickets, dropped in shared folders, collected weeks ago. Answering “which compute board is bad?” for each one meant a person opening the archive and reading event logs by hand. This tool automates that reasoning entirely offline: point it at a directory of archives and it parses the captured telemetry, classifies the fault, isolates the board, and produces a triage report: renaming the files to self-describing names on request.
Goals
- Work purely from what's already captured. Parse the telemetry JSON inside the archive rather than querying live hardware, so historical and remote logs triage the same way.
- Safe by default. Default to a dry run that only reports; never rename until explicitly told to.
- Explainable. Every categorization and board attribution is traceable to the specific evidence that produced it.
Development process
- Classification, then attribution. The engine first maps event-log entries to a severity-ordered set of fault categories (a log can match several, joined into a compound label), then runs isolation routes to attribute the fault to a specific board.
- Multi-route isolation. Power-rail/module decode, per-event device mapping, boot-status logs, and scoring of critical entries each get a shot; the tool distinguishes a single-board failure from a shared/tray-level fault that implicates no single board.
- Batch-oriented CLI. Built to process a directory of archives at once, emitting a CSV report; flags let an operator filter by fault, force a category, skip clean logs, or apply renames only to high-confidence rows.
- Reuse. Once the logic was solid, I ported the same engine into the live collection tool so both share one source of truth for how a fault maps to a board.