Skip to content
Samar Siddiqui
All projects

2026SMS InfoComm

NVDebug Suite

Technician GUI for GB200/GB300 log collection

  • PowerShell
  • Windows Forms
  • Bash
  • SSH
  • IPMI
  • jq
failure categories, each with log guidance
21
power-cycle only when the category needs it
Full / Quick

At a glance

  • A one-screen Windows GUI that turns NVIDIA nvdebug log collection into scan, pick a failure category, and click Collect.
  • Orchestrates everything from the technician's PC over SSH: a pure-Bash worker runs collection and validation on the jump host that can reach the BMCs.
  • Cross-checks scanned chassis and compute-board serials against the unit's live FRU, so a typo blocks the upload instead of mislabeling a log.
  • Safe-by-default power handling: only categories explicitly marked for full collection are power-cycled; everything else captures current state.
  • Full-mode runs name the log after the fault actually found in the event logs, not just the category the technician picked.
  • Blocked logs are quarantined for manual review rather than discarded, with a Teams alert to engineering.
  • Per-category guidance panel tells technicians which collectors to open and what to validate.
  • Background ZIP download, a bulk log finder driven by an Excel list of serials, and daily and master CSV run reports.

A desktop front end for the repair floor's diagnostic workflow. A technician scans a unit, picks what went wrong, and gets a validated, self-describing log bundle without ever opening a terminal.

Purpose

Collecting an nvdebug bundle from a failed GB200/GB300 unit used to mean a terminal session on a jump host, a long command line, and a hand-typed filename. Every manual step was a chance to collect from the wrong slot, mistype a serial, or power-cycle a unit that should have been captured as-is. I built NVDebug Suite so the whole process fits on one screen and the risky decisions are made by the tool, not from memory.

Goals

  • One screen, top to bottom. Tech ID, chassis serial, rack location, board serials, failure category, description, then a single Collect button.
  • Never lose a log. Anything that fails validation is moved to a review folder with a REVIEW_ prefix instead of being deleted or mislabeled.
  • Safe by default. A unit is only power-cycled when its failure category explicitly calls for a full collection.
  • Configuration over code. Categories, guidance text, integrations and addresses live in config files, with secrets in a git-ignored local override.

Architecture

The PC side is a Windows Forms app in PowerShell, split into modules for SSH orchestration, background downloads, CSV reporting and integrations. The heavy lifting runs remotely: the GUI ships a single Bash worker to the jump host over SSH, and that worker drives nvdebug, reads FRU data over IPMI, parses the collected bundle with jq, and reports one structured result line back.

Technician PC (Windows)
  └─ NVDebugUploader.ps1     (Windows Forms GUI)
       ├─ Nvdebug.Ssh        background SSH job, live output streaming
       ├─ Nvdebug.Download   background ZIP transfer
       ├─ Nvdebug.Summary    daily + master CSV reports
       └─ Nvdebug.Webhook    Teams alert on blocked runs
            │  SSH (key auth, keepalives)
            ▼
Jump host
  └─ nvdebug_worker.sh       collect → validate FRU → attribute fault → rename
            │  BMC network
            ▼
GB200/GB300 unit

Development process

  • Two collection modes. Each category resolves to Full (auto power-on and board detection) or Quick (collect current state only). The label beside the dropdown tells the technician which one will run before they click.
  • Validation against live hardware. Scanned serials are compared with what the unit's FRU actually reports. An auto-fill button can read a board serial straight from the FRU when there is nothing to scan.
  • Fault attribution. In Full mode the worker reads the event logs inside the bundle, lists every critical entry in the status box, and names the file after the fault it found, falling back to the technician's choice when there is nothing to attribute.
  • Responsive while it waits. Collection takes several minutes, so it runs as a background job with output streamed live, a working Cancel that also stops the remote process, and copy or save buttons for the activity log.
  • Hardened against real hardware. Testing on production units surfaced quirks the documentation didn't mention, such as which FRU field actually holds a board serial, and idle SSH sessions being dropped during nvdebug's silent zip step. Both are handled and documented so they don't get “fixed” back.
  • Tested offline. The ZIP download path has a test suite that runs against a fake ssh, so it can be verified without network access.

Outcomes

  • Log collection went from a terminal procedure to a guided form technicians run in a couple of clicks.
  • Serial typos and wrong-slot collections are caught before a log is filed, and the log is kept for review instead of lost.
  • Every run, successful or not, is recorded in a daily and master CSV, giving leads a history of what was collected and why.

Tech stack

  • PowerShell 5.1 / 7: GUI, modules, launcher
  • Windows Forms: two-tab desktop UI with light and dark themes
  • OpenSSH: key-based SSH and SCP to the jump host
  • Bash, ipmitool, jq, awk: remote worker for collection, FRU reads and bundle parsing
  • Teams webhooks: alerts on blocked uploads