Super Stepper: Hybrid Debugging of Python‒to‒C++ Deep Learning Workflows

Abstract

We present Super Stepper, a hybrid runtime debugging system enabling synchronized introspection and step control across both Python and C++ layers in deep learning frameworks. Targeting PyTorch and HuggingFace Transformers, it integrates source-level breakpoints, symbolic stepping, and remote runtime triggers via the TranSymbolics protocol.

1. Motivation

Deep learning frameworks like PyTorch implement a Python API layered atop an optimized C++ backend. Python debuggers can trace control flow, but lose visibility when execution crosses into compiled or CUDA layers. Native debuggers lack Python-layer context. This creates a blind spot at the boundary.

2. Overview of Super Stepper

• Python-side introspection via forward/backward hooks and debugpy
• C++-side injection of symbolic breakpoints and SIGTRAP-style traps
• TranSymbolics messaging for synchronized stepping and logging

Each "step" is a symbolic pause at a transition point. Triggered by input, context, or controller.

3. Python Integration

• Hooks into nn.Module, forward(), generate()
• Access to inputs, outputs, hidden states
• Optional sys.settrace() control or debugpy backend

4. C++ Runtime Control

Insert SuperStepperPause() calls in compiled ops.

inline void SuperStepperPause() {  std::cout << "[SuperStepper] paused" << std::endl;  std::cin.get();}

• Trigger pause after matmul, layernorm, etc.
• Alternatives: signal, shared memory, socket control

5. TranSymbolics Protocol

• pause, continue, step, log, inject
• External tools coordinate hybrid-layer flow
• Can issue symbolic commands at runtime

6. Use Cases

• Debug tensor values mid-run
• Control inference by symbolic step
• Visualize model evolution or inject changes

7. Implementation Notes

• Build PyTorch with debug symbols
• Insert macros or interpose via preload
• Minimal changes needed to wrap core ops

8. Future Work

• JAX and TensorFlow support
• Graphical overlay and trace viewer
• Script generator from trace
• Symbolic replay and distillation pipeline

9. Conclusion

Super Stepper bridges the Python↔C++ divide with symbolic control, giving developers fine-grained visibility and power over hybrid model execution. It becomes a core debugging primitive within the TranSymbolics system.