lyric wedge
Hey everyone, I'm an 18-year-old independent researcher transitioning my SNN project from a proof-of-concept to a scalable architecture. Following a viral reception of v3 (51K views on Reddit), I’ve completely rewritten the core engine.
Current Architecture (v4.2):
Built in PyTorch, Nord operates entirely on discrete spikes rather than continuous activations.
• Spike-Driven MoE: Tokens are routed to experts based on emergent cluster firing rates (zero-cost routing).
• Memory Cortex: Persistent LIF neurons (
) with gated temporal attention for long-context retention.
• Zonal Specialization: The model autonomously organizes into Sensory, Association, and Executive zones with distinct firing patterns.
Current Dev Status:
I'm currently training a 618M parameter version on a single RTX A5000 (24GB VRAM). By utilizing custom deterministic sparse routing and gradient checkpointing, the model achieves 91-95% neuronal sparsity at inference with a rapidly stabilizing loss.
Where I need feedback/collaboration:
I would be incredibly grateful for code reviews or PRs, specifically regarding:
CUDA/PyTorch Optimization: Any tips on optimizing backprop for highly sparse temporal tensors to increase tok/s.
Scaling: Best practices for moving this architecture toward the 3B parameter mark efficiently.
Dynamic Computation: Ideas on implementing an "uncertainty-triggered" halt signal for the executive layers.
If you are interested in Neuromorphic Computing, Green AI, or pushing the limits of consumer hardware, I'd love your input!
Live Neuron Visualizer & Docs: https://www.nord-ai.net/
GitHub: https://github.com/gtausa197-svg/-Project-Nord-Spiking-Neural-Network-Language-Model
tardy plinth