Thinking Machines wows with Inkling 2 Mini looped transformer model that runs locally on laptops

Thinking Machines wows with Inkling 2 Mini looped transformer model that runs locally on laptops

Thinking Machines Lab Releases Inkling 2 Mini: A Compact, Powerful AI Model

Thinking Machines Lab, an AI startup based in San Francisco, led by former OpenAI CTO Mira Murati, is reshaping the AI landscape with the release of Inkling 2 Mini, an open-weights multimodal model designed for offline operation on a 32GB MacBook Pro or a single desktop GPU.

Challenging Conventional Paradigms

Inkling 2 Mini competes with much larger systems in software engineering and tool-use benchmarks, boasting 26.4 billion total parameters and a ~20GB memory footprint. This represents a strategic shift towards compact hardware for AI reasoning performance, as opposed to scaling parameter counts upward.

The Power of Looping

The model's innovation lies in its "looped transformer" architecture, where data passes through the middle layers multiple times per token. This design, contrasted with standard transformers that process tokens in a linear pass, allows Inkling 2 Mini to achieve superior performance at a fraction of the cost.

Technology: Looping for Deeper Reasoning

Standard transformer models struggle with complex problems that require more sequential reasoning steps than their depth allows, often resorting to generating long chains of thought (CoT). This approach increases latency and costs.

Inkling 2 Mini, however, takes a different approach. It's a 24-layer, decoder-only sparse Mixture-of-Experts (MoE) model, where each token is routed to 8 out of 128 experts, activating 3.4 billion parameters on a standard pass. The key twist is the looping mechanism: the first six and final six layers run once, while the middle 12 layers run three times using shared weights, akin to a human engineer double-checking their work.

This results in 48 effective layer applications per token, without increasing the model's storage. Attention residuals ensure information flows cleanly across iterations. Internal tests showed that a non-looping baseline required 140% more compute to match the looped model's performance.

Making it Practical

To make Inkling 2 Mini practical on consumer silicon, Thinking Machines implemented several optimizations:

  • Pruning two-thirds of the experts mid-training.
  • Distilling 1 trillion tokens from a larger internal Inkling model.
  • Supervised fine-tuning and 2.2 trillion tokens of reinforcement learning.
  • Native 4-bit NVFP4 precision for the first 20 layers and 8-bit MXFP8 for the final four layers.

This ensures developers benefit from the exact numeric format used during training, avoiding post-training quantization accuracy issues.

Safety Considerations and Interpretability

While looping enhances performance, it raises safety concerns. Thinking Machines addresses this by releasing a variable-loop checkpoint, TML-d24-varloop-research, trained for one to five passes, and partnering with NDIF and Goodfire for interpretability studies. Their preliminary tests show looping extends the model's no-CoT reasoning horizon from 16 seconds at one pass to 3 seconds at five passes.

Leave a Reply