Transcript of State of AI in 2026 | Nathan Lambert & Sebastian Raschka | Lex Fridman Podcast
Podcast Summaries
Key takeaways
- The early‑2025 release of DeepSeek's open‑weight R1 model delivered near‑state‑of‑the‑art performance while requiring less compute and cost than comparable closed models.
- Open‑weight LLMs from Chinese firms (DeepSeek, Miniax, ZAI's GLM, Quen) and Western groups (Allen Institute's Omo, Mistral AI, Nvidia's Nemo) expanded rapidly in 2024‑2025, improving accessibility despite the massive expense of training large models.
- Recent architectural tweaks—mixture‑of‑experts layers, group‑query attention, sliding‑window multi‑head latent attention, linear‑scaling attention, and state‑space‑inspired substitutes—enhance memory‑compute trade‑offs but remain incremental extensions of the original transformer design.
- Scaling laws continue to show power‑law links between compute, data, and model size, yet pre‑training returns are diminishing and inference‑time costs now dominate the financial picture, shifting focus to post‑training methods such as RLHF and RLVR.
- DeepSeek's RLVR (reinforcement learning with verifiable rewards) lets models generate answers, self‑grade their correctness, and iteratively reinforce accurate problem‑solving, unlocking stronger reasoning and code‑generation capabilities beyond pure pre‑training.
