← The Energy Lab · TR-2026-40, The Physical AI Hardware Lottery
Energy-based models, and the machine that would run them.
Eight questions about one idea, and the seventy-eight works that answer them. A robot can hold many right answers at once by learning a landscape instead of a number, and there is a kind of computer whose own physics does the hardest part of that for free. This page assumes those machines arrive. What it spends its attention on is the pace: which dots connect, which measurement unlocks which decision, and what would bring the timeline forward.
8
questions
78
works that answer them
9
already running on real chips
6
benches we ran ourselves
The short version.
A robot that learns one right answer per situation will average two good answers into a bad one. A robot that learns a landscape keeps both and picks one. Landscapes are harder to train, for a reason that turns out to be about the shape of our computers rather than about the idea, and there is a kind of machine whose own physics does the hard part while sitting still.
The idea
Learn a landscape over answers instead of a single answer. Low ground means a good answer. Several valleys means several good answers, which is what the world usually offers.
The physics
Finding low ground means letting the model settle, one step after another. A chip built for matrix multiplication has tens of thousands of lanes that all want to work at once, and settling gives it one thing to do at a time.
The opening
Build the landscape out of matter and it settles by itself, at the speed of physics, for the price of sitting still. Those machines are being fabricated now, in magnets, in light, and in ordinary transistors.
Every question below has a direct answer and the papers behind it. A field moves from a proof, to software, to a simulation, to a fabricated chip, to a product. Every stage is progress, and the column tells you which one each result has reached.
That the future arrives. The question is when.
There is a habit, strongest in industry and present in research, of asking any new approach to justify itself against whatever is already installed. That is a capital question and it is a fair one to ask with a budget in hand. It is not an engineering question, and the physics is indifferent to what was purchased last year.
Every machine is a snapshot
The processor most AI runs on today was designed to shade triangles. It became the engine of a field because a particular kind of mathematics happened to fit it, and that fit then shaped which ideas looked promising for fifteen years. Reading that history as a destination rather than a moment is the single assumption most likely to slow what comes next.
The economics point the same way
Energy is now the binding cost of computation, and a body carries its own power supply. When the constraint moves from how many operations you can buy to how many joules you can spend, machines that get their randomness from thermal noise instead of manufacturing it stop being a curiosity and start being the efficient choice.
So this page does not argue about whether settling machines are coming. It reports where they have got to, connects the parts of the field that have not yet noticed each other, and asks what would bring the timeline forward.
Question 1
What is an energy-based model?
A model that learns a landscape. Show it a situation and a candidate answer, and it gives back a single number: the energy. Low energy means the two fit together. High energy means they do not. That is the whole model. Answering a question means finding low ground: start somewhere on the landscape and let the answer settle downhill, the way a marble finds the bottom of a bowl. This buys something a direct predictor cannot have. A landscape can have more than one valley, so the model can hold more than one right answer at the same time. Ask a robot to pick up a mug and it can go left of the handle or right of it. Both work. A model trained to emit one number averages the two and reaches for the middle, which is the mug. A landscape keeps both valleys and picks one.
Where that leads. Joint-embedding predictive architectures are this same shape, arranged so the landscape sits over a compressed description of a scene rather than over its pixels. The model never has to draw the leaves on a tree to know a tree is there.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| A Learning Algorithm for Boltzmann Machines Ackley, Hinton & Sejnowski · 1985 doi:10.1207/s15516709cog0901_7 Cognitive Science 9(1) | The founding idea. A network learns by settling into a distribution and comparing two phases of itself: one with the world clamped on, one running free. The difference between them is the learning signal. | Everything on this page. The 2024 Nobel Prize in Physics recognised this lineage, and the machines in question seven are built to run it in materials rather than in code. | proved |
| A Tutorial on Energy-Based Learning LeCun, Chopra, Hadsell, Ranzato & Huang · 2006 cited via prior Institute record in Predicting Structured Data, MIT Press | The clearest statement of why the landscape view is worth having: the normalising constant is optional, and what really decides whether a model works is the loss you choose to shape the landscape with. | Two decades of architectures, including every JEPA below. | mapped |
| Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture Assran, Duval, Misra et al. · 2023 arXiv:2301.08243 CVPR 2023 | The first image JEPA. Predict the description of a hidden part of an image from the part you can see, entirely in a compressed space, with no hand-designed data augmentation. | Predicting descriptions instead of pixels is what makes video and planning affordable, which is exactly where the family went next. | running |
| V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning Assran, Bardes, Fan et al. · 2025 arXiv:2506.09985 arXiv | The same objective on video, and then used for planning. This is the point where the family stops being about recognising images and starts being about acting in the world. | A world model you can plan inside is the piece a robot needs. Question four is what it costs to run one. | running |
| LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics Balestriero & LeCun · 2025 arXiv:2511.08544 arXiv | Works out which shape the compressed space should have, rather than tuning toward it. The answer is an isotropic Gaussian, and the paper gives a single regulariser that reaches it, replacing a decade of accumulated tricks. | Once the target shape is a choice rather than an accident, you can choose a different one on purpose. The next entry does exactly that. | proved |
| Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations Kuang, Dagade, Rudner, Balestriero & LeCun · 2026 arXiv:2602.01456 ICML 2026 | Chooses a target shape modelled on the brain. Real neurons are mostly quiet, and this paper asks the representation to be mostly zero, with a dial for how quiet. On ImageNet-100 the backbone still reads 82.72 percent with 99 percent of the representation at zero. It contains LeJEPA as the special case where the dial is turned off. | Sparsity is a property today's chips cannot spend, since a processor burns the same power multiplying by zero. It is precisely the property the machines in question seven could turn into energy saved. | running |
| TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems Vo, Nguyen, Nguyen et al. · 2025 arXiv:2503.06380 arXiv | Text and images in one landscape, with the authors naming the strategy energy-based in their own title. | One landscape over several senses is the shape a robot needs, since a robot has several senses. | running |
| When Does LeJEPA Learn a World Model? Klindt, LeCun & Balestriero · 2026 arXiv:2605.26379 arXiv | Works out the conditions under which the objective produces a model of how the world moves, rather than a good description of how it looks. | Naming the conditions makes them engineerable. | proved |
| A Generalization Theory for JEPA-Based World Models Cui, Zhang, Wen & Wang · 2026 arXiv:2606.27014 arXiv | Bounds on how well a JEPA world model transfers to situations it was not trained on. | A robot meets new situations constantly, so a transfer guarantee is a safety argument in waiting. | proved |
| Learn from your own latents and not from tokens: a sample-complexity theory Korchinski, Favero & Wyart · 2026 arXiv:2605.27734 arXiv | Puts a price on the choice of learning target. On a compositional grammar of depth L, learning from tokens needs a number of samples exponential in L; predicting your own latent representations needs a number constant in L up to logarithms. The gap is not an efficiency gain. It is the difference between a data requirement that compounds with structure and one that does not. | Biology found this signal in 1950: the efference copy, a copy of the motor command sent inward for the system to predict against, is a self-latent. TR-2026-43 takes this as one of three legs for why recorded experience alone does not unlock Physical AI. | proved |
| Graph-level Representation Learning with Joint-Embedding Predictive Architectures Skenderi, Li, Tang & Cristani · 2023 arXiv:2309.16014 arXiv | The same objective learns over graphs, which are the natural shape for molecules, circuits and mechanisms. | One learning principle across images, video, graphs and signals is what makes it a principle. | running |
| Self-Supervised JEPA-based World Models for LiDAR Occupancy Completion and Forecasting Zhu & Choromanska · 2026 arXiv:2602.12540 arXiv | A JEPA world model on LiDAR, predicting which parts of the world will be occupied next. This is a sensor that ships on vehicles today. | The distance from research architecture to production sensor is now short enough to walk. | running |
| Video Joint-Embedding Predictive Architectures for Facial Expression Recognition Eing, Luna-Jiménez, Silva et al. · 2026 arXiv:2601.09524 arXiv | The video objective carried to reading human expression. | A machine that works alongside people benefits from noticing them. | running |
| Hierarchical Self-Supervised Representation Learning Framework for Multivariate Time Series (Hierarchical-JEPA on ECG) Kim · 2026 arXiv:2607.01145 arXiv | The objective on heart signals, built deliberately small so it can run on the device taking the measurement. | Learning at the edge of the network, next to the sensor, is the same destination this whole page is walking toward. | running |
Question 2
What makes a landscape hard to learn?
Turning a landscape into a probability means dividing by its total volume, and that total is a sum over every configuration the system could take. For even a small image it has more terms than the universe has atoms. So you do what a physicist does with any system too large to count: stop trying to add it up, let the system wander, and watch where it spends its time. Wander long enough and time spent is probability. That is Markov chain Monte Carlo, and it works. The wandering is a chain, which means each step starts from where the last one finished. The difficulty is order rather than cost. One step of the walk costs about the same as one forward pass of the network, so the arithmetic is ordinary. What makes it expensive is that the steps have to happen one after another.
Where that leads. Once you see the difficulty is order rather than cost, the research programme writes itself: shorten the chain, correct it, reweight it, or learn a network that jumps to the end of it. All four are live, and the works below are the best of each.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| How to Train Your Energy-Based Models Song & Kingma · 2021 arXiv:2101.03288 arXiv tutorial | The best single place to start. Lays out the walk-based route, then every route that avoids the walk: score matching, noise-contrastive estimation, and the rest. | Read this first and everything else in this section reads as a variation on a theme. | mapped |
| Implicit Generation and Generalization in Energy-Based Models Du & Mordatch · 2019 arXiv:1903.08689 arXiv | The paper that made landscape models work at real scale, on ImageNet and on robotic hand trajectories, and demonstrated what the family uniquely gives: models that compose, fill in missing pieces, and know when they are out of their depth. | Composition is the capability that keeps pulling researchers back to energy, and it recurs throughout question three. | running |
| Improved Contrastive Divergence Training of Energy Based Models Du, Li, Tenenbaum & Mordatch · 2020 arXiv:2012.01316 arXiv | Finds a term the standard training recipe had been quietly dropping for two decades, and shows that putting it back changes how training behaves. | A recipe that had been treated as an approximation turns out to have been missing something specific and fixable. | running |
| Training Deep Energy-Based Models with f-Divergence Minimization Yu, Song, Song & Ermon · 2020 arXiv:2003.03463 arXiv | Widens the family of objectives you can train a landscape with, beyond the single classical choice. | More objectives means more ways to trade sample quality against the length of the walk. | proved |
| Training Energy-Based Models with Diffusion Contrastive Divergences Luo, Jiang, Hu et al. · 2023 arXiv:2307.01668 arXiv | Replaces the walk with a diffusion process, which is a walk you already know the answer to. | Borrowing the diffusion machinery to fix the energy training problem is a pattern that repeats through the next three entries. | running |
| Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood Zhu, Xie, Wu & Gao · 2023 arXiv:2309.05153 arXiv | The most effective chain-shortening idea so far. Train a series of landscapes on progressively noisier data, so each walk only has a short distance to travel. | Short walks are the practical route to landscape models at scale on today's machines. | running |
| Efficient Training of Energy-Based Models Using Jarzynski Equality Carbone, Hua, Coste & Vanden-Eijnden · 2023 arXiv:2305.19414 arXiv | Reaches into non-equilibrium thermodynamics for an identity that lets you correct a walk that stopped early, exactly. | Physics supplying the fix for a machine-learning problem, in a page of algebra. There is more of this available than the field has used. | proved |
| Learning Energy-Based Prior Model with Diffusion-Amortized MCMC Yu, Zhu, Xie et al. · 2023 arXiv:2310.03218 arXiv | Trains a network to jump to where the walk would have ended, so the cost is paid once during training rather than on every use. | Amortisation is the software answer to a sequential cost. Question seven is the hardware answer. | running |
| Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective Ghio, Dandi, Krzakala & Zdeborová · 2023 arXiv:2308.14085 arXiv | Analyses when neural samplers succeed and fail using spin-glass theory, the same physics the machines in question seven are built out of. | This is the bridge between the two halves of the page: the mathematics describing the models and the mathematics describing the hardware are the same mathematics. | proved |
| Incorporating Inductive Biases to Energy-based Generative Models Li & Liu · 2025 arXiv:2505.01111 arXiv | Builds structure you already know about into the landscape rather than making the model rediscover it. | Evidence the training question is still an active research programme in 2025, not a settled one. | running |
| A Diffusive Classification Loss for Learning Energy-based Generative Models OuYang, Grenioux, Hernández-Lobato et al. · 2026 arXiv:2601.21025 arXiv | A new training objective, published January 2026. | The most recent entry in a line that runs back to 1985 and is still producing results. | running |
Question 3
Why do today's models predict slopes instead of landscapes?
Because a slope is one pass through the network and a landscape is two. A diffusion model and an energy-based model can describe exactly the same distribution. They differ in what they store. The energy version stores the height of the landscape, then takes its gradient to find which way is downhill. The score version stores the downhill direction itself. Physicists know this pair well: it is a potential and its field, the same information in two coordinates. Getting a gradient out of a neural network means running it forward and then backward, on every step of the walk. Getting a direction out means running it forward. On a machine that charges by the pass, one of these is half the price of the other, and that is the one the field built on.
Where that leads. The energy view comes back the moment you want something a bare direction field cannot give: composing two models into one, scoring a sample, or guaranteeing the field is the gradient of anything at all. Every entry below is a return trip.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| A Connection Between Score Matching and Denoising Autoencoders Vincent · 2011 doi:10.1162/NECO_a_00142 Neural Computation 23(7) | The identity the whole modern generative stack rests on: learning to remove noise from something and learning the downhill direction of its landscape are the same task. | Every diffusion model you have used is estimating the slope of a landscape. This paper is why. | proved |
| Should EBMs model the energy or the score? Salimans & Ho · 2021 OpenReview:9AS-TF2jRNb ICLR 2021 Energy-Based Models workshop | Asks the question in the title directly, and tests it. Storing the height has the cleaner theory, since its slope is guaranteed to be a real slope. Storing the direction worked better on the machines of 2021. | A choice made on the hardware of one year, recorded plainly enough that it can be revisited on the hardware of another. That is what makes it the pivot of this page. | running |
| MCMC-Correction of Score-Based Diffusion Models for Model Composition Sjöberg, Lindqvist, Önnheim et al. · 2023 arXiv:2307.14012 arXiv | You can get much of the benefit of storing the height while still storing the direction, by adding an accept-or-reject step. | The gap between the two coordinates is smaller in practice than the theory suggests, which widens the options for anyone designing a system. | running |
| Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC Du, Durkan, Strudel et al. · 2023 arXiv:2302.11552 arXiv | Why composition needs energy. Adding two landscapes together is a meaningful operation, and it gives you a model of both conditions at once from two models trained separately. | Compose a model of where objects are with a model of what the arm can reach, and you get a plan. This is the capability nothing else provides. | running |
| Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo Thornton, Bethune, Zhang et al. · 2025 arXiv:2502.12786 arXiv | Composition and control together, with the expensive part distilled down in advance. | Control is composition with the robot's own limits as one of the terms. | running |
| Energy-Based Transformers are Scalable Learners and Thinkers Gladstone, Nanduru, Islam et al. · 2025 arXiv:2507.02092 arXiv | The strongest recent case for storing the height. The model finds its answer by walking downhill on a learned landscape until it stops improving. It learns up to 35 percent faster with scale than the standard transformer recipe, and beats a diffusion transformer at denoising while using fewer passes. | A model that decides for itself how long to think about a problem, because the landscape tells it when it has arrived. That is a different kind of machine from one that always takes the same number of steps. | running |
| Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics Jayaram & Thickstun · 2021 arXiv:2105.08164 arXiv | That the one-step-at-a-time structure belongs to the sampling procedure, not to the model. Swap the procedure and a sequential model samples in parallel. | A clean demonstration that the bottleneck on this page is a choice of algorithm meeting a choice of machine, and both are ours to change. | running |
Question 4
What does settling cost a robot?
A diffusion policy settles on every action it emits. Ten to a hundred passes through the network, then one command to the motors, then it starts again for the next command. At ten passes a policy runs at a comfortable rate. At a hundred it does not, and a robot that thinks too slowly walks into the thing it was avoiding. Almost every robot-policy paper since 2023 is an engineer buying those passes back. Distil a hundred-step model into a one-step one. Train a one-step model from scratch instead. Reshape the path so fewer steps arrive at the same place. Published results move action rates from 1.5 Hz to 62 Hz, and a one-step flow policy reports about 71 Hz.
Where that leads. Every one of those gains comes from removing settling steps, which raises the question the next section answers with a measurement: what were the settling steps doing?
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| Implicit Behavioral Cloning Florence, Lynch, Zeng et al. · 2021 arXiv:2109.00137 CoRL 2021 | The mug problem, solved. The policy is a landscape over actions and the robot picks a valley, so it keeps the several ways of doing a task instead of averaging them into a way that does not work. | This is where landscapes entered robotics, and why they stayed. | running |
| Diffusion Policy: Visuomotor Policy Learning via Action Diffusion Chi, Xu, Feng et al. · 2023 arXiv:2303.04137 RSS 2023; IJRR 44 (2025) | The policy that made this mainstream, and it describes its own inference as a series of Langevin steps: physics vocabulary, in a robotics paper, describing what the robot does between seeing and moving. | Sampling a landscape is now a standard component of a working robot stack. The rest of this section is the field making it fast. | running |
| Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models Singh, Tu & Sindhwani · 2023 arXiv:2309.05803 arXiv | Training objectives that give a landscape policy without needing a walk during training at all. | Removes the training-time cost, leaving only the inference-time one, which is the one question seven addresses. | running |
| EBT-Policy: Energy Unlocks Emergent Physical Reasoning Capabilities Davies, Huang, Gladstone et al. · 2025 arXiv:2510.27545 arXiv | Energy-based transformers driving a robot. The authors report the landscape view removing the schedule-tuning that diffusion policies need, and holding accuracy better over long action sequences. | A policy that verifies its own answer before acting is the beginning of a robot that knows when it is unsure. | running |
| Energy-based Compositional Diffusion Planning Sun, Mishra, Lu et al. · 2026 arXiv:2606.21646 arXiv | Composition applied to planning, June 2026. Build a plan by adding together landscapes for each thing the plan has to satisfy. | New goals become new terms in a sum, rather than a new model to train. | running |
| Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations Urain, Mandlekar, Du et al. · 2024 arXiv:2408.04380 arXiv | The whole robotics side of this page in one place, with the inference cost of each approach stated. | The best single orientation if you are entering the field here. | mapped |
| One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation Wang, Li, Mandlekar et al. · 2024 arXiv:2410.21257 arXiv | Compresses a many-step policy into a single step, taking action prediction from 1.5 Hz to 62 Hz. That is the difference between a robot that pauses and a robot that flows. | A fortyfold rate gain from removing settling steps sets up the natural question: how many of those steps were load-bearing? | running |
| Hybrid Consistency Policy: Decoupling Multi-Modal Diversity and Real-Time Efficiency in Robotic Manipulation Zhao, Shen, Zhai et al. · 2025 arXiv:2510.26670 arXiv | Names the trade in its title and then separates the two sides, keeping the several-right-answers property while still running fast. | Speed and multimodality stop being a single dial, which is a real design win. | running |
| ElasticFlow: One-Step Physics-Consistent Policy with Elastic Time Horizons for Language-Guided Manipulation Chen, Long, Li & Long · 2026 arXiv:2605.08799 arXiv | About 71 Hz from a single network evaluation, with physical consistency kept as an explicit objective rather than hoped for. | One evaluation per action is the floor for this approach. Below it, the gain has to come from the machine. | running |
| Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control Gao, Shen, Zhang et al. · 2026 arXiv:2604.03540 arXiv | One-step behaviour learned directly, with no slow teacher model needed first. | Removes the dependency on training an expensive model in order to get a cheap one. | running |
| Ising Machines for Model Predictive Path Integral-Based Optimal Control Werthen-Brabants & Simoens · 2025 arXiv:2512.15533 arXiv | A sampling controller mapped onto Ising hardware. The two halves of this page, joined for the first time in the published literature we surveyed. | The controller here is a classical sampling method rather than a learned policy. Putting a learned policy on the same hardware is the obvious next experiment, and it is open. | simulated |
| Two published control rates, converted to a budget per evaluation Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab: research/ebm_substrate/policy_budget.py arithmetic on the published rates in this trail | A policy running at H hertz with N evaluations per action has 1/(H x N) seconds for each evaluation, whatever machine it runs on. The distilled one-step policy at 62 Hz gets 16.13 ms per evaluation and the one-step flow policy at 71 Hz gets 14.08 ms, a difference of 1.145 times against 14.5 percent in rate. Read as budgets they are the same measurement, so quoting one against the other compares model sizes rather than approaches. | The undistilled policy is the interesting row: 1.5 Hz across 100 evaluations gives each one only 6.67 ms. Distillation therefore made each evaluation 2.42 times more expensive and ran 100 times fewer of them, for a net 41-fold rate gain. The acceleration literature is buying fewer sequential steps rather than cheaper arithmetic, which is exactly the quantity a settling substrate removes rather than reduces. | proved |
Question 5
What is the settling actually buying?
We built a test where the right answer is known exactly, so there is nothing to argue about. A sampling controller has two knobs. Width is how many futures it imagines at once, and every one of them can be imagined on a separate lane of a chip. Depth is how many times it refines its guess, and those have to happen in order. We swept both against the exact optimum. Width hits a wall. At one refinement pass, 200 imagined futures and 800 imagined futures land in the same place: 9 targets reached out of 24. Four times the parallel work, the same result. Depth does not hit that wall. Give the same total compute to refinement instead and the arm reaches all 24, using 17.8 joules against the reference controller's 26.4. The gap grows with the number of joints, which is to say it grows with how real the robot is. At one joint the one-pass error is 0.57 percent. At eight joints it is 78.74 percent, and eight refinement passes bring it back to 5.48.
Where that leads. The knob that carries the accuracy is the one that will not divide across lanes. That is the sentence this whole page turns on, and it is why the shape of the machine matters so much.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| Refinement depth against rollout width, measured against an exact optimum Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab bench: examples/sequential_depth.rs TR-2026-40 section 7.1 | Width reaches a floor. At one refinement pass, 800 imagined futures give 20.97 percent excess cost over the exact optimum and 3,200 give 21.04, a difference smaller than the run-to-run spread. Eight refinement passes at the same width reach 0.64 percent, a factor of 33. | Buying more parallel work stops helping at a point you can measure. That point is where the machine's shape starts to matter. | measured here |
| How the effect grows with the number of joints Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab bench: examples/depth_vs_dimension.rs TR-2026-40 sections 7.2 and 7.3 | The width floor rises steeply with the size of the action space: 0.57 percent at one joint, 21.29 at two, 44.20 at four, 78.74 at eight. Eight refinement passes bring the eight-joint case back to 5.48 percent. Across four different coupling structures the eight-joint figure lands between 77.64 and 79.03, so it is the number of joints that sets this, not the details of the mechanism. | The effect gets stronger as the robot gets more real. A humanoid has far more than eight joints. | measured here |
| A three-joint arm with gravity, actuator lag, torque limits and a measured power model Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab bench: examples/arm_width_vs_depth.rs TR-2026-40 sections 7.4 and 7.5 | At an identical compute budget of 19,200 dynamics evaluations per action, spending it on width reaches 19 of 24 targets using 30.9 joules, and spending it on refinement reaches all 24 using 17.8, against a reference controller at 26.4. The power model's copper, viscous and friction coefficients are calibrated to a measured 71.5 joule reach on a Unitree G1 arm. | Refinement wins on the task and on the joules at the same time. Both currencies point the same way, which is rare and worth building on. | measured here |
Question 6
Why is a graphics processor the wrong shape for a walk?
An accelerator is tens of thousands of lanes that all want to work in the same instant. Hand it a matrix multiply and every lane fills. Hand it a walk and one lane works while the rest wait for it. The loss shows up as bandwidth rather than arithmetic: a step that depends on the previous step cannot be made wider, so the chip spends its time moving numbers instead of multiplying them. The hardware community describes the same effect from its own side, in the analog memory work below. Sara Hooker named the pattern in 2020 and called it the hardware lottery. An idea wins partly because it fits the machine of its day. The useful half of that observation is the other half: some ideas lose the same way, and losing a lottery is not the same as being wrong.
Where that leads. If an idea lost because of the machine, then the machine is the thing to change. That is a far more tractable problem than changing the idea, and people are already building the answer.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| The Hardware Lottery Hooker · 2020 arXiv:2009.06489 arXiv | That research directions win and lose partly on how well they suit the machines available when they are proposed, and that a lost lottery can look exactly like a failed idea for years. | The framing that makes this whole page a research programme instead of a history lesson. If an idea lost to a machine, build a different machine. | mapped |
| AI+HW 2035: Shaping the Next Decade Chen, Cong, Mirhoseini et al. · 2026 arXiv:2603.05225 arXiv | Where the AI hardware community expects the next decade to go. | The document any substrate proposal has to position itself against, and a useful read for spotting which lanes are crowded and which are open. | mapped |
| Ising machines as hardware solvers of combinatorial optimization problems Mohseni, McMahon & Byrnes · 2022 doi:10.1038/s42254-022-00440-8 Nature Reviews Physics 4 | The full family tree of machines that compute by settling: spintronic, optical, memristive, digital, and superconducting. | The best single map of question seven, written by physicists for physicists. | mapped |
| Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models Leroux, Manea, Sudarshan et al. · 2024 arXiv:2409.19315 Nature Computational Science 5(9), 2025, doi:10.1038/s43588-025-00854-1 | The mismatch named from the hardware side: at every generation step the stored values have to be hauled back into fast memory, and that haul is where the time and the energy go. Computing where the data already sits, in analog gain cells, reports up to 70,000 times less energy and 100 times the speed for the attention operation on a 1.5-billion-parameter model. | The figure is for one operation rather than a whole task, which is exactly the kind of distinction question eight exists to keep straight. What it establishes is the mechanism, and the mechanism is the same one that slows a walk. | simulated |
Question 7
What kind of machine fits a walk?
One whose own physics does the walking. Here is the idea in a sentence. Take the landscape you want to sample from, and build a physical system whose energy is that landscape. Let it sit at a temperature. The system now spends time in each configuration in exact proportion to how probable that configuration is. You have not simulated the walk. The walk is what the device does while sitting there. This is not speculative physics. It is the Boltzmann distribution, which is why this family of models was named after Boltzmann in the first place. What is new is that we can build the systems. Two hundred and fifty magnetic tunnel junctions running as a probabilistic Ising machine. A programmable photonic Ising machine at 200 billion operations per second. An embedded Ising machine that answers in about 100 milliseconds and, remarkably, holds that time nearly flat as the problem grows. Training has an answer too. Equilibrium propagation reads gradients off the difference between a settled state and a slightly nudged one. No backward pass, no stored tape of activations. A device that settles can learn by settling twice, and in June 2026 that rule reached ImageNet.
Where that leads. Sampling and learning both become things the material does, rather than things a processor imitates. That is what makes this a change of kind rather than a change of degree.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| An efficient probabilistic hardware architecture for diffusion-like models Jelinčič, Lockwood, Garlapati, Schillinger, Chuang, Verdon & McCourt · 2025 arXiv:2510.23972 npj Unconventional Computing, July 2026, doi:10.1038/s44335-026-00075-3 | A denoising model built out of ordinary transistors used as sources of controlled randomness. The system-level analysis puts it at parity with a GPU on an image benchmark using roughly 10,000 times less energy per generated sample. The paper also states the connection this page is built on, from the hardware side: probabilistic computing reaches AI through energy-based models, and it names robotic path planning as one of the tasks. | The figure comes from a physical model rather than a fabricated chip, and the authors say so plainly. What makes it remarkable is the parts list: standard CMOS, no exotic devices, which is what makes it something a foundry could actually run. | modelled |
| Machine Learning-assisted High-speed Combinatorial Optimization with Ising Machines for Dynamically Changing Problems Hamakawa, Kashimata, Yamasaki & Tatsumura (Toshiba) · 2025 arXiv:2503.23966 Nature Communications, 2026 | An embedded Ising machine answering in about 100 milliseconds end to end, and holding roughly that time as the problem grows. It also compresses the problem's couplings by about 32 times to fit the fabric. | Flat time against problem size is the property a control loop needs, because a controller cannot have its deadline depend on how complicated the world got this second. | built |
| 250 magnetic tunnel junctions-based probabilistic Ising machine Nature Communications · 2026 doi:10.1038/s41467-026-72020-8 Nature Communications, April 2026 | 250 magnetic devices whose natural thermal jitter is the randomness, wired into a working sampling machine. The team also found a way to update whole clusters at once on sparse problems, giving a tenfold speed-up with no change to the hardware. | That cluster update is the sequential bottleneck of this page being attacked directly, in silicon, and winning a factor of ten for free. | built |
| A Unified Performance-Cost Landscape of Parallel p-bit Ising Machines Based on Update Dynamics Onizawa & Hanyu · 2026 arXiv:2604.01564 Scientific Reports, April 2026, doi:10.1038/s41598-026-47285-0 | The engineering design guide for these machines. Synchronous updates turn out to be workable rather than unstable, and reusing each probabilistic bit across time gives equal or better answers at less than half the hardware on 800 to 2,000 node benchmarks. Three or four bits of precision suffice. | Low precision and time-shared hardware are what make a fabric small enough and cheap enough to sit on a robot. | simulated |
| Physics-inspired Ising Computing with Ring Oscillator Activated p-bits Aadit, Grimaldi, Carpentieri et al. · 2022 arXiv:2205.07402 arXiv | Probabilistic bits built from ring oscillators, which is to say from ordinary digital logic, with no special material required. | A route to a sampling fabric that any digital design team can build today, on any process. | built |
| Predicting sampling advantage of stochastic Ising Machines for Quantum Simulations Berns, Rodrigues et al. · 2025 arXiv:2504.18359 arXiv | Works out in advance which problems a sampling machine will actually beat a processor on, rather than finding out by building one. | Knowing where the advantage lives before you tape out a chip is how this field avoids a decade of expensive guessing. | modelled |
| Free-Energy Machine for Combinatorial Optimization Shen, Pan, Wang et al. · 2024 arXiv:2412.09285 arXiv | Uses free energy itself as the computational primitive, which is the same quantity a landscape model minimises. | When the hardware's native quantity and the model's objective are the same quantity, the translation layer between them disappears. | running |
| Thermodynamic Algorithms for Quadratic Programming Lipka-Bartosik, Donatella et al. · 2024 arXiv:2411.14224 arXiv | Carries thermodynamic computing from sampling into constrained optimisation. | Constrained optimisation is what a robot does when it plans around its own joint limits, so this widens the set of tasks a fabric could host. | proved |
| VCMA-MRAM In-Memory Stochastic Sampling for Edge Boltzmann Machine Inference Electronics 15(8) · 2026 doi:10.3390/electronics15081622 Electronics, April 2026 | Magnetic memory doing the sampling in place, at roughly 10 femtojoules per sample, with the edge device named as the target from the outset. | Ten femtojoules is a number worth sitting with. It is the energy a single photon of visible light carries, about twenty thousand times over. | simulated |
| Memristive Boltzmann machine: A hardware accelerator for combinatorial optimization and deep learning Bojnordi & Ipek · 2016 doi:10.1109/HPCA.2016.7446049 IEEE HPCA 2016 | 57 times the performance and 25 times lower energy than a multicore processor, for the Boltzmann machine, worked out in 2016. | The hardware argument is a decade old and was correct when it was made. What changed is the other side: this model class now drives robots, so the machine finally has something to run. | simulated |
| Photonic restricted Boltzmann machine for content generation tasks Luo, Fang, Zhang & Chen · 2025 arXiv:2508.20472 arXiv | An optical encoding that takes one step of the sampling walk from growing with the problem size to being constant in it. | Attacking the exponent rather than the constant. That is the kind of change that moves a field rather than a benchmark. | built |
| Programmable 200 GOPS Hopfield-inspired photonic Ising machine Nature · 2025 doi:10.1038/s41586-025-09838-7 Nature, December 2025 | A programmable optical settling machine at 200 billion operations per second, in Nature. | Programmable is the word that matters. A fixed optical circuit is a demonstration; a programmable one is a computer. | built |
| Maximum-Entropy Analog Computing Approaching ExaOPS-per-Watt Energy-efficiency at the RF-Edge Undavalli, Rashed, Xiao et al. · 2025 arXiv:2510.24975 arXiv | Maximum entropy, the same principle that sets the target shape in the JEPA work above, used as the organising idea for an analog edge computer, with an operations-per-watt figure attached. | The same principle appearing in the objective of a model and in the design of a chip is the kind of coincidence that usually means something. | built |
| Sampling binary sparse coding QUBO models using a spiking neuromorphic processor Henke, Pelofske, Hahn & Kenyon · 2023 arXiv:2306.01940 arXiv | Sampling, not just optimisation, running on a fabricated brain-inspired processor, on sparse coding. | Sparse coding is exactly the representation the Rectified LpJEPA work asks for. The model class and the hardware are converging on the same property from opposite directions. | built |
| Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation Scellier & Bengio · 2016 arXiv:1602.05179 arXiv | How to learn without a backward pass. Let the network settle, nudge the output slightly toward the right answer, let it settle again, and the difference between the two settled states is the gradient. | No backward pass and no stored tape of activations means the memory cost of training collapses. This is what makes learning on a device at the edge physically plausible. | proved |
| Training End-to-End Analog Neural Networks with Equilibrium Propagation Kendall, Pantone, Manickavasagam et al. · 2020 arXiv:2006.01981 arXiv | A circuit of nonlinear resistors is already an energy-based model, and each resistor can compute its own weight update from the voltage across itself. Nothing has to be sent anywhere. | Learning becomes a local physical process. There is no central update, no gradient bus, and nothing to move. | proved |
| EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementations Martin, Ernoult, Laydevant et al. · 2020 arXiv:2010.07859 arXiv | The same learning rule expressed in spikes, which is the language neuromorphic chips already speak. | Connects the rule to a class of hardware that ships today. | simulated |
| Hybridizing Equilibrium Propagation with Ising Machines for Efficient Energy-Based Learning Fan, Lu, Wu & Wang · 2026 arXiv:2606.09112 arXiv | The learning rule that needs a settling network, running on the hardware that settles. The authors aim it explicitly at low-power edge and sensor deployment. | Both halves of the problem answered by the same physics: the device samples to think and settles twice to learn. | simulated |
| Optical Implementation of Equilibrium Propagation Using Spatial Photonic Ising Machines Vanden Abeele, Veraldi et al. · 2026 arXiv:2606.13454 arXiv | The same union built in light. | Two independent groups reaching the same architecture in different physics, within months of each other. | built |
| FlowEqProp: Training Flow Matching Generative Models with Gradient Equilibrium Propagation Gower · 2026 arXiv:2604.08150 arXiv | Applies the settling learning rule to flow matching, which is the recipe behind current robot policies. | Closes the loop: the training rule a settling device can run, applied to the model class robots actually use. | simulated |
| Training a Predictive Coding Network on ImageNet using Equilibrium Propagation Kerjan, Høier, Scellier & Bengio · 2026 arXiv:2606.03584 arXiv | The scale result the rule had been waiting a decade for. Equilibrium propagation, on ImageNet, June 2026. | The question shifts from whether the rule scales to which material should run it. | running |
| Trainable neuromorphic spintronic hardware via analog finite-difference gradient methods Nature Communications · 2026 doi:10.1038/s41467-026-73624-w Nature Communications, May 2026 | A magnetic chip that trains itself in place, reaching 97.9 percent on handwritten digits without a processor computing its gradients. | Learning inside the material, on fabricated hardware, in 2026. The remaining distance is from digits to a robot policy. | built |
Question 8
What would get us there faster?
Four things, and each of them is already in hand. The first is the free tenfold. On sparsely connected problems a fabricated probabilistic Ising machine can update whole clusters of spins at once instead of one at a time, which attacks the sequential bottleneck of this entire page and wins a factor of ten with no change to the silicon. The second is that these machines can be small. Three or four bits of precision are enough, and time-sharing each probabilistic bit across the problem cuts the hardware in half again, which is the difference between a rack and something that rides on a robot. The third is that you do not need an exotic material to start: probabilistic bits built out of ring oscillators are ordinary digital logic on an ordinary process. The fourth is that training scales, which the field waited a decade to learn. Equilibrium propagation reached ImageNet in June 2026. We added a fifth from our own bench. A controller re-conditions a resident model at every tick, and we measured that operation at 5.4 to 41 microseconds against 0.44 to 5.26 milliseconds to load a model from scratch. Even paying the full load every tick, a 100 Hz budget fits over a hundred settling steps at a quarter of a million nodes. Conditioning bandwidth is comfortable at robot rates and becomes interesting above a kilohertz, which tells a hardware team exactly which regime is worth their attention.
Where that leads. All three quantities now carry a first figure from this bench: conditioning costs 5.4 to 41 microseconds against 0.44 to 5.26 milliseconds to load, the compute half of an arm's completed task is about 3 percent, and a 75-million-parameter policy is 1.2 to 1.4 percent of a humanoid's body power. The two published share figures differ because they measure different things, a policy against a full autonomy stack. What remains is the same measurement on a dedicated sampling fabric, where the clamp is a physical port rather than a buffer upload, and a metered autonomy stack on a real humanoid.
| The work | What it shows | Where it leads | Stage |
|---|---|---|---|
| What it costs to load a model against what it costs to re-condition one Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab bench: gpu/examples/clamp_rate.rs ferrotherm 0.10.0, Apple M5 Max | Loading a model runs from 0.44 milliseconds at 256 nodes to 5.26 at 262,144. Re-conditioning one already loaded runs from 5.4 to 41 microseconds. The load is worth 51 to 128 re-conditionings. Even paying the full load on every tick, a 100 Hz budget still fits over a hundred settling steps at a quarter of a million nodes. | Conditioning bandwidth is comfortable at robot rates and becomes the limit above roughly a kilohertz. That tells a hardware team which regime to design for, and it tells a robotics team that this particular worry can come off the list. | measured here |
| What the thinking costs, beside what the moving costs Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab bench: gpu/examples/planner_joules.rs ferrotherm 0.11.0, Apple M5 Max, SoC power counters | The compute half of a completed task, measured at the wall against an idle baseline. A dynamics evaluation of the arm controller's rollout loop costs 2.06 to 2.40 times ten to the minus seven joules across two clean passes on separate runs. A completed reach costs 2,592,000 of them, so the planning costs 0.53 to 0.62 joules against 17.8 joules of actuation. Compute is about 3 percent of the task, and the motors outweigh it roughly thirty to one. | Set that beside the published AMR figure, where autonomy compute took 37.3 percent and the motors 16.6. The compute share is not a constant of Physical AI: it spans at least 3 percent to 37 across machines, so any whole-robot energy ceiling has to name the body it was computed for. This figure also prices a general-purpose processor, which is the machine the argument is about escaping, so it is an upper bound and the distance from it is what a substrate has to buy. | measured here |
| What a humanoid's compute share would have to be, from the vendor's own numbers Institute for Physical AI @ BMI, Charlot Lab, on Unitree's published figures · 2026 Unitree G1 published specification 9000 mAh, 54 V charger, about 2 h, 35 kg, 23 degrees of freedom | Unitree publishes a 9 Ah pack, a 54 V charger and about two hours of battery life for the G1. A 13-series lithium pack sits near 48 V nominal, so the pack holds 432 to 486 Wh and the whole machine averages 216 to 243 W across the stated runtime. At a 50 Hz whole-body control rate, sitting at the arm's 3 percent share means a policy drawing 6.5 to 7.3 W, and sitting at the mobile robot's 37.3 percent means 81 to 91 W. | The two published shares are 12.4 times apart in policy power on one unchanged body, and that ratio is independent of which body figure you use. A whole-robot energy ceiling is therefore a claim about a body AND a policy, and the measurement of which model size lands where inside that band has now been taken: see the policy budget entry below. | proved |
| Assessment of AI Impact on Energy Utilization in Robotics Vodovozov & Raud · 2026 doi:10.3390/en19143364 Energies 19(14), 16 July 2026 | A 2026 review of how artificial intelligence changes energy use across robotic platforms, which is the survey any compute-share claim has to be positioned against. | The full text was not reachable from this review's tooling, so it is recorded as the standing survey on this axis rather than mined for figures. A measured compute-versus-actuation split for a humanoid was not located in it or in the other 2026 sources searched. | mapped |
| Where each policy size lands on a humanoid's power budget Institute for Physical AI @ BMI, Charlot Lab · 2026 Charlot Lab bench: research/ebm_substrate/policy_budget.py Apple M5 Max SoC counters, batch 1, seven sizes, run record preserved beside the bench | A policy evaluation costs 5.0e-5 joules at 49 thousand parameters and 5.9e-2 joules at 75.5 million, measured at the wall with every size against a fresh idle baseline whose wander stayed within eight percent of the signal. At a 50 Hz whole-body rate even the 75-million-parameter policy draws 2.9 W, which is 1.2 to 1.4 percent of a G1's average body power: below the arm's three percent and a twenty-fifth of the mobile robot's 37.3. The bench's own prediction expected the tested range to reach into the band and was falsified in the informative direction. | A control policy is not where a humanoid's compute share comes from: the 37.3 percent figure is a full autonomy stack, perception and planning and infrastructure, not a policy. Joules per evaluation sit flat near 7e-10 per parameter in the bandwidth-bound regime, so reaching that share at 50 Hz takes a policy of order two billion parameters on this silicon, an extrapolation stated as one. The energy question for humanoid control policies is therefore already comfortable on general-purpose silicon, and the joules a substrate can win live in the stack around the policy. | measured here |
Six places where two literatures are reaching for each other.
These are the connections that do not appear in either field's reading list, because the people who would draw them publish in different venues. Every one of them is sourced from the works above.
Sparsity meets a machine that can spend it
The newest JEPA work learns representations that are 99 percent zero, deliberately copying how sparingly real neurons fire. A processor burns the same energy multiplying by zero as by anything else, so today that sparsity is free to compute and worth nothing. A sampling fabric does not work that way. The model class and the hardware are converging on the same property from opposite ends of the stack, and neither paper cites the other.
Maximum entropy appears twice
It is the principle that sets the target distribution in the JEPA objective, and it is the organising idea behind an analog edge computer reaching for exa-operations per watt. Same principle, two layers apart, arrived at independently. When a idea shows up in both the objective and the substrate, that is usually the shape of the thing.
The model's mathematics is the machine's mathematics
Spin-glass theory is what physicists use to work out when a neural sampler will succeed or fail. Spin glasses are also, literally, what the hardware is made of. The analysis of the algorithm and the physics of the device are the same subject, which almost never happens and makes the design loop unusually short.
The name was always the clue
These are Boltzmann machines. The Boltzmann distribution is not a thing a chip has to compute; it is what matter does at a temperature, without being asked. The field spent forty years simulating in software a behaviour that is free in physics, and named the models after the physicist the whole time.
The memory wall and the activation tape are one wall
Training a network means storing every intermediate value so you can run backwards through them, and that storage is a large part of why training lives in data centres. Equilibrium propagation needs none of it: settle, nudge, settle again, and the difference is the gradient. The constraint that keeps learning centralised is the one a settling machine removes as a side effect.
The 2016 answer was right and early
A memristive Boltzmann machine was worked out in 2016 at 57 times the performance and 25 times lower energy than a multicore processor. The engineering was sound and the timing was not: the model class had nothing important to run. It does now, because these same models drive robots. The hardware argument did not need to be discovered, only met halfway.
What this looks like when it lands.
None of what follows needs new physics. Every piece exists somewhere on the road above, and the remaining work is engineering: interfaces, memory hierarchies, device characterisation, and measurements this bench has now started taking.
A robot that considers its options
Multimodal policies at full control rate, with the thinking time removed rather than shortened. Today a robot buys its speed by giving up refinement steps. On a settling fabric the refinement is what the material does between one tick and the next, so it stops being something to economise on.
A machine that learns where it works
Equilibrium propagation needs no backward pass and stores no tape of activations, so the memory cost that keeps training in data centres largely disappears. A device that settles can learn by settling twice, in the field, from what it encounters there.
Quiet representations that cost nothing
The newest JEPA work learns representations that are 99 percent zero, in deliberate imitation of how sparingly real neurons fire. A processor burns the same energy multiplying by zero as by anything else. A sampling fabric does not, which turns a property of the model into joules saved.
Energy budgets that change what a body can be
Compute is one term and the motors are the other, so a chip that wins by four orders of magnitude moves the whole-robot figure by a bounded amount. What it changes is everything downstream of that: thermal envelope, duty cycle, how long a battery lasts, and how small a useful machine can get.
The measurement that turns all of this into a design target, now taken
Joules per completed task on a body, with the compute counted in the same sum as the motors. Every four-orders-of-magnitude figure on this page is quoted against a different denominator, so none of them can be spent by an engineer choosing what to build. This one can. On the three-joint arm from question five, a dynamics evaluation of the controller's rollout loop costs about two ten-millionths of a joule measured at the wall; a completed reach costs 2,592,000 of them, so the planning costs roughly 0.6 joules against 17.8 joules of actuation.
Compute is about 3 percent of that task, and the motors outweigh it thirty to one. The published figure for a mobile robot puts autonomy compute at 37 percent and its motors at 17, so across two bodies the compute share differs by more than a factor of ten. A whole-robot energy ceiling has to name the machine it was computed for. That is a narrowing of something this Institute had stated more generally than the evidence supports, and it came from taking the measurement rather than from re-reading the sources.
Read it as an upper bound on the compute half: it prices a general-purpose processor doing the planning, which is the machine this whole page is about escaping. An embedded part, a fixed-function sampler or a settling fabric each land lower, and the distance is exactly what a substrate has to buy. The compute figure is measured; the actuation figure is modelled from coefficients calibrated to a measured 71.5 joule reach, so the two are printed as two numbers rather than one total.
Why this Institute works on it. Energy, compute and physical AI are one subject, and we think the productive order is energy first: start from the joules a task costs, let that choose the physics, and let the physics choose the machine. Almost everything above was found by following that order rather than by asking which model is most accurate. A landscape is a physical object. Settling is a physical process. Once you are willing to say that out loud, the hardware question stops being speculative and becomes a schedule.
Compiled 2026-08-16. Every arXiv identifier here was resolved against the arXiv API and every DOI against Crossref. Each work carries where it stands on the road from an idea to a product, so you can see how far the field has come and where the frontier is this month.
Read TR-2026-40 →Try it: width against depthWho is building these machinesThermodynamic computing