Computational Philosophy | Mahyar's world 🌏

The Two Arrows of Time Inside a Video Model

Mon, 25 May 2026 00:00:00 +0000

A small word that bothered me

I noticed something the other night that I cannot let go of. It started as a tiny single sentence in a paper I had read past twice before. The paper was on a system called Vid2World, and the sentence said, in a friendly tone, that the model “causalizes” a pretrained video diffusion backbone. This word stopped me.

What does it mean to causalize something? It basically implies the thing was, in some sense, not causal before. And the thing in question was a video diffusion model, the kind of model that produces videos of cats falling off shelves and astronauts riding horses. Sora is one of these. Cosmos is one. CogVideoX is one. We have spent the last two years insisting these models are world simulators. And here was a paper, in 2025, telling us that the first thing you have to do to turn one of them into an interactive world model is to teach it which way time runs.

I want to take this seriously. I want to take it as a sign that something is structurally interesting about how time lives, or fails to live, inside these models. So this post is my attempt to sit with the puzzle carefully. The math will be light but real. The thinking, I hope, is the point.

Two arrows inside the same model

Let me start by being precise. There are two distinct axes inside a video diffusion model that both deserve to be called “time,” and confusing them is the first mistake to avoid.

The first axis is the noising axis. In a diffusion model we take a clean sample $x_0$ and corrupt it with Gaussian noise across $k = 0, 1, \ldots, K$ steps, ending at pure noise $x_K$. We then train a network to reverse this process, mapping noise back to data. This noising axis has a strict arrow built into its definition. Going forward means adding noise. Going backward means removing it. The two directions are not the same operation, and the reverse process is the only one the network has to learn. The math, going back to Anderson’s 1982 result on time-reversal of stochastic differential equations, makes the asymmetry explicit. Forward and reverse are mirror images, but not symmetric ones.

The second axis is the content axis. A video is a sequence of frames $x = (x^{(1)}, x^{(2)}, \ldots, x^{(T)})$ that, in the real world, were captured in temporal order. By convention, $x^{(1)}$ is “earlier” and $x^{(T)}$ is “later.” When a video diffusion model receives a noised video and tries to denoise it, the natural question is this: when computing the update for $x^{(t)}$, which other frames is the network allowed to see?

The default answer in 2024 and 2025 is, all of them. Standard video diffusion transformers use full bidirectional attention over the temporal axis. When the network refines its estimate of frame $x^{(3)}$, it attends to $x^{(1)}, x^{(2)}, x^{(4)}, \ldots, x^{(T)}$ without distinction. From the model’s point of view, all frames are present at once. They are just different positions on a grid, no more directional than the rows and columns of an image.

I find it striking that this should ever have felt normal. A movie has an arrow of time. The water that fell from the glass does not jump back into it. The cigarette burns down rather than up. And yet the architecture we use to model movies treats time as if it were a spatial coordinate.

Why this seemed fine for a while

For a while, this did not seem to matter. The training data has an arrow of time. The cats fall off shelves; they do not fly up onto them. So the model learns to generate samples that go forward, statistically. If you ask it for a video starting from a still image, it produces a plausible forward continuation. The asymmetry of the data leaks into the asymmetry of the samples.

I want to grant this immediately. A bidirectional video model does, in practice, produce videos that go forward in time. The arrow of time is not lost. It is in the data, and the model inherits it through samples. This is exactly analogous to how a Boltzmann machine trained on natural images can produce natural images even though the model itself is undirected. The data tells the model which direction to lean.

But “leans in the right direction on average” is a much weaker property than “has an arrow of time as part of its structure.” And the difference, I will argue, is exactly what separates a video generator from a world model.

What variational free energy quietly assumes

Here is where, if you have spent time with active inference or with predictive coding, you will recognize an old friend.

Variational free energy, in the Friston tradition, is defined against a generative model. Concretely, you assume hidden states $s_{1:T}$ evolving over time, observations $o_{1:T}$ produced from those hidden states, and a generative model that factors like this:

$$ p(s_{1:T}, o_{1:T}) = p(s_1) \prod_{t=2}^{T} p(s_t \mid s_{t-1}) , \prod_{t=1}^{T} p(o_t \mid s_t). $$

This is a directed graphical model. It is a Bayesian network. It has an explicit, forward-running Markov chain in time. The transition kernel $p(s_t \mid s_{t-1})$ specifies what causes what, and the arrow goes one way.

Against this generative model, we minimise variational free energy,

$$ \mathcal{F}[q] = \mathbb{E}_{q(s)}\left[\log q(s) - \log p(s, o)\right], $$

where $q(s)$ is an approximate posterior over the hidden states. A short calculation rewrites this as

$$ \mathcal{F}[q] = \mathrm{KL}\left[q(s) , | , p(s \mid o)\right] - \log p(o), $$

so $\mathcal{F}$ is an upper bound on the negative log-evidence and is tight exactly when $q$ matches the true posterior. The KL divergence here is asymmetric on purpose. $\mathrm{KL}[q | p]$ and $\mathrm{KL}[p | q]$ are different objects, and choosing the first picks out the mode-seeking variational form that gives us the variational autoencoder, predictive coding networks, and the whole Bayesian-brain story.

I want to draw attention to something that, in my experience, often gets blurred. The KL asymmetry is not what gives active inference its arrow of time. The KL is a choice about how to fit $q$ to $p$. The arrow of time comes from the structure of $p$ itself, from the directed factorisation of the generative model. If you replaced $p$ with an undirected joint over $s_{1:T}$, say a Markov random field or a Boltzmann machine over frames, you would still have an asymmetric KL, but you would have no notion of “the next state given the current one.” You would just have a joint blob with no preferred direction.

This is the structural mismatch I want to put on the table. A bidirectional video diffusion model parameterises a joint $p(x^{(1)}, \ldots, x^{(T)})$ in a way that does not commit to the forward chain rule decomposition

$$ p(x^{(1)}, \ldots, x^{(T)}) = \prod_{t=1}^{T} p\left(x^{(t)} \mid x^{(<t)}\right). $$

Both kinds of model can describe the same joint distribution mathematically. The two are not statistically distinguishable in their support. But only the directed factorisation is what a world model needs to plug into a planner, a controller, or an agent.

Pseudo-likelihood, said carefully

Let me try to make this technical point feel less abstract.

Imagine I train a model that, for each frame index $t$, learns

$$ \hat{p}\left(x^{(t)} \mid x^{(\neq t)}\right). $$

That is, given all other frames, predict the missing one. This is a powerful self-supervised objective. It is also, in the classical statistical sense, a pseudo-likelihood. A celebrated result of Besag is that, in many practical cases, fitting a model by summing these conditional pseudo-likelihoods recovers something close to the true joint distribution, even though the individual conditionals were never explicitly stitched together by the chain rule of probability.

A bidirectional video diffusion model is doing something philosophically close to this. The denoising score $\nabla_{x} \log p_k(x)$ at each noise level $k$ is computed with bidirectional attention, so the score for frame $t$ depends on every other frame. The model defines, implicitly, a joint distribution over the entire clip, but the joint is most naturally accessed through “all conditioned on all” updates rather than through a one-step-at-a-time forward rule.

Now ask the world-modeling question: what is $p(x^{(t+1)} \mid x^{(\leq t)})$? That is, what does the model predict for the next frame, given only past frames? For a bidirectional model, this is not a quantity it has been trained to express directly. You can extract it by sampling the joint conditional on the past frames, but in practice that means running a full reverse-time chain over the entire window each time you want to advance one frame, and you also pay for the fact that the model never saw, at training time, the deployment regime where some frames are clean and some are still noised. The mismatch is severe.

A causal video model, by contrast, parameterises exactly this quantity. Frame by frame, it answers “given everything I have seen, what comes next?” That is the question a world model has to answer. That is the question an agent has to answer in order to plan. Active inference, model-based reinforcement learning, and any closed-loop controller need a forward-Markov transition. They need a model that knows where “now” is.

A short detour through the second law

I find it irresistible to draw a physics analogy here, but I want to draw it carefully, because the analogy can mislead as much as it can illuminate. I should also flag, before going any further, that I am not a physicist. I am the sort of person who has read enough about statistical mechanics to know what the second law of thermodynamics is, and not nearly enough to know what it is not, which is precisely the demographic that makes real physicists reach quietly for their pens. If one of them is reading, I beg a small amount of grace. I will try to deserve most of it.

In statistical mechanics, the microscopic equations of motion (Newton’s, Schrödinger’s, the underlying ones) are time-reversal symmetric in their core form. Yet the macroscopic world has a glaring arrow of time. The standard story has two ingredients. First, we start from a special initial condition, namely an early universe of unusually low entropy. Second, we coarse-grain. We do not track every molecule. When you average over the microstates compatible with each macrostate, the dynamics looks irreversible even though the underlying laws are not.

The arrow of time, in this story, is not in the laws. It is in the boundary condition and the coarse-graining.

Compare this to a bidirectional video model. The architecture is, in a precise sense, time-symmetric: it treats past and future positions as equivalent grid coordinates. The training data has an arrow of time, by virtue of being captured by cameras pointed at a real world that obeys the second law. The samples acquire an arrow of time by virtue of the data. But the model has no internal commitment to a direction. It can be asked to fill in the past from the future just as readily as the reverse, and it answers both equally well.

A causal model, by contrast, has the arrow baked into its parameterisation. It cannot be asked, even in principle, to fill in the past from the future without inverting the model in some external way. This sounds like a limitation. It is also exactly what makes a model behave like an agent embedded in time. To stand at a moment, to have only the past as evidence, to face a future that has not yet happened: this is what it means to be inside time rather than outside it.

The causalization surgery, philosophically

With all of this in mind, look again at what Vid2World actually does. The paper performs three kinds of surgery on a pretrained bidirectional video diffusion model.

First, it converts bidirectional temporal attention into causal temporal attention by adding a triangular mask. The transformer’s attention is no longer allowed to look at future frames when refining the current one. This costs no parameters. It is purely a mask.

Second, it converts temporal convolutions into causal convolutions by transferring weights from a kernel that originally used both past and future positions into a kernel that uses only past positions. The paper introduces a clever “extrapolative weight transfer” trick to make this transfer smooth, but the conceptual operation is just, redistribute the kernel’s weights so the future-facing taps become past-facing.

Third, it changes the training objective. Instead of corrupting every frame in a clip with the same noise level, it uses Diffusion Forcing, which draws an independent noise level per frame. This exposes the model to the autoregressive deployment regime, in which past frames are clean and future frames are still noised.

Each of these is a small technical adjustment. Taken together, they are a philosophical re-installation. The model is being told, retroactively, that there is a “now,” that the past has happened and the future has not, and that its job is to predict forward.

This is what I mean by saying that we hand the model an arrow of time it did not earn from data. The data had an arrow, yes. But the architecture refused to commit to one, and we had to install the commitment by hand.

Is the famous question even the right one?

I have come to think that the much-debated question “is Sora a world model?” is partly the wrong question, because it conflates two different things.

The first thing is, does Sora’s joint distribution over videos contain a lot of knowledge about how the world behaves? Almost certainly yes. The samples obey 3D consistency, object permanence, basic physics, and a great many subtler regularities that nobody wrote down. The model has learned, in some sense, the statistics of the visible world.

The second thing is, does Sora know which way time runs for it? Does it have a “now”? Can I ask it the question every agent has to answer, which is, given everything that has happened, what comes next? Here the answer is murkier. The model can be coaxed into producing forward continuations of an initial image, but the architecture itself does not parameterise a forward-Markov chain. The arrow of time is in the data, and in the prompt convention. It is not, structurally, in the model.

I will not pretend to know what to do with this distinction. But I think it is the right cleavage to draw. A model can be statistically aware of a world without being temporally embedded in one. Sora, taken architecturally, is in the first category. Vid2World is an attempt to drag it into the second.

What I am still chewing on

A few questions this puzzle leaves me with, that I cannot answer tonight.

Can the arrow of time emerge? At sufficient scale, could a bidirectional model effectively become forward-only? That is, does the data’s asymmetry get so deeply absorbed that bidirectional inference collapses onto causal inference in practice? I do not know of an experiment that has tried to measure this directly. I would love to see one.
Are the two asymmetries related? Is the KL asymmetry inside variational free energy structurally analogous to the data-induced asymmetry in a bidirectional model, or are they independent? My current guess is that they are independent. The KL asymmetry tells you how to fit $q$ to $p$. The temporal asymmetry tells you how $p$ is factored. But the analogy is suggestive enough that I want to think about it longer.
Is there a “principle of least action” for video models? In physics, the arrow of time emerges from low-entropy initial conditions plus a variational principle. Is there an analogous setup for video models in which the arrow of time is the boundary condition (the prompt, the initial frame) and the variational principle is the diffusion training objective? If something like this is true, then causalization may not be surgery at all but a natural consequence of choosing the right variational structure.
What about non-causal active inference? Active inference has, to my knowledge, always been derived against forward-Markov generative models. Has anyone tried to derive it against an undirected joint, that is, against a generative model that does not factor in time? If you wrote $p(s_{1:T})$ as a Markov random field over frames and asked what minimising $\mathcal{F}[q]$ looks like, what does the resulting “agent” do? Is there even a notion of policy? I think this is a tractable theoretical question, and I have not seen it asked.

A small parting thought

Part of what makes this puzzle hold me is that it turns a philosophical question into something concrete. Time, in the way philosophers usually talk about it, is hard. Block universe, growing block, presentism. These are positions one takes in seminar rooms.

But a video diffusion model with bidirectional attention is, in some quite literal sense, a block-universe device. It sees the whole clip at once and treats every frame as equally present. A causal video model with a forward-Markov chain is, in some quite literal sense, a presentist device. It only knows the past, and it has to predict its way into the future one step at a time. We have been building both kinds, more or less unintentionally, and we are now starting to notice that they are not the same thing.

I keep coming back to one image. A bidirectional model is something like a librarian who has read every book on the shelf and can tell you, when asked, what happens on page 73 of a novel given the contents of pages 1 to 72 and pages 74 to 200. A causal model is something like a reader, sitting at page 72, who only knows what has already happened and has to guess what comes next. These are very different epistemic positions. They produce different kinds of knowledge. They feel different from the inside, if a model can be said to have an inside.

It is not obvious to me that we want a librarian when we say we want a world model. It might be that the world model we have been chasing, all along, is the reader.

That, I think, is what kept me up.

References, if you want to chase these threads further: the Vid2World causalization recipe is in Huang et al., arXiv:2505.14357 (ICLR 2026). Sora’s “video generation models as world simulators” report is the OpenAI technical post from February 2024. The most accessible derivation of variational free energy is still Rafal Bogacz’s 2017 tutorial in the Journal of Mathematical Psychology, and the original “is RL the same thing as active inference?” question is laid out in Friston et al., PLoS ONE 2009. The reverse-time SDE result I lean on is Anderson, Stochastic Processes and their Applications 1982.

Kalman Filtering in the Age of PyTorch: State Estimation, Differentiability, and the Philosophy of Uncertainty

Wed, 08 May 2024 00:00:00 +0000

Introduction

The Kalman filter, a paragon of recursive estimation, has long stood at the intersection of mathematics, engineering, and epistemology. Conceived in the 1960s to address the challenges of navigation and control in aerospace, its recursive structure and optimality under Gaussian assumptions have made it indispensable across robotics, signal processing, finance, and beyond. Yet, as machine learning frameworks like PyTorch have redefined the computational landscape, the Kalman filter finds itself in a new context—one where differentiability, GPU acceleration, and integration with deep neural architectures are not just desirable, but essential.

In this blog post I want to embark on a dual journey. On one hand, I want to delve into the technicalities of implementing Kalman filters in PyTorch, leveraging its tensor operations and automatic differentiation to enable new research and applications. On the other, I want to reflect on the philosophical questions about the nature of uncertainty, the meaning of optimality, and the evolving relationship between model-based and data-driven approaches. By weaving together rigorous mathematics, practical coding insights, and reflective inquiry, we aim to illuminate both the power and the limitations of state estimation in the age of neural computation.

The Mathematical Foundations of Kalman Filtering

The State-Space Model: Dynamics and Observations

At the heart of the Kalman filter lies the state-space model, a mathematical abstraction that describes the evolution of a system’s hidden state over time and its relationship to noisy observations. Formally, the discrete-time linear state-space model is given by:

$$ \begin{aligned} x_{k} &= F_{k} x_{k-1} + B_{k} u_{k} + w_{k} \\ z_{k} &= H_{k} x_{k} + v_{k} \end{aligned} $$

Where:

$x_{k}$: State vector at time $k$
$F_{k}$: State transition matrix
$B_{k}$: Control input matrix
$u_{k}$: Control vector
$w_{k}$: Process noise $\sim \mathcal{N}(0,Q_{k})$
$z_{k}$: Observation vector
$H_{k}$: Observation matrix
$v_{k}$: Observation noise $\sim \mathcal{N}(0,R_{k})$

This model encodes two key assumptions: linearity and Gaussianity. The linearity allows for closed-form recursive updates, while the Gaussianity ensures that all conditional distributions remain Gaussian, making the mean and covariance sufficient statistics for the state estimate.

Recursive Estimation: Prediction and Update

The Kalman filter operates in two alternating steps: prediction (time update) and correction (measurement update). In the prediction step, the filter projects the current state estimate forward in time, using the system dynamics:

$$ \begin{aligned} \hat{x}_{k|k-1} = F_{k} \hat{x}_{k-1|k-1} + B_{k} u_{k} \\ P_{k|k-1} = F_{k} P_{k-1|k-1} F_{k}^{T} + Q_{k} \end{aligned} $$

Here $\hat{x}_{k|k-1}$ is the predicted state mean, and $P_{k|k-1}$ is the predicted state covariance.

In the update step, the filter incorporates the new measurement $z_{k}$ to refine the state estimate:

$$ \begin{aligned} K_{k} &= P_{k|k-1} H_{k}^{T} \left( H_{k} P_{k|k-1} H_{k}^{T} + R_{k} \right)^{-1} \\ \hat{x}_{k|k} &= \hat{x}_{k|k-1} + K_{k} \left( z_{k} - H_{k} \hat{x}_{k|k-1} \right) \\ P_{k|k} &= \left( I - K_{k} H_{k} \right) P_{k|k-1} \end{aligned} $$

Where $K_{k}$ is the Kalman gain, which determines how much the measurement should be trusted relative to the prediction. Its derivation is rooted in the minimization of the mean squared error of the state estimate, balancing the uncertainty in the prediction and the measurement

The Geometry of Uncertainty: Covariance Propagation

A subtle yet profound aspect of the Kalman filter is its treatment of uncertainty. The covariance matrices $P_{k|k-1}$ and $P_{k|k}$ encode not just the spread of possible states, but also the correlations between different state variables. The propagation of covariance through the system dynamics involves the transformation:

$$ P_{k|k-1} = F_{k} P_{k-1|k-1} F_{k}^{T} + Q_{k} $$

This operation reflects how uncertainty “flows” through the linear transformation $F_{k}$, and how process noise $Q_{k}$ injects additional uncertainty. The measurement update, in turn, reduces uncertainty by incorporating information from the observation, as modulated by the Kalman gain.

Understanding the covariance as a bilinear form, rather than just a matrix, reveals the deep connection between the algebra of estimation and the geometry of probability distributions. This perspective is crucial for appreciating the filter’s optimality and for extending it to more complex, nonlinear, or high-dimensional settings.

Kalman Filtering Meets PyTorch: Implementation and Differentiability

Why PyTorch? Beyond Deep Learning

PyTorch, originally designed for deep learning, offers a flexible tensor computation library with automatic differentiation and seamless GPU acceleration. While its primary use case has been neural networks, its capabilities make it an attractive platform for implementing classical algorithms like the Kalman filter. The motivations are manifold:

First, PyTorch’s tensor operations enable efficient batch processing, which is invaluable when filtering multiple signals or running ensembles of filters in parallel. Second, the autograd engine allows for differentiable programming, making it possible to optimize filter parameters or integrate the filter as a module within a larger neural architecture. Third, PyTorch’s ecosystem encourages modularity, extensibility, and integration with probabilistic programming frameworks such as Pyro.

Coding the Classical Kalman Filter in PyTorch

Implementing the Kalman filter in PyTorch involves translating the recursive equations into tensor operations. Consider the following minimal implementation for a batch of signals:

import torch
from torch import nn
from torch.linalg import inv
class KalmanFilter(nn.Module):
"""Kalman Filter implementation for state estimation in linear dynamic systems.

 Attributes:
 F (Tensor): State transition matrix.
 B (Tensor): Control input matrix.
 H (Tensor): Observation matrix.
 Q (Tensor): Process noise covariance.
 R (Tensor): Observation noise covariance.
 state_dim (int): Dimensionality of the state.
 """
def __init__(self, F, B, H, Q, R, state_dim):
super().__init__()
self.F = F.clone()
self.B = B.clone()
self.H = H.clone()
self.Q = Q
self.R = R
self.state_dim = state_dim
# placeholders for the current state, covariance, observation and control
self.x = None # [state_dim, 1]
self.P = None # [state_dim, state_dim]
self.zs = None # [obs_dim, 1]
self.us = None # [control_dim, 1]
def project(self):
"""Projects the state and covariance forward."""
x_pred = torch.matmul(self.F, self.x) + torch.matmul(self.B, self.us)
P_pred = torch.matmul(self.F, torch.matmul(self.P, self.F.T)) + self.Q
return x_pred, P_pred
def correct(self, x_pred, P_pred):
"""Corrects the state estimate with the current observation."""
S = torch.matmul(self.H, torch.matmul(P_pred, self.H.T)) + self.R
K = torch.matmul(P_pred, self.H.T) @ inv(S)
# state update
self.x = x_pred + torch.matmul(K, (self.zs - torch.matmul(self.H, x_pred)))
# covariance update
I = torch.eye(self.state_dim, device=P_pred.device)
self.P = torch.matmul((I - torch.matmul(K, self.H)), P_pred)
def forward(self, zs, us):
"""
 Processes a batch of observation/control sequences.

 Args:
 zs: [timesteps, batch, obs_dim] sequence of observations
 us: [timesteps, batch, control_dim] sequence of control inputs
 Returns:
 xs: [batch, state_dim, timesteps] filtered state estimates
 pred_obs: [batch, obs_dim, timesteps] one-step predictions of observations
 residuals: [batch, obs_dim, timesteps] observation residuals
 """
xs = []
pred_obs = []
residuals = []
# initial state & covariance
self.x = torch.zeros((self.state_dim, 1), device=zs.device)
self.P = torch.eye(self.state_dim, device=zs.device)
# iterate over time
for z_t, u_t in zip(zs.transpose(0, 1), us.transpose(0, 1)):
self.zs = z_t.unsqueeze(1)
self.us = u_t.unsqueeze(1)
x_pred, P_pred = self.project()
self.correct(x_pred, P_pred)
xs.append(self.x.detach().clone())
y_pred = torch.matmul(self.H, x_pred)
pred_obs.append(y_pred)
residuals.append(self.zs - y_pred)
xs = torch.cat(xs, dim=1)
pred_obs = torch.cat(pred_obs, dim=1)
residuals = torch.cat(residuals, dim=1)
return xs, pred_obs, residuals

Differentiable Kalman Filters: Learning and Optimization

One of the most transformative aspects of implementing the Kalman filter in PyTorch is the ability to make the entire filtering process differentiable. By treating the system matrices ($F$, $H$, $Q$, $R$) as learnable parameters, one can optimize them using gradient-based methods, either to fit data or to tune the filter for specific tasks. This approach blurs the line between classical estimation and machine learning, enabling hybrid models that combine the structure of state-space models with the flexibility of data-driven learning.

Recent research has focused on improving the efficiency of backpropagation through the Kalman filter. While PyTorch’s automatic differentiation can compute gradients, it may incur significant computational overhead, especially for large-scale problems. Novel closed-form expressions for the derivatives of the filter’s outputs with respect to its parameters have been developed, offering substantial speed-ups (up to 38 times faster than PyTorch’s autograd in some cases). These advances make it feasible to embed Kalman filters within deep learning pipelines, trainable end-to-end, and responsive to the demands of modern applications.

PyTorch Libraries for Kalman Filtering

Several open-source libraries have emerged to facilitate Kalman filtering in PyTorch:

torch-kf: A fast implementation supporting batch filtering and smoothing, capable of running on both CPU and GPU. It is particularly efficient when filtering large batches of signals, leveraging PyTorch’s parallelism.
DeepKalmanFilter: Implements deep variants of the Kalman filter, where neural networks parameterize parts of the state-space model. This enables modeling of nonlinear dynamics and observations, bridging the gap between classical filtering and deep generative models.
Pyro: A probabilistic programming framework that supports differentiable Kalman filters and extended Kalman filters, with learnable parameters and integration with variational inference.
torchfilter: Provides advanced filters such as the square-root unscented Kalman filter, supporting both state and parameter estimation in nonlinear systems.

Extensions and Hybrid Models: Beyond the Classical Filter

Nonlinear and Non-Gaussian Filtering

While the classical Kalman filter assumes linear dynamics and Gaussian noise, many real-world systems violate these assumptions. Extensions such as the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) address nonlinearities by linearizing the dynamics or propagating sigma points, respectively. Particle filters, in turn, approximate arbitrary distributions via Monte Carlo sampling.

Implementing these advanced filters in PyTorch follows the same principles: tensorized operations, differentiability, and integration with neural modules. For example, the EKF can be implemented by computing Jacobians using PyTorch’s autograd, while the UKF can leverage batched sigma point propagation for efficient parallelism.

Deep Kalman Filters and Latent Dynamics

The fusion of Kalman filtering with deep learning has given rise to deep Kalman filters, where neural networks parameterize the transition and observation functions. This approach enables modeling of complex, nonlinear, and high-dimensional systems, such as video sequences or sensor fusion in robotics. The deep Kalman filter retains the probabilistic structure of the classical filter but augments it with the representational power of neural networks.

In PyTorch, this is achieved by defining neural modules for the transition and observation models, and using the filtering equations to propagate means and covariances through time. The entire model can be trained end-to-end using stochastic gradient descent, with the Kalman filter acting as a differentiable layer within the network.

Hybrid Estimators: Neural Networks and Kalman Filters

Hybrid models that combine neural networks and Kalman filters have demonstrated superior performance in state estimation tasks, particularly in scenarios with complex dynamics or partial observability. These models can be categorized into two main types:

NN-KF: Neural networks learn the parameters or functions of the state-space model, which are then used by the Kalman filter for estimation.
KF-NN: The Kalman filter provides state estimates or uncertainty measures that are used as inputs or features for a neural network.

Such hybridization leverages the strengths of both approaches: the interpretability and optimality of the Kalman filter, and the flexibility and expressiveness of neural networks. In PyTorch, these models can be implemented as composite modules, trained jointly or sequentially, and deployed in a wide range of applications from battery state-of-charge estimation to autonomous navigation.

Philosophical Reflections: Uncertainty, Knowledge, and Learning

The Epistemology of State Estimation

At a deeper level, the Kalman filter embodies a philosophy of knowledge under uncertainty. It formalizes the process of updating beliefs in the face of incomplete and noisy information, balancing prior expectations (the model) with new evidence (the measurements). The recursive structure mirrors the Bayesian paradigm, where beliefs are continuously revised as new data arrives.

Yet, the filter’s optimality is contingent on its assumptions: linearity, Gaussianity, and known noise covariances. When these assumptions are violated, as is often the case in complex systems, the filter’s estimates may become biased or inconsistent. This raises fundamental questions: What does it mean to “know” the state of a system? How do we quantify and manage uncertainty? Can we trust our models, or must we adapt them in light of new evidence?

The Fusion of Model-Based and Data-Driven Approaches

The integration of Kalman filtering with PyTorch and neural networks reflects a broader trend in computational science: the synthesis of model-based and data-driven approaches. Classical estimation theory offers structure, interpretability, and guarantees of optimality. Machine learning provides flexibility, scalability, and the ability to discover patterns from data.

Hybrid models, differentiable filters, and end-to-end learning challenge the traditional dichotomy between “hard-coded” models and “black-box” learning. They invite us to reconsider the boundaries between theory and data, deduction and induction, certainty and doubt. In this sense, the Kalman filter is not just an algorithm, but a lens through which to explore the nature of inference, prediction, and adaptation.

The Philosophy of Differentiable Programming

The advent of differentiable programming—where algorithms are designed to be composed, differentiated, and optimized—raises new philosophical questions. When we make the Kalman filter differentiable, we enable it to learn from data, to adapt its parameters, and to participate in the broader ecosystem of neural computation. But we also introduce new forms of uncertainty: about the correctness of gradients, the stability of optimization, and the interpretability of learned models.

Is the differentiable Kalman filter still a Kalman filter, or has it become something new? What are the implications of treating classical algorithms as modules within a deep learning pipeline? How do we balance the desire for optimality with the need for flexibility? These questions invite ongoing reflection and experimentation.

Conclusion

The Kalman filter, once a symbol of control theory and aerospace engineering, has found new life in the era of PyTorch and machine learning. Its recursive structure, principled handling of uncertainty, and optimality under Gaussian assumptions remain as compelling as ever. Yet, its implementation and interpretation are evolving, shaped by the demands of differentiability, scalability, and integration with neural computation.

By exploring the mathematical foundations, practical coding strategies, extensions to nonlinear and hybrid models, and the deeper philosophical questions that arise, we have sought to illuminate both the enduring relevance and the transformative potential of Kalman filtering in the age of PyTorch. As we continue to blur the boundaries between model-based and data-driven approaches, the filter serves as a bridge—not just between past and future, but between certainty and doubt, theory and practice, knowledge and learning.

The journey of the Kalman filter is far from over. Its recursive dance of prediction and correction, its geometry of uncertainty, and its adaptability to new computational paradigms ensure that it will remain a central figure in the ongoing dialogue between mathematics, engineering, and philosophy. Whether as a standalone estimator, a differentiable module, or a component of a deep generative model, the Kalman filter challenges us to rethink what it means to know, to predict, and to learn.