Cracking Hilbert Space
Building on our previous discussions of L² norms and inner products, let’s dive into the mathematical foundation that makes signal processing actually work — and discover why Hilbert spaces are the perfect playground for Digital Signal Processing.
Note: Before we start our discussion, you may have to read my previous blogs:
👉 Signals are Vectors
👉 The L2 Norm and Inner Products
What We Actually Do in Signal Analysis
Here’s what most of the signal analysis work really boils down to: vector decompositions! Lots and lots of vector decompositions.
We take signals and break them down into components. Some examples (don’t worry, we’ll cover these later):
- Fourier series:
$$f(t) = \sum_{n=-\infty}^{\infty} c_n e^{i n \omega t}$$
- Wavelet decompositions:
$$f(t) = \sum_j \sum_k c_{jk} \psi_{jk}(t)$$
- Infinite impulse response filters: convolutions that integrate to infinity
Notice the pattern? We work in L² spaces (from our previous blog on L² norms), and we constantly deal with infinite series sums and integrations that expand to infinity.
That means… we deal with limits. We talk about going to infinity.
Once Upon a Time, Mr. Cauchy…
Once upon a time, a French mathematician named Augustin-Louis Cauchy noticed that people were working with limits in a pretty casual way. He didn’t like that mathematicians never asked: “What does it actually mean for a sequence to ‘approach’ a limit?”
Approaching a limit means progressing toward that number with infinite steps. That means we’re dealing with a sequence of values. Cauchy’s genius insight was:
“You don’t have to know the limit to see if a limit exists — you just need a sequence where terms are getting arbitrarily close to each other as it progresses.”
This is equivalent to saying something reaches limit L when the sequence index approaches infinity.
Important clarification: This doesn’t mean each pair of consecutive terms should be smaller than the previous pair. Instead, given any number ε (no matter how small), you can always find two indices m and n in the sequence where the distance between the mth and nth terms is less than ε.
Such sequences are called Cauchy sequences.
The Mathematical Definition
Cauchy Sequence: A sequence where terms get arbitrarily close to each other (not necessarily to some known target).
Mathematical notation: A sequence ${x_n}$ is Cauchy if:
$$\forall \varepsilon > 0, \exists N \text{ such that } \forall m,n \geq N: |x_m - x_n| < \varepsilon$$
Key insight: Every converging sequence is automatically Cauchy. When you say a limit exists, the whole sequence must be Cauchy.
Quick check:
👉 Is a wiggly sine wave that’s exponentially getting smaller a Cauchy sequence? Yes
👉Is e⁻ⁿ a Cauchy sequence? Yes
Spaces and the Problem of Holes
We can set mathematical rules and define entities within those rules. The collection of objects obeying those rules forms a “Space” defined by those rules. From our previous blog, we defined the L² norm as generalized length, giving us L² spaces.
We also have simpler spaces, like the set of rational numbers.
Here’s the problem: For some spaces, certain limits don’t make sense. The limit can exist mathematically, but the limit value L doesn’t obey the space’s rules. Such limits are therefore not in the space.
In other words, there can be Cauchy sequences in a space where the convergence limit doesn’t belong to that space.
A Simple Example: The Hole in Rational Numbers
Consider computing √2 using only rational numbers (Each xₙ is an approximation to √2 but with 1 more decimal place added to the end):
x₁ = 1 (error: |1² — 2| = 1)
x₂ = 1.4 (error: |1.4² — 2| = 0.04)
x₃ = 1.41 (error: |1.41² — 2| = 0.0119)
x₄ = 1.414 (error: |1.414² — 2| = 0.000604)
x₅ = 1.4142 (error: |1.4142² — 2| = 0.000038)
…
This sequence of rational numbers is clearly “trying” to converge — each term gets closer to the previous ones. The differences |xₙ₊₁ — xₙ| are getting smaller and smaller.
But here’s the problem: √2 is not a rational number! This sequence converges to a point that’s not within the rational number space. There’s a hole where the limit should be.
The sequence inches toward a limit, each step moving forward within the space, but at infinity it reaches a “hole.”
Such spaces are called “incomplete”. When the limit of a Cauchy sequence doesn’t exist within the space, it becomes incomplete. The rational numbers form an incomplete space.
Why Should We Care About Completeness?
Remember our L² norms and inner products from previous discussions? Since you have the freedom (through mathematical axioms) to choose your L² norm, you might sometimes create L² norm spaces that are incomplete.
Same with inner products. You may define your inner product operation (satisfying the inner product axioms) but end up with an incomplete space.
Even more fundamentally: The incompleteness can arise from the nature of your underlying vector space itself. You might choose a vector space that simply doesn’t contain all the “limit points” of inner products that your sequences are trying to reach, regardless of how you define norms or inner products on it.
Why does this matter? Because these inner product operations usually involve infinite sums or integrals with infinite limits. In incomplete spaces, these may not make sense — your limits don’t exist.
The catastrophe: Your signal decomposition (this is all about inner products) into infinite components becomes meaningless. Your L² norms from inner product operations don’t have limits in your own space.
Enter the Hilbert Space
A Hilbert space is simply an inner product space that is complete.
Definition: A Hilbert space is a complete inner product space — meaning every Cauchy sequence converges to a point within the space.
What this gives us:
- All the geometric intuition from inner products (angles, orthogonality, projections)
- All the measurement capabilities from norms (distances, lengths)
- Completeness: No holes! Every infinite sum that should converge actually does converge to something in the space
Why Hilbert spaces are perfect for DSP:
- Fourier analysis works: Infinite Fourier series converge to actual signals in the space
- Optimization algorithms converge: When you minimize error functions, you reach real solutions
- Signal reconstruction is guaranteed: Sampling and reconstruction theorems have solid foundations
- Filter design is reliable: Iterative design algorithms converge to achievable filters, etc…
The Bottom Line
In signal processing, we constantly work with:
- Infinite series decompositions
- Optimization algorithms that should converge
- Approximation methods that should reach real signals
Hilbert spaces provide the mathematical foundation that ensures all these processes actually work. They combine the geometric elegance of inner product spaces with the analytical reliability of complete spaces.
No holes, no mathematical ghosts, no convergence to empty space — just robust, reliable mathematical machinery that makes modern signal processing possible.
Final thought: The next time someone casually mentions that “your quantum state exists in a Hilbert space,” you’ll know exactly what they mean — it’s living in a mathematical universe where infinite sums actually make sense, limits exist where they should, and there are no holes for your physics to fall through!
Fun Historical Facts
Plot twist: Hilbert didn’t actually invent “Hilbert spaces”! Here’s the real story:
- Cauchy (early 1800s): “Let’s understand what convergence really means” → invented Cauchy sequences
- Hilbert (late 1800s/early 1900s): “Let’s work with infinite-dimensional spaces with geometric structure” → developed theory of infinite-dimensional inner product spaces
- Later mathematicians like John von Neumann (1920s-1930s): “Let’s combine these ideas” → formalized what we now call Hilbert spaces
So Hilbert gets the naming credit for his groundbreaking work on infinite-dimensional inner product spaces, but the complete modern definition is really a collaborative mathematical achievement.
This completes our journey from vector spaces → L² norms → inner products → Cauchy sequences → completeness → Hilbert spaces. Next time, we’ll explore how these abstract concepts manifest in concrete Digital Signal Processing applications and Quantum Mechanics applications.