Understand the latent space
How templates organize images
Go from pixels to some coordinates. Understand what they represent, how the models organize them and what changes when we move there.

- 01
Picture
784 pixels for a figure of 28 × 28.
- 02
Code
A few numbers to summarize this image.
- 03
Reconstructed image
Check what the code has preserved.
Understand what a model keeps from an image
An MNIST cipher contains 784 pixel values. However, we can describe it with less information: its shape, its inclination, the thickness of the line... A latent space is a more compact coordinate system, calculated from these pixels. A point represents an image.
My goal: link coordinates to images
I compare several ways of constructing this representation on numbers and fruits. What do they bring together? What details do they lose? Can we go back to pixels, then invent a new image? The journey starts from the axes of the PCA, goes through the prototypes, then arrives at the networks which learn to encode and decode.
- 01
One point
An encoded image.
- 02
Color
Its label, added to interpret.
- 03
Decoding
The concrete meaning of its coordinates.
A point, an image; a neighbourhood, similarities
In a two-dimensional map, each image becomes a point: its first coordinate gives the horizontal position, its second the vertical. The colors indicate the known number or fruit, only to read the result: these models learn without these labels.
Why also look at reconstructions?
Separate color groups suggest that the code distinguishes classes. Mixtures can come from similar shapes or from lost information. But a beautiful separation does not guarantee a faithful image. We must therefore read the map, the originals and the reconstructions together. With more than two coordinates, a 2D map only shows part of the representation. To build this map, it remains to choose the axes on which to place the images. This is the starting point of the PCA.
PCA: summarize images with fewer coordinates
PCA, or principal component analysis, chooses the axes of the map while retaining as much variation between images as possible. To understand this choice of directions, imagine a banana that you open to look for rot: a cut lengthwise exposes its extent more than a small slice. This analogy illustrates the idea of looking in the most informative direction first, before examining what other directions provide.
How PCA selects directions of variation
PCA centers the cloud of images around their mean, then searches for a first axis of maximum variance. The second is perpendicular to the first and maximizes the remaining variance; the following continue this principle. Each centred image is then projected onto the selected axes: these projections give its coordinates. The analogy helps choose a direction, but PCA does not look for rot and does not guarantee its detection. It maximizes projected variance, not any form of information.
The first two PCA axes
Each dot represents a number. The colors overlap: the maximum variance does not necessarily correspond to the best separation of classes.
The axes bring together silhouettes, not names
On MNIST, the colors overlap in the projection: two coordinates are not enough to distinguish all the numbers. On the fruits, cucumbers draw branches while apples and pears share part of the center. Switch between datasets to read these neighbourhoods.
Why these mixtures and these branches?
PCA processes pixel variations, without knowing “apple” or “7”. The silhouette and its orientation can contribute to the observed directions; you have to go back to the images to verify this. PC1 is therefore not a universal “size” axis. In the Fruits figure, the crosses are codes generated outside the cloud: being able to choose coordinates does not mean that their decoding will be plausible.
Original / PCA on the test
In each pair, the original precedes the reconstruction. This notebook retains enough components for 75% variance: it uses more than the two axes of the map.
More coordinates, more details retrieved
To reconstruct, we start from the average image and add the patterns of the axes, weighted by the coordinates of the code. The Fruits series shows the transition from a blurred mass to a silhouette then to details. Each additional coordinate restores a variation that the previous ones did not describe.
Why are two dimensions not enough?
Two numbers allow you to draw a map, but strongly compress an image. The ratio reaches approximately 90% variance with 87 components on MNIST and 43 on fruits at 64 × 64 pixels. These are not recognition rates. Between two codes, PCA linearly mixes the reconstructions: it can superimpose contours. Now let's look for another summary: typical images.
- 01
Assign
Compare the pixels at the K centers.
- 02
Average
Recalculate each prototype with your group.
- 03
Code
Store the number of the chosen prototype.
Replace an image with its closest prototype
Instead of placing each image on axes, K-means learns K typical images, called centroids. It assigns each image to the nearest center, then replaces each center with the average of the images in its group. These two steps are repeated until stabilization.
What changes in the representation
The code here is a group number: for example, “prototype 12”. To reconstruct, we find the average image stored at this number. All images in the same group therefore have the same reconstruction. Unlike the PCA, this code is discreet: the number 13 is not necessarily closer to 12 than the number 80.
10 prototypes learned
The bottom shows the ten centers. Some numbers are repeated and others are missing. The center is an average of images, not a learned label.
10, 50, 250 centers: what do we gain?
With 10 centers on MNIST, the prototypes roughly summarize the scripts and do not necessarily cover all ten digits. At 50 then 250, they can represent more variations. The report also tries 500 fruit centers: some describe a round apple, others an elongated cucumber or a pear.
Why are there still blurry images?
A center averages pixels of similar but not identical shapes. Staggered lines become blurred; numbers like 4 and 9 can share a group. Increasing K gives more prototypes, without guaranteeing one group per class. The displayed grid is an arrangement of centers, not a plan where neighbours are similar. This is precisely what Kohonen will seek to organize.
- 01
Find the winner
The prototype closest to the image.
- 02
Adapt the neighbourhood
Also move neighbouring squares.
- 03
Organize the grid
Make shape transitions more local.
Give a neighbourhood to the prototypes
A Kohonen map, or SOM, arranges the prototypes on a grid. For each image, it finds the most similar neuron, then brings its prototype and those of its neighbours closer to this image. Throughout the passages, areas of similar shapes appear.
Why also move the neighbours?
K-means adjusts each center independently. Kohonen links their evolutions thanks to the proximity of the grid: two close boxes are encouraged to describe similar shapes. The code of an image becomes the position of its winning box. We rebuild with its prototype; we do not yet have a learned decoder capable of freely transforming any point of the plan into an image.
Map of Kohonen · 15 × 15
225 positions and their prototypes, after testing 22 epochs on 5,000 images. Follow the areas of neighbouring loops, lines and numbers: this time, the neighbourhood is part of the learning.
The grid is the representation to look at
On the MNIST card of 15 × 15 squares, follow the areas of loops, vertical lines and shapes close to 4 or 9. On the Fruits card of 10 × 10, locate the elongated silhouettes then the rounder shapes. Each thumbnail is the prototype of a position on the map.
What the card allows, and its limit
The neighbourhoods give a reading of the resemblances that the simple K-means number did not give. They remain imperfect: borders, repetitions and ambiguous prototypes are visible. The size of the grid and the extent of the neighbourhood influence this trade-off. To move beyond choosing a box and learning true continuous code, let's move on to autoencoders.
- 01
Encoder
Image → z code.
- 02
Latent bottleneck
Two numbers in the cards shown here.
- 03
Decoder
z code → pixels, compared to the original.
Learn to compress and rebuild together
An autoencoder passes the image through an encoder, which produces some numbers, then through a decoder, which attempts to remake the image. We compare the reconstruction to the original and adjust the two networks to reduce the pixel error. The code becomes an obligatory passage for information.
What changes compared to the PCA
PCA applies a common linear projection. Network activations allow the autoencoder to learn a nonlinear transformation: more complex relationships can fit in the code. Unlike Kohonen, the coordinates are continuous and the decoder is learned. But the only reconstruction error does not require it to separate the classes, nor to make all the space usable.
Autoencoder · z₁ and z₂ codes
On the left, training; right, test. The color zones are found, but mix at their borders. The axes are learned and do not carry an imposed meaning.
Return to test figures
Original on the left, reconstruction on the right. The 4 here takes a shape close to a 9: reducing the average pixel error does not guarantee that the identity of the number will be preserved.
A space that folds to restore images
The maps show the codes for the training images on the left and the test images on the right. The colors occupy irregular areas, with overlaps. Look at the associated reconstructions: a proximity in the code must be interpreted through the shapes that the decoder renders.
Why are coordinates difficult to name?
The network invents combinations useful for its reconstruction; no one imposes an “incline” axis and a “thickness” axis on it. Several properties can vary together. In these tests, a tanh activation limits the code between −1 and 1. The structures of the test make it possible to examine whether the organization is found on images not used for learning.
Decode a coordinate grid
Each sticker corresponds to a code chosen in the plan. Compare stroke transitions and repeated shapes. The edges of this grid leave the −1 to 1 zone accessible to the tanh encoder.
An empty square on the map can give a strange image
These mosaics decode a coordinate grid, even where the encoder has not placed any images. Traversing a line varies one coordinate, traversing one column varies the other. Repetitions and degraded shapes reveal the limits of what the decoder has learned.
Why doesn't interpolation guarantee a good image?
The network was trained on the codes of training images, not on every point in the plane. A line between two codes can therefore cross a poorly learned region. Here, the explored grid even extends beyond the tanh encoder’s −1 to 1 bounds: some images are extrapolations. Adding dimensions helps preserve detail but makes the map less directly visible.
Convolutions · two-dimensional code
Report test: 100 × 100 images, convolutional layers then a two-dimensional tanh code, 50 epochs. The three colors largely overlap.
Images decoded in the same test
The originals on the left retain textures absent from the reconstructions on the right. A more specialized architecture does not eliminate the loss of the two-coordinate bottleneck.
Better read shapes before compressing them
Dense networks treat the image as a long list of pixels. Convolutions use small shared filters that run through neighbouring areas: local contours and patterns can thus be exploited before compression. The report tests this variant on fruit.
Why does the result remain unclear?
The map still represents a two-dimensional code: improving the encoder does not remove this narrow bottleneck. Reconstructions in this experiment mainly recover masses and silhouettes. Spatial reduction and mean pixel error can also soften details. This experiment does not establish that convolution is always better; it shows why architecture and code capacity should be considered together.
- 01
Encode
A mean μ and a dispersion σ.
- 02
Pull a code
z = μ + σ × Gaussian noise.
- 03
Learn
Reconstruction + weight × KL divergence.
Learn a probable region for each image
The autoencoder gives a fixed code. The VAE produces a mean μ and a dispersion σ, which define a code distribution for the image. During learning, we draw a code from this region and then decode it. The network must therefore reconstruct from several close codes.
Why add a constraint on distributions?
The loss combines reconstruction fidelity and KL divergence, which brings the code distributions closer to a Gaussian reference centred at zero. This encourages a more regular space to sample. The formula z = μ + σ × ε separates the average, the amplitude of the noise and a draw ε: it allows the network to be trained despite the random draw.
VAE · moyennes μ₁ et μ₂
The points show the averages, not the draws or their dispersion. Training on the left, testing on the right; the colors remain partially mixed.
VAE · rebuild test figures
Read each original/reconstruction pair. These examples illustrate the fidelity obtained in this essay; they are not sufficient to globally classify VAE and autoencoder.
Visible averages, distributions around
A point on these maps is the average μ calculated for an image; the dispersion around this average is not drawn. The colors still overlap: the VAE receives no instructions to separate the numbers or the fruits. Compare the map to its reconstructions, then to that of the autoencoder.
Why doesn't the cloud necessarily become a circle?
The KL constraint concerns the code distributions of each image. A cloud of their averages alone is not a drawing of all the codes sampled. Its shape also depends on the weight of the constraint and the reconstruction. A more regular space aims to facilitate exploration; it alone proves neither good images nor a separation of classes.
VAE · a grid of generated figures
The chosen position controls the decoded digit: look at the transitions between silhouettes. A regular mosaic of codes is an exploration of the decoder, not a batch of original images.
Moving from a code map to an image map
The mosaic shows what the decoder produces at different coordinates of the latent plane. We can follow the transformation of shapes by browsing the rows and columns. Unlike reconstruction, generating does not require starting from an image: we choose or draw a code, then we decode it.
Why do some transitions remain ambiguous?
Overlapping distributions can lead to intermediate forms; a loss of pixels can favor average contours. Regularization makes sampling more consistent with learning, without ensuring a sharp image everywhere. In the workshop, the marked point moves at the same time as the image is recalculated: we can directly observe this effect.
VAE · fruit, weight KL = 0.0001
The averages form irregular regions. The original legend “Digit” here corresponds to the three fruits. The low constraint leaves a lot of freedom for representation.
VAE · reconstructions, KL = 0.0001
The masses of the fruits are found with attenuated details. The decoder restores pixels from a compact code, learned with a probabilistic constraint.
Organizing more space has a cost
The report compares a weak KL constraint, weighing 0.0001, to a stronger constraint, weighing 0.01, on fruits. Watch how the maps change, then the rebuilds. You have to read both: bringing the distributions closer to the reference is not the same objective as preserving every detail.
Why do these goals compete with each other?
To recreate an image exactly, the network would like to assign it a distinct, low-noise code. KL penalises distributions that are too far from the reference or too concentrated. Its weight controls the balance. These trials illustrate the tension under the report’s protocol; they do not identify an optimal weight for every dataset.
- 01
Axes and prototypes
Summarize with a base or a dictionary.
- 02
Codes learned
Adapt the representation to the reconstruction.
- 03
Distributions
Encourage a space that can be sampled.
Five ways to summarize, five ways to reconstruct
The PCA stores contributions to axes. K-means chooses a prototype; Kohonen organizes these prototypes on a grid. The autoencoder learns a continuous code and its decoder. The VAE adds distributions and a constraint to better explore the possible codes.
The question to ask each experiment
What does a point represent, what defines its neighbours, and how do we get back to pixels? Then only: is the image faithful, are the transitions legible, have the explored regions been learned? Compare the same test images when the protocol allows. The original figures use different settings and resolutions: they explain the mechanisms, without constituting a single classification of the models.
An experience to explore in this chapter.
Connect a point, its code and its image
Choose MNIST or Fruits, then an image on the map. The target image, on the right, is another test example chosen from the list: it serves as the end point for the interpolation.
What you can do
The sliders are below the map and the three images: scroll the panel to reach them. Change just one setting at a time to better see its effect.
PCA · compare details and transitions
- Keep the same image and set “Components Retained” to 1, then 2, 16 and 64. Spot the details that reappear and compare the error displayed under the sliders.
- Then fix 16 components and move the interpolation from 0 to 50, then 100%. At the end of the journey, you obtain the reconstruction of the target image. Start again with 2 then 64 components to compare the intermediate images.
Autoencoder · explore around an image
- At 0% interpolation, move z₁ alone, then z₂ alone. Try a small move, then push the slider to one end. Look at what changes and what deteriorates as the cross moves away from the points on the map.
- Click “Revert to Initial Code,” then step through the interpolation to the target frame. Spot a gradual transition or an abrupt change. The back button resets the interpolation and displacements.
VAE · produce several variants
- Return to the initial code, then click “Sample” several times without changing the image. Compare the variants produced. Also try 50% interpolation to explore variations between the two images.
- After a draw, move z₁ or z₂: the noise drawn remains the same, which makes it possible to isolate the effect of the movement. “Return to initial code” also cancels this noise and recovers the decoding of the initial average.