The normal distribution is everywhere in machine learning. Off of the top of my head, we use it for

• Modeling measurement error and uncertainty (Gauss’s starting point)
• A popular choice for the prior of a latent distribution
• Sampling noise for methods such as autoencoders and GANs
• As a target distribution for learned representations, most recently seen in (Balestriero & LeCun, 2025).

among many others. Our goal today is to look at the multivariate ($n$-dimensional) normal distribution in high-dimensional space.

Recalling the Gaussian

The normal (or Gaussian) distribution $\mathcal{N}(x; \mu, \sigma^2)$ for $x \in \mathbb{R}$ is defined with the following probability density function in the single variable case:

We usually drop the $x$ part and write $\mathcal{N}(\mu, \sigma^2)$.

Visually, we know it more commonly as the “bell curve”:

Note that while the probability decreases exponentially with distance from the mean, the possible values for $x$ technically range from $-\infty$ to $\infty$.

The normal distribution also has the corresponding cumulative distribution function (CDF), which is obtained by integrating the pdf from $-\infty$ to a target $x$:

As $\sigma$ increases the distribution becomes flatter, and as a result the CDF converges to $1$ more slowly. In the opposite direction, as $\sigma\to 0$ the peak of the PDF gets sharper until it becomes an infinitely tall spike on $x=\mu$ with total mass 1 (i.e., a Dirac delta function), which produces a step function as the CDF.

Given the historical context that we’ll uncover later in the article, from this point on I’ll refer to the distribution as the Gaussian.

Two core properties of the Gaussian

We briefly go over the two most “obvious” but crucial properties of the distribution.

• The mean of the Gaussian is $\mathbb{E}[X] = \mu$.

• The variance of the Gaussian is $\mathbb{V}[X] = \sigma^2$.

While these are well known, deriving them from scratch feels somewhat reassuring.

Where does the Gaussian come from?

As its alternative name suggests, the Gaussian distribution is attributed to Gauss, although de Moivre derived the curve as an approximation to binomial probabilities for many trials (de Moivre, 1738).

We’ll go with Gauss’s starting point of modeling measurement error.

The setup

Suppose we’re taking many measurements $x_1, x_2, \cdots, x_n$ of some unknown quantity $\mu$, and each measurement sample is independent and identically distributed. As we accrue more measurements, the values $x_i$ seem to form a bell curve, but we don’t know the underlying function that explains the curve. Let’s name the measurement density $p(x)$, and the error density function $\varphi(x)$, with $\varphi(x-\mu) = p(x)$.

We have an initial guiding principle to follow: the resulting distribution of $p(x)$ is smooth, symmetrical, and has a single peak which is its mean $\mu$. Likewise, $\varphi(e)$ shares the same properties, and its peak is specifically $0$.

However, note that this doesn’t uniquely determine the Gaussian. In fact, look how similar a Gaussian and a logistic distribution look:

They are both smooth and symmetric, and aside from the logistic function’s heavier tail, there is little discernible detail between them visually. The Gaussian distribution arises from the following postulate.

The rest of the derivation goes into a bit of detail, so I’ll wrap it up in a dropdown block.

The multivariate Gaussian

We address the $n$-dimensional Gaussian more commonly as the multivariate normal distribution. It is the extension of the 1D case into multiple dimensions. The general form of the probability density function for the $n$-dimensional Gaussian is

where $\boldsymbol{\mu} \in \mathbb{R}^{n}$ is the mean vector and $\mathbf{\Sigma}$ is a symmetric covariance matrix between the dimensions of the distribution. If we squint at it, it does resemble its univariate form.

The core property to remember is that each dimension in the distribution is itself a Gaussian distribution.

The multivariate Gaussian has two main categories, depending on the structure of its covariance matrix.

Isotropic Gaussians

An isotropic multivariate Gaussian is one where each dimension has the same variance $\sigma^2$, so $\mathbf{\Sigma} = \sigma^2\,I$:

They’re generally nice to work with.

We will later relate the $n$-dimensional unit ball to the $n$-dimensional isotropic Gaussian distribution. This intuitively makes a lot of sense because the unit ball and the isotropic Gaussian centered at the origin share a property called rotational invariance.

Anisotropic Gaussians

A multivariate Gaussian can also be anisotropic, which means the individual dimensions have different variance and can correlate with each other:

However, note that we can translate and rotate a Gaussian (i.e., choose new bases) and rescale each dimension’s variance to convert it into a standard normal distribution $\mathcal{N}(0, I_n)$. This is a common process known as whitening; all anisotropic Gaussians can be converted to isotropic Gaussians in this way.

The figure below shows that anisotropic Gaussians are still constructed out of 1D Gaussian distributions:

Scaling the Gaussian to high dimensions

Let’s look at the concentration of density of the Gaussian, as we did for the $n$-dimensional unit ball earlier.

How far away is the typical point?

We have two similar ideas of the “mean” distance of a point. Our random variable of concern is $X \sim \mathcal{N}(\mu, \sigma^2)$.

RMS distance

The first one is the simple root-mean-square (RMS) distance, defined by

For the univariate Gaussian, we know that

This notion of distance generalizes to many other distributions as well.

Mean absolute distance

The second one is the mean absolute distance $\mathbb{E}\lvert X - \mu \rvert$.

For a one-dimensional Gaussian, it is equal to

This result is more specific to the Gaussian, but it is literally what we mean by the distance of the point from the center.

Extension to the multivariate case

Recall that our intention is to look at the average distance of a sampled point from the $n$-dimensional Gaussian distribution. These two notions of distance don’t scale quite the same way (…or do they?)

Let’s slightly make our work easier by sampling points from an isotropic multivariate Gaussian centered at the origin ($\boldsymbol{\mu} = 0$) with marginal variance $\sigma^2$ in all directions, so

The corresponding radius is then $R = \lVert X - \boldsymbol{\mu} \rVert = \lVert X \rVert$.

The RMS distance. This one behaves quite nicely, given that

meaning that we have

This indicates that distance of the typical point from the Gaussian, based on $d_{\mathrm{RMS}}$, is around an $(n-1)$-sphere of radius $\sigma\sqrt{n}$. Before reaching a conclusion, let’s check if this also applies to the other distance as well.

The actual expected distance from the center. Recall that we showed that the mean absolute distance for the 1D case is

Scaling this to $n$ dimensions is not as straightforward as it might first seem.

The quantity we’re after here is the actual distance, obtained by

The derivation of this is in the appendix, but we see that this value is equal to

with $\Gamma(z)$ being Euler’s Gamma function that we saw before in the volume of the $n$-ball. As a quick check, $n=1$ gives us $\sqrt{2}\left[\Gamma(1)/\Gamma(1/2)\right] = \sqrt{2/\pi}$, which is what we found for the 1D case previously with $\sigma^2 = 1$.

As $n \to \infty$, we see that this expectation also behaves like

So the two distances kinda converge to the same “shell”. It’s best to verify this visually as well:

Left figure: Since all of the individual coordinates are independent from each other for a Gaussian, a 2D projection of an $n$-dimensional Gaussian looks, well, exactly like a 2D Gaussian.

Right figure: What we’ve uncovered is that plotting their distances to the center should give us something like a “thin shell”, which is reminiscent of a sphere.

What’s more, the variance of this radius is bounded by $\sigma^2$ (again, shown in the appendix): we have

which can be used to bound the variance

which indicates that the variance of the thin shell is actually bounded independent of the dimension $n$.

In fact, we know that the radius of the shell grows like $\sigma\sqrt{n}$, but its spread $\sqrt{\mathbb{V}[R]}$ doesn’t grow at all. Comparing the two we have

so the shell’s thickness relative to its radius shrinks like $1/\sqrt{n}$.

Comparing the ball, the sphere, and the Gaussian

In the volume of the n-ball, we saw how we can uniformly sample from the unit sphere by sampling a vector $\mathbf{v}$ from a multivariate Gaussian and normalizing it with $\mathbf{v} \leftarrow \mathbf{v} / \lVert \mathbf{v} \rVert$ exploiting rotational symmetry. We could also sample from the unit ball as well by scaling the sphere result with $U^{1/n}$.

Let’s uncover more about these three, from the other direction this time.

One interesting comparison opportunity arises from comparing the marginal distributions for each of these three objects.

From the volume post, we know that the unit ball $B_{n}$ has a marginal probability of

and from the earlier surface post we know the unit sphere $S_{n-1}$ has a marginal probability of

Finally, we know (from this post) that the Gaussian has independent coordinates with marginal probabilities of

Let’s scale it with $\sigma^2 = \frac{1}{n}$, so that its points form a thin shell around $R = \sigma\sqrt{n} = 1$, just like the boundaries of the ball and sphere.

Here’s how the marginal distributions of the unit $n$-ball and the $(n-1)$-sphere change with increasing dimensionality, compared to a scaled Gaussian with $\sigma^2 = \tfrac1n$:

The uniformly distributed cases of $n=1$ for the ball and $n=3$ for the sphere are quite interesting (the 2-sphere’s uniform distribution is Archimedes’ remarkable hat-box theorem!), however, we see that in the asymptotic case these three all blend into each other.

Of course, the marginal distributions are interesting, however the joint distribution is what uniquely determines the particular distribution. The unit cube and the $2$-sphere both have uniform marginals for $x$, $y$, and $z$, however the sphere’s joint distribution is also constrained by the requirement of $x^2 + y^2 + z^2 = R^2$ for some $R \geq 0$.

This leads us to the unique property of Gaussians, as stated by Herschel-Maxwell’s theorem:

Fun fact! This property was derived from Herschel’s 1850 work on measurements of astronomical star positions, and extended with Maxwell’s 1860 analysis of gas molecule velocities.

The important takeaway of the story is that with increasingly larger $n$ (and appropriate scaling), the line between a ball, a sphere, and a (scaled) Gaussian seems to blur into the same thing:

• We saw previously that the volume of the $n$-ball is increasingly concentrated near its boundary, which is the $(n-1)$-sphere.

• We see that the typical point of a Gaussian concentrates again on a thin shell around $\sigma\sqrt{n}$, which resembles a sphere of radius $\sigma\sqrt{n}$.

Closing remarks

There is obviously so much more about Gaussians in high dimensional space, but I think this is a good point to finish the post in terms of length. I might revisit this topic again soon though.

I think the next post will be on distances and angles. It will probably start off from the very famous property of near-orthogonality of random vectors, and will continue from our Gaussian discussion:

Appendix

Deriving $\mathbb{E}[R]$

Throughout this section we have $X \sim \mathcal{N}(0, \sigma^2 I_n)$ and $R = \lVert X \rVert$. Claude was a huge help in the main idea and the heavier math parts.

The idea is to find the distribution of $R$, and then compute the moments of $R$. $\mathbb{E}[R^m]$ is called the $m$-th (raw) moment of $R$, and we’ll use the first and second raw moments to find $\mathbb{E}[R]$ and $\mathbb{V}[R]$.

We’ll proceed in a series of steps.

Step 1: the distribution of the radius via the onion method

How is $R$ distributed? Since the Gaussian is unbounded, technically any radius is possible. We’re going to use the same onion method from the surface post.

The density of the isotropic Gaussian only depends on $\mathbf{x}$ through its length $r = \lVert \mathbf{x} \rVert$:

So the density is constant on every sphere centered at the origin, and we can slice $\mathbb{R}^n$ into onion layers, just like in the surface post. The probability of landing in the thin shell between $r$ and $(r + dr)$ is the density on that shell multiplied by the shell’s volume, $S_{n-1}(r)\,dr$:

Using $S_{n-1}(r) = \frac{2\pi^{n/2}}{\Gamma(n/2)}r^{n-1}$, the density of $R$ is

Conceptually, the two factors pull in opposite directions: $r^{n-1}$ (the sphere getting bigger) pushes the mass outwards, while $e^{-r^2/2\sigma^2}$ (the Gaussian) pulls it back in. The thin shell is where these two balance out.

Step 2: one integral for every moment

Every moment $\mathbb{E}[R^m]$ needs an integral of the form $\int_0^\infty r^k e^{-r^2/2\sigma^2}\,dr$, so let’s do it once. Recall that the Gamma function is defined by

We have

Now the $m$-th moment of $R$ is \eqref{eq:radial} multiplied by $r^m$, i.e. \eqref{eq:gammaint} with $k = n - 1 + m$:

Let’s do a sanity check on \eqref{eq:moments} with two values before using it:

• $m = 0$ gives $\mathbb{E}[1] = \Gamma(n/2)/\Gamma(n/2) = 1$, so $p_R$ integrates to $1$ as a density should.
• $m = 2$ gives $2\sigma^2\cdot\Gamma(\frac{n}{2} + 1)/\Gamma(\frac{n}{2}) = 2\sigma^2\cdot\frac{n}{2} = n\sigma^2$ using $\Gamma(z+1) = z\Gamma(z)$, which matches the RMS result $\mathbb{E}[R^2] = n\sigma^2$ from the main text.

Step 3: the expected radius

With $m = 1$ we get our result:

For $n = 1$, $\Gamma(1) = 1$ and $\Gamma(\frac{1}{2}) = \sqrt{\pi}$ give $\sqrt{2/\pi}\,\sigma$, the mean absolute distance from the 1D case.

Bounding $\mathbb{E}[R]$ and $\mathbb{V}[R]$

We now show that

In the appendix of the surface post, we defined

and showed with the squeeze theorem that for $x > 0$,

The Gamma ratio in $\mathbb{E}[R]$ is exactly $g$ in disguise: setting $x = \frac{n-1}{2}$ gives $x + 1 = \frac{n+1}{2}$ and $x + \frac{1}{2} = \frac{n}{2}$, so

Applying the bounds for $n \ge 2$:

The case $n = 1$ can be checked directly, since $0 \le \sqrt{2/\pi}\,\sigma \approx 0.798\,\sigma \le \sigma$. $\Box$

We can now follow this up with the variance bound. Using $\mathbb{E}[R^2] = n\sigma^2$ and the lower bound on $\mathbb{E}[R]$,

Let’s talk about another simple way of calculating the volume of the $n$-ball, with a method that on paper seems more intuitive than the cross-section approach.

I had planned this one to have less math than the previous post, but it turned out to require even more! It’s nice that the “mathy” blocks can be toggled on/off with the “Why?” blocks.

The onion method

Recall that in the volume of the 4-ball, we showed that the 2D disk can be thought of as a vertical stack of horizontal disks, which were actually $1$-balls of varying radii:

If we’re going to be slicing up the unit ball, there is a much more intuitive way of doing so: radially outwards, like an onion.

Inside a disk of radius $R$, each one of these circles has a circumference of $2\pi r$ with varying $r \in [0, R]$. The integral for adding all of these circle circumferences gives us the volume:

That was pretty easy, wasn’t it?

This also works in the three-dimensional case as well. We know that the surface of the ball of radius $R$ has a surface area of $4\pi R^{2}$, and the $3$-ball can be thought of as a similar 3D onion with infinite layers.

Again, we find that the integral here gives us the volume of the $3$-ball.

The n-dimensional case

So why didn’t we use this for volume of the $n$-ball, as it looks easier to calculate? It’s because we didn’t know the surface area of the $n$-ball, of course!

We actually go the other way around: the surface area $S_{n-1}(R)$ for the $n$-ball is the volume of the $(n-1)$-sphere, and is equal to the rate of change of the volume of the $n$-ball with respect to the radius:

And by our onion construction, we can express the volume of the $n$-ball using the volume of the $(n-1)$-sphere as

Does the onion method always work?

In the volume post, there was one nice property that we didn’t touch upon: Fubini’s theorem and the cross-section idea work not just for Euclidean distance, but for other norms as well.

An $\ell_{p}$ norm for a vector $\mathbf{u} \in \mathbb{R}^n$ is a measure for the length of a vector $\mathbf{u}$, defined as:

For $p=2$, this is our good old Euclidean distance, for example for $\mathbf{u} \in \mathbb{R}^3$ with $\mathbf{u} = (x, y, z)$:

The shapes below are all $3$-balls, just with different norms.

The green ball (cube) uses the max norm. The max-norm is the resulting norm we have as $p \to \infty$. It is defined as

What’s also cool is that they all encapsulate each other with increasing $p$:

We’re going to talk about norms and distances in greater detail in a later post.

So why talk about the norm now?

Our onion approach is a special case of the $\ell_2$ (Euclidean) norm. Interestingly enough, it also holds for the $\ell_{\infty}$ norm as well.

• With $\ell_{\infty}$, the “$1$-sphere” of radius $r$ gives us a square of $4$ edges of length $2r$. Integrating this gives us

which is the volume of the “$2$-ball” here, the square of edge length $2R$.

• In 3D this again holds: a max norm $2$-sphere of radius $r$ has $6$ faces of surface area $2r \times 2r = 4r^2$, and integrating gives us

which is again the volume of the $3$-ball with max norm, in other words, a cube with edge length $2R$ with volume $(2R)^3 = 8R^3$.

For other norms though, we have the issue of slantedness. It’s most intuitively observed in the figure below for the disk with various $\ell_{p}$ norms.

In our argument above we said that for sufficiently small $h > 0$ the difference between the volumes of the $n$-ball of radius $(R+h)$ and $R$ can be approximated by $S_{n-1}(R) \cdot h$.

Uniquely for $\ell_{2}$ and $\ell_{\infty}$, we see that increasing the radius $R$ by some $h$ directly pushes the outer shell away by $h$, whereas there is a slant factor for all other $\ell_{p}$ norms (for example, $\ell_{1}$ has $\frac{h}{\sqrt{2}}$ instead). That means we can’t use the $S_{n-1}(R) \cdot h$ volume approximation for the other norms.

The rollercoaster of the volume of the $(n-1)$-sphere

Given that Eq. \eqref{eq:eq1} looks awfully similar to the $n$-ball volume formula, we might expect this next property: the volume $S_{n-1}(R)$ of the $(n-1)$-sphere with radius $R$ also increases for a bit before superexponentially plummeting to zero.

Note that the unit $n$-ball volume peaks at $n=5$, while the unit $(n-1)$-sphere volume peaks at $n=7$ (i.e., with the $6$-sphere). Similar to the $n$-ball volume, increasing the radius only prolongs the inevitable.

It’s still quite surprising, but not as much as before given that it’s basically the same argument as the $n$-ball.

The equator phenomenon

The principle holds here for the surface as well, but I want to look at this topic through the perspective of the volume of the $n$-ball. I think the general idea and a similar bound come from (Ball, 1997).

Consider the unit $n$-ball centered in the origin. Pick any direction $\mathbf{u}$, and consider the normal hyperplane to that direction passing through the origin: that is the equator. If we consider a band of width $2/\sqrt{n}$ around this equator, what is the fraction of the volume of the $n$-ball within this band?

Here’s how it looks for 2D and 3D:

The band stands out relatively wide and as we increase the dimension, both the width of the band, as well as the ratio of the volume of the equator band to the unit ball start decreasing. But they don’t decrease in the same manner.

Let’s try to visualize what happens when we increase the dimension, such as $n = 400$. Even though we can’t directly visualize a $400$-ball, we can show its 2D projection while trying to accurately represent the approximate density of the ball at each point while moving along the direction of $\mathbf{u}$:

That looks quite unbelievable! It is fascinating how this thin equator band contains $\approx 68.3\%$ of the total volume of the $n$-ball, given that we can repeat it along any random direction and still its corresponding equator band will have the same percentage of the total volume. They all look like they share the “center” section, but let’s also not forget that the volume of the $n$-ball is also concentrated near the surface of the sphere, the center has basically no volume.

What is more interesting is that as we further increase $n$, the band grows thinner and thinner, while the volume ratio of the equator band to the unit ball stays $\approx 68.3\%$.

Another intuitive way to visualize this is to directly look at the marginal distribution of the points of the $n$-ball onto the 1D line of the direction vector $\mathbf{u}$:

The marginal distributions above show how the distribution of points along a random direction changes as $n$ increases. For high dimensions, $68.3\%$ of points lie within the band of $2/\sqrt{n}$ width. This is slightly intuitive when we consider that most individual coordinates for randomly sampled points on the sphere should have magnitude close to $1/\sqrt{n}$, which is one of the core ideas for a following post on angles in high-dimensional space.

A practical note on “Why the surface?”

One thing that frequently pops up in machine learning is to normalize high-dimensional samples from a dataset for reasons like measuring cosine similarity or retrieving samples. Given a sample $\mathbf{x} \in \mathbf{X} \subset \mathbb{R}^d$, L2-normalizing the sample $\mathbf{x} \leftarrow \frac{\mathbf{x}}{ \lVert \mathbf{x} \rVert}$ puts it on the surface of the $d$-dimensional unit ball, i.e., the $(d-1)$-dimensional unit sphere.

Representation spaces are commonly $512$, $768$ or $1024$-dimensional for stuff like word embeddings (see a previous post for an example) and vision-language embeddings (e.g., CLIP, SigLIP), where both the volume and surface area are practically in the funky high-dimensional regime, looking at the volume scaling graph.

Hopefully, now we have a bit more insight towards the fact that this $(d-1)$-sphere does not have our 2D or 3D expectations in appearance, distribution, or concentration of volume.

Appendix

The limit of the Gamma ratio

Recall that we’re concerned with the limit

Let $g(x) = \frac{\Gamma(x+1)}{\Gamma(x + \frac{1}{2})}$. We’re going to need $g(n/2)$. We have the very useful property of

using $\Gamma(z+1)=z\,\Gamma(z)$.

Similarly, we see that

The function $g(x)$ is monotonically increasing, since $\log\Gamma(x)$ is convex (a commonly known property). Thus we have

Multiply everything by $g(x) > 0$:

Using the identities we just proved:

then take the square root:

Divide by $\sqrt{x}$ to get

and finally substitute $x = n/2$ to recover $A_{n}$:

All that’s left is to look at the limit as $n\to\infty$. We have

Therefore, by the squeeze theorem we have

In the previous post, we were able to calculate the (hyper-)volume of the $4$-dimensional ball of radius $R$ as

How do we extend this to any $n$-dimensional ball? How does the volume change with $n$? The $n$-ball contains many super interesting mysteries.

What follows next is some math, then a couple of counterintuitive properties. For example:

• We show that the volume first increases, and then plummets to zero with increasing $n$.
• Nearly all of the $n$-ball’s volume concentrates in an arbitrarily thin shell near the boundary as $n$ increases.

Let’s dive in!

Intermediate properties of the $n$-ball

Before we tackle the question head-on, we need some intermediate results. Many of these results come from building up the $n$-ball from the $(n-1)$-ball.

A recurrence between $V_{n}(R)$ and $V_{n-1}(R)$

Recall that we showed

where we defined $r_{w} = \sqrt{R^2 - w^2}$ as the radius of the cross-section along the $w$ dimension, and it had a corresponding hypervolume of $V_{3}(r_{w}) = \frac{4}{3}\pi r_{w}^3$.

This extends to any dimension $n$. The cross-sections of the $n$-balls that we integrate to find the hypervolume are themselves $(n-1)$-balls, so in general we obtain the following expression for the $n$-dimensional hypervolume:

This is a very promising recurrence relation, as it allows us to find the general formula to the volume of the $n$-ball.

Mining the recurrence for simplicity

The first instinct we have with a recurrence relation like \eqref{eq:eq1} is to see if unrolling reveals any useful properties. Our efforts are rewarded with the following result:

This lets us “skip” a dimension for a practically much easier integral, which will be useful later.

A general form of the $n$-ball volume

As we saw with previous volume formulas $2R, \pi\,R^2, \frac{4\pi}{3}R^3$, it looks like all of the $n$-ball hypervolumes follow a specific sequence. It turns out that we can also formalize this as

where $C_{n} = V_{n}(1)$ is the volume of the unit ball!

Deriving the general recurrence relation

Combining \eqref{eq:eq2} with \eqref{eq:eq3} gives us a new recurrence relation:

The closed-form of the $n$-ball volume formula

Eq. \eqref{eq:eq4} gives us the insight we need to instantly calculate the volume of the $n$-ball.

Intuitively, it says that whenever we increase the dimension by two (from $n-2$ to $n$), we scale the previous volume by $\frac{2\pi R^2}{n}$. We can obtain the formula for the $n$ case by collecting these multipliers with each added dimension.

The resulting $n$-ball volume formula is then

where $\Gamma(z)$ is the Gamma function.

For even values $n = 2k$ we have the simpler

which is very handy in examining scaling for the $n$-ball.

Here are some example values for the volume of the $n$-ball, up to $n=9$.

But a table is hard to look at.

Analyzing the $n$-ball

We did all of the hard work, let’s see some visuals!

For a moment, let’s think of the $n$-cube: for a side length of $\ell$ its volume is $\ell^{n}$.

• if the side length is $\ell = 1$, then the volume is $1\times1\times\cdots\times1=1$.
• If the side length is $\ell < 1$, then with each dimension we’re multiplying with a value smaller than $1$, which means as $n \rightarrow \infty$ we have $\mathrm{volume} \rightarrow 0$. The decrease is exponential in $n$ due to $\ell^n$.
• If the side length is $\ell > 1$, then with each dimension we have $\mathrm{volume} \rightarrow \infty$ as $n \rightarrow \infty$. The increase is exponential in $n$.

So, with this in mind: how does the $n$-ball change? Recall the formula is

With $R = 1$, the volume increases as we increase $n$:

… until we reach $n = 5$. After that, the volume decreases. And not only does it decrease, it decreases superexponentially, faster than any exponential function!

This looks like a very thrilling rollercoaster ride. Not only is the volume asymptotically going to $0$, but beyond a very reasonable $n$ value it is practically zero as well!

For $n=16$ we have

and for $n = 32$ we have

For most applications, that is practically zero.

What about a larger ball?

We can try to beat this process by looking at a slightly larger ball, since we would also scale the volume by $R^{n}$. Instead of $R=1$, let’s take $R = 2$ or $R=3$: that should give us a $2^{n}$ and $3^{n}$ scale factor, respectively.

It does not help. There is no escaping the rollercoaster that is the volume of the $n$-ball.

Where is the volume of the $n$-ball?

Consider a very thin shell of the unit $n$-ball, obtained by drawing the $n$-ball with unit radius $R = 1$ and then putting a slightly smaller $n$-ball of radius $0.98R$. What is the ratio of the volume of the inner ball to the outer ball?

In the figure above, you can barely see the gap between the inner and the outer $n$-balls.

• For $n = 1$, we have

• For $n = 2$, we have

• For $n = 3$, we have

… you get the picture. We start off pretty strong, and visually we can see that most of the volume of the $n$-ball is inside the volume of the smaller $n$-ball.

However, by $n=128$ the ratio is now

For a $128$-ball, approximately $92.47\%$ of the outer ball’s volume is concentrated in the thin shell!

We can express the specific ratio for some $n$ and $R$ with

As $n \rightarrow \infty$, we see that the ratio becomes $1$. There is basically no volume unless you are near the boundary.

Sampling from the unit $n$-ball

We end this one with sampling points in the unit $n$-ball. In my case, it was quite peculiar when I had to deal with it recently.

How do you sample from the $n$-ball of radius $R$? Suppose you have access to the uniform distribution and the normal distribution.

One idea is rejection sampling. It’s very fast and intuitive, let’s look at the 2D case.

1. We sample $x$ and $y$ such that $x,y \sim \mathcal{U}[-1, 1]$ are from the uniform distribution.
2. We calculate $r = \sqrt{x^2 + y^2}$.
3. If $r \leq R$ we keep the sample (it’s in the disk), otherwise we reject it.

Here’s how it looks for 1000 candidate $(x, y)$ pairs in step (1).

Empirically, $773 / 1000 = 0.773$ were in the disk. On paper, we expect around

of sampled points to be valid samples, or

candidates to successfully sample 1000 points inside the unit disk. As you might expect this doesn’t scale to higher dimensions very well.

For $n=20$ we have

or approximately

or $40000000000$ candidate points just to perhaps sample 1000 points inside the unit $20$-ball. (Hicks & Wheeling, 1959) states that generating acceptable random points from the $20$-cube would have yielded roughly “40 points per century”, albeit on an ancient 704 program.

Rejection sampling is basically impossible for higher dimensions, due to its inefficiency.

The Gaussian trick

I’m not sure who the entire process is attributed to, although the normalized Gaussian idea appears to be from (Muller, 1959). This is actually from the same journal issue as the Hicks and Wheeling quote above, and Muller also credits George W Brown on suggesting the method.

The trick works in three steps.

1. We start by sampling $\mathbf{g} \sim \mathcal{N}(0, I_{n})$, the $n$-dimensional Gaussian distribution. We then normalize it as $\mathbf{v} = \frac{\mathbf{g}}{\lVert \mathbf{g} \rVert}$.

Why the Gaussian?

The short answer is that the Gaussian is rotationally symmetric, so normalizing the points gives us a uniform distribution over the unit $n$-sphere (the boundary of the $n$-ball).

Now, we have to disperse the points on the $n$-sphere with the appropriate radius distribution of the $n$-ball.

2. We sample $u \sim \mathcal{U}[0, 1]$ and let $r = u^{1/n}$.

The product $r \cdot \mathbf{v}$ already gives us a random uniform sample from the unit $n$-ball.

3. Scale it by multiplying with $R$ to randomly sample from the $n$-ball with radius $R$ instead.

After all three steps, the product $R\cdot r \cdot \mathbf{v}$ gives us a uniformly sampled random variable in the $n$-ball with radius $R$, quite efficiently and without any rejection sampling.

Closing remarks

There are many more properties about the $n$-ball that I want to talk about, and this is only scratching the surface. We saw a couple major properties in this post.

• The closed-form volume formula can be derived from the cross-section integral, as we showed in the first section.

• The volume of the $n$-ball is a rollercoaster that briefly rises before plummeting to its doom (zero).

• As $n$ increases, the volume increasingly concentrates near the boundary of the $n$-ball.

• It’s slightly difficult to sample from at first glance, but efficient with some tricks.

Next time, I’ll talk about the surface of the $n$-ball and an alternative formulation for the volume.

“To deal with hyper-planes in a 14-dimensional space, visualize a 3-D space and say ‘fourteen’ to yourself very loudly. Everyone does it.”

Geoffrey Hinton

Humans are really good at visualizing one, two, and three dimensions. Unfortunately, this doesn’t scale past three very well¹. What’s more interesting is that some of the intuition that we have for 2D and 3D really doesn’t extend to higher dimensions as we might initially expect.

Here is the series in chronological order (to be updated with new entries over time):

1. The volume of the $4$-ball

2. The volume of the $n$-ball

3. The surface of the $n$-ball

4. The $n$-dimensional Gaussian distribution

In the familiar low-dimensional spaces around us, we have a way to quantify how much room something takes up.

In one dimension it is the length, in two dimensions it is the area, and in 3D we have the volume.

What do we call this notion with higher dimensions? It’s called the hypervolume¹.

The two most basic geometric objects we have are cubes and balls. The unit cube is not as interesting at first, because it always occupies $1 \times 1 \times … \times 1 = 1$ unit of space regardless of its dimension.

In contrast, the ball is a lot more fun.

Some definitions

We denote by $V_{n}(R)$ the hypervolume of the $n$-ball, which is the generalization of a ball in $n$-dimensional space.

The closed $n$-ball with center $\mathbf{x} \in \mathbb{R}^{n}$ and radius $R$ is defined as

So it’s just all of the points around a point $\mathbf{x}$ that are at most $R$ units away from $\mathbf{x}$ based on some distance metric $d$.

In this post we are concerned with Euclidean distance $d(\mathbf{x}, \mathbf{y}) = \lVert\mathbf{x} - \mathbf{y} \rVert_{2}$, and the ball centered at the origin (so $\mathbf{x} = 0$). Thus, our ball definition simplifies to just a set containing points $\mathbf{p} = (p_{1}, …, p_{n})$, such that

That’s all nice, but how do we compute the hypervolume of an $n$-ball?

Naturally, we start from $n=1$ and build up.

$1$-ball

The $1$-ball of radius $R$ is an interval $[-R, R]$ around the origin. It has a length of $R - (-R) = 2R$, which is also its 1-dimensional hypervolume $V_{1}(R) = 2R$.

It is not very round, is it? However, it is in fact the roundest thing one can ever find in 1D.

$2$-ball

The closed $2$-ball is our good old friend, the disk. From school we know by heart that it has area equal to $V_{2}(R) = \pi R^{2}$.

$3$-ball

The $3$-ball is the most intuitive one, in that it is what you would call a ball in everyday life. It has volume equal to

We again learn it by heart during school.

However, in multivariate calculus we are able to show it with the aid of the following idea: a ball is a vertically stacked arrangement of two-dimensional disks:

With our knowledge of integrals, we know that we can represent this infinite sum of disks as an integral as we move in the $z$-axis. However, we also notice that each disk has its own radius that depends on the particular value of $z$. Geometrically, we can show it as this:

For each disk with height $h$, the radius is

and its area is $\pi r_{h}^2$. Thus, the total volume is

Which brings us to the case of the 4-ball. What’s one more dimension?

4-ball

Unfortunately, we can’t visualize the $4$-dimensional ball.

We don’t know what a 4-dimensional ball looks like; all we know is that it is composed of points $\mathbf{p} = (x, y, z, w)$ such that

How can we calculate the (hyper-)volume of this thing that we can’t visualize?

To progress, we actually have to go backwards.

Retracing our steps

Upon closer inspection, we can see that our stack-of-disks explanation for the $3$-ball is just as applicable for the two-dimensional disk as well. The disk is a stack of horizontal intervals, which are actually $1$-balls as we discussed earlier!

Again, each $1$-ball has some radius $r$ that depends on the current $y$, with ${r_{y} = \sqrt{R^2 - y^2}}$ as $y$ moves along $[-R, R]$. They have a length of $2 r_{y}$, which we’ll sum up to obtain the area.

The integral here is actually slightly trickier than the 3-dimensional case, but it turns out that

as we would expect.

This even works for the $1$-ball (interval) as well, if we consider it to be a stack of points (zero-dimensional balls), each of volume $V_{0} = 1$. The integral for the 1D case is just

To summarize, we see that constructing the $n$-dimensional ball is possible by vertically stacking $(n-1)$-dimensional balls along the new dimension and summing up their respective hypervolume.

Back to the $4$-ball

Before we continue, I want to sketch one aspect of the $4$-ball.

• When we slice the 1D interval on $x$, we get a point.
• When we slice the disk along the new dimension $y$, we get an interval (a $1$-ball).
• When we slice the $3$-ball along the new dimension $z$, we get a disk (a $2$-ball).

That means that when we slice the $4$-ball, we’re going to get a $3$-ball when we look at the cross-section.

This is my best attempt at actually visualizing this idea, slicing across the new dimension $w$:

Let’s not be fooled, these balls are actually not intersecting. All of the cross-sections live in their own three-dimensional hyperplane with a constant $w$. None of them share any points, and they have zero thickness² along $w$. It’s also incredibly interesting how the cross-sections stay quite large for most of the way, and only collapse near $w=R$.

This idea of summing up cross-sections is called Cavalieri’s principle, and the rigorous version is known as Fubini’s theorem.

The $4$-ball, revisited

We’re back here again, but equipped with new machinery. Our idea is simple: Given a fixed $w$, the points $\mathbf{p} = (x, y, z, w)$ are in the cross-section

of the $4$-ball centered at the origin with radius $R$. We know that (2) is the definition of a $3$-ball with radius $r_{w} = \sqrt{R^2 - w^2}$ and volume $\frac{4}{3}\pi r_{w}^3$. We can sum their volumes together over $w$ to obtain our prize, the hypervolume of the $4$-ball.

We express this as the integral

which turns out to be… with drum roll please…

What’s really nice is that this method extends to the $5$-ball, a $100$-ball, or any arbitrary $n$-ball hypervolume by summing up cross-sections of $(n-1)$-balls.

Its recursive nature also allows us to obtain closed-form solutions for any high-dimensional ball. We’ll explore general properties of the $n$-ball in the next post.

Appendix

Deriving $\int_{0}^{\pi / 2}\cos^2\theta \, d\theta$

This one is kinda pretty. Substituting $\theta \to \frac{\pi}{2} - \theta$, we have the useful

Over the same interval we sum them together:

making use of $\sin^2\theta + \cos^2 \theta = 1$.

Therefore,

Deriving $\int_{0}^{\pi / 2}\cos^{4}\theta\,d\theta$

Note that we have $\cos^{4}\theta = \cos^{2}\theta(1-\sin^{2}\theta)$, meaning that we can write everything as

The first term, the one with $\cos^{2}\theta$, is exactly what we want. We showed that it is equal to $\frac{\pi}{4}$ in the previous section.

For the second term, we can use the double angle identity $\sin\theta\cos\theta = \frac{1}{2}\sin 2\theta$:

Applying the change of variables $u = 2\theta$, so that $d\theta = \frac{1}{2}\,du$ and the limits become $[0, \pi]$:

We can reuse our previous result. Since $\sin^2(\pi - u) = \sin^2 u$, the halves $[0, \frac{\pi}{2}]$ and $[\frac{\pi}{2}, \pi]$ contribute equally, so

where we used $\int_0^{\pi/2}\sin^2 \theta\, d\theta = \int_0^{\pi/2}\cos^2 \theta \, d\theta = \frac{\pi}{4}$ from (3). Therefore

Putting it all together:

It’s been a week or two since I launched wordsim, and I’ve had a lot of questions from people on how and why it works behind the scenes. In this post, I want to expand on these topics and my own experience in building this simple but surprisingly addictive game.

In the first part, we’ll go over the fundamental concepts of representing words and determining word similarities. I want to go into details of representations with much more detail in a future post, but here I’ll try to keep the discussion here to a minimum.

Then, I’ll give a technical breakdown of how everything is implemented and how the pipeline works. We’ll have a short section on adapting the game to the Turkish language, and about some unexpected challenges in the process.

Let’s start!

About wordsim, and how it works

There comes a time for many people where they make a simple game about guessing words or numbers. wordsim is such a game focusing on guessing a hidden word using semantic similarities.

It started as a spin-off of my curiosity in exploring how deep language models internally represent various concepts and the relations between these concepts. I had some prior experience in working with embeddings of vision-language models like CLIP (Radford et al., 2021), or recently leveraging them for slightly more complex tasks such as jazz standard recognition. I then wanted to explore the geometry of word embeddings.

The core mechanism of word similarity games

This particular genre dates itself back to the childhood classic “hot or cold” guessing game, where one player hides an object within a room and guides another player to find that object using clues such as “cold” and “warm”. This idea naturally extends to a word guessing game: one player thinks of a word, and upon receiving a guess, gives clues about the semantic “closeness” of their guess until the word is eventually found. That is precisely how wordsim is played.

Of course, implementing this idea is not nearly close to being a novel thought: looking for such a game leads one to games like Contexto, the daily semantic-guessing game Semantle, and many more implementations. I am not aware of any Turkish-focused word similarity game of this nature other than wordsim at the moment, so perhaps there is a small but unique contribution of the project there.

To successfully implement wordsim, conceptually we need two things:

1. A way to represent words.
2. A way to compute similar words to a target word.

Let’s start with item #2 first.

Thinking about word similarities

How does one determine which words are similar? Let’s start from some simple ideas:

• “forest” reminds us of “tree”, because forests are made of trees. Relations such as “has a”, “is a”, and words with the same or very similar meanings exhibit naturally strong similarity.

• There are multiple angles to approach a word. You can reach “earth” by starting from the solar system and the planets, as well as starting from seeds, farming and soil. Although they are seemingly unrelated paths, these two concepts should both lead to “earth” via increasing similarity.

• Similarity should be somewhat transitive: if we only report high similarity for immediate neighbors of a word, it’s very hard to reach that word from a random starting point. If the word is “egg”, then we expect “chicken” to be a very similar word, and “feather” to be similar to “chicken”. Thus, “egg” and “feather” should be somewhat similar as “chicken” is highly similar to both of them.

…and so on. We can’t establish all of these requirements by hand, as they would be too specific and brittle. We’d like to have a system that implicitly infers these kinds of relations.

Luckily for us, this is the natural area of language modeling in machine learning. With enough data and an appropriate model, we might be able to satisfy all of these requirements directly by learning them from the data.

How do we represent words?

At the most basic level we need to find a way to represent a word so that we can give it to a model. In natural language processing (NLP), the traditional way to do this was to build “one-hot encodings” for each word to give to a model:

1. Collect all words that you know from the text. Let’s say we collected 8 words, this is now our vocabulary.
2. Give them a unique ordering, so that each word has its unique index.
3. Represent a word by a series of 0s and 1s, such that only its unique index is 1 while all other entries are 0. This is called a “one-hot encoding”.

We can afford skipping grammatical details like word normalization, filtering, and out-of-dictionary words to simplify the general idea. Let’s go over a small example:

docs = [
    ['this', 'is', 'a', 'dog'], # an example sentence from the dataset
    ['cat', 'and', 'dog'],
    ['a', 'mouse'],
    ['cat', 'and', 'mouse', 'game']
]
words = set()
for doc in docs:
    for word in doc:
        words.add(word)
vocab = sorted(list(words))
print(vocab) # ['a', 'and', 'cat', 'dog', 'game', 'is', 'mouse', 'this']

The code above builds a vocabulary of known words from all of our text documents. We can assign a unique representations made of 0s and 1s for each word, which would look like:

print(onehot(vocab, 'cat'))   # [0, 0, 1, 0, 0, 0, 0, 0]
print(onehot(vocab, 'mouse')) # [0, 0, 0, 0, 0, 0, 1, 0])

With this sequence-of-numbers representation, we can achieve many nice things. We could represent a sentence by summing these one-hot vectors together (called a bag of words), or feed them one by one as features into a neural network model to be used in tasks like sentiment analysis and next word prediction.

However, there is one problem with this way to representing words. Do you see it?

Why one-hot encoding isn’t ideal

When representing words as one-hot vectors, all words inside a dictionary $V$ of size $\lvert V \rvert$ occupy a discrete $\lvert V \rvert$-dimensional space filled where each dimension is either $0$ or $1$. Each word representation $w$ has the same norm $\lVert w \rVert = 1$ and the same Euclidean pairwise distance of

independent of the specific pair of words selected. Geometrically, they are all equally far apart points on the surface of a high-dimensional sphere. The embedding space of words here is discrete and devoid of any meaningful distance between words.

This isn’t ideal for our task of simple and cheap similarity computation, and with this approach we don’t have a semantically meaningful word embedding space. Can we do better?

Continuous word embeddings and word2vec

Just when we need it the most, “A Neural Probabilistic Model” (Bengio et al., 2003) comes to our rescue. It focuses on the concept of training neural networks that learn distributed word embeddings, where the representations of words are learned from the data as dense, continuous vectors in an embedding space. Although this idea isn’t immediately established as the defacto method of representing words until later, it forms the foundation of modern language modeling.

What advantages does it provide?

With continuous word embeddings, we now have the ability to represent words as unique directional vectors within a smaller but meaningful vector space.

• We don’t need a new dimension to uniquely identify each word, reducing the problem of the curse of dimensionality.
• A model using continous word embeddings is able to learn a representation for a word that puts it semantically closer to similar words.
• Learning embeddings is made possible by backpropagation from a loss function all they way back to the embeddings.
• It lets us explicitly and cheaply compute a notion of similarity between words $w_{1}$ and $w_{2}$ using simple cosine similarity:

While this work is 20+ years old at this point, it wasn’t until the simple and efficient methods introduced by the 2013 word2vec paper, “Efficient Estimation of Word Representations in Vector Space” (Mikolov et al., 2013), that continuous word embeddings really took off. In particular, the geometry of word2vec vectors shows us an extremely interesting property of word embeddings: composition of concepts using word vector addition is possible in representation space. The king - man + woman = queen is a classic example of “word arithmetic”:

This is a very promising development in our search for cheap and efficient semantic similarity estimation.

Modern word embeddings

From 2017, there have been massive changes across NLP.

• The Transformer architecture (Vaswani et al., 2017) was initially about sequence-to-sequence translation, but it has entirely transformed NLP (and vision, and everywhere else in deep learning) due to its efficiency and incredible scalability to enormous amounts of data.
• Massively scaled language models like the GPT family (Radford et al., 2018) of decoder-only Transformer architectures and the encoder-only BERT family of models (Devlin et al., 2019) have been hugely successful in multiple NLP tasks.

Modern large langugage models (LLMs) build contextual representations of words and sentences from token embeddings, and so their word representations are potentially better than the static word2vec mappings. However, LLMs are in general used for sentence-level tasks, while here we are looking at individual words.

For wordsim, I use a transformer-based embedding model particularly trained for producing general-purpose embeddings, as on paper it provides the best of both worlds.

With everything set in place, we are now ready to talk about how everything works under the hood.

How wordsim is implemented

wordsim is composed of an offline generation module in Python in the backend and a responsive frontend that serves the puzzles and the UI with TypeScript. The code for the project is on GitHub.

The generation pipeline

The generator’s task is to prepare an initial list of possible vocabulary words, process and filter them according to the grammar rules, compute representations of words, and precompute similarities for each puzzle to pass to the frontend. The vocabulary-building process is something like this:

graph TB
    A[1. Determine list of words for the game dictionary] --> B[2. Preprocess words into canonical forms] --> C[3. Extract representations of words] --> D[4. Build static word embedding table]

For English, these steps are all fairly straightforward.

• The wordfreq repository is a pretty good resource that we can use to determine a list of frequent words in the English dictionary based on multiple corpora. This ensures a nice and rich dictionary of words. For English, a vocabulary of approx. 30000 most frequent words is satisfactory for an enjoyable game experience.
• We don’t even have to preprocess words unless the candidate word list is contaminated with plural endings or past tenses, which is generally not the case for English puzzles.
• LLMs are very good at learning semantic information about tokens that make up words in the English language. This is due to the abundance of English training data and downstream tasks of LLMs requiring a good knowledge of the language.
• As a consequence of the above factor, it turns out language models that focus on producing embeddings (e.g., Google’s EmbeddingGemma, Qwen’s Qwen3-Embedding series and many others) produce good enough representations of words to encode semantically-related words together. Empirically, we confirm this on based on user feedback on generated English puzzles.

Since a word’s representation doesn’t change, we can think of the first part as building a static word -> embedding mapping.

We can also do a high-level visualization of embeddings of all the words in the vocabulary with a 2D UMAP projection:

UMAP lets us broadly visualize the organization of the embedding space. Without interpreting the distances too literally, we can see same-category puzzles are generally in the vicinity of each other. This is most true for categories such as food and animal, but more abstract categories such as adjective and action seem to be more dispersed rather than clustered together. We can also see that the general object category is perhaps too vague, perhaps we could narrow it down a little in the future!

Puzzle generation

The puzzle generation process is then simply:

graph TB
    A[1. Select a target word from the vocabulary] --> B[2. Retrieve representation of word] --> C[3. Compute similarity against other words in the dictionary] --> D[4. Compute rankings and store them]

where we have already computed and stored each word representation in the dictionary beforehand. The current version uses EmbeddingGemma-300m (Vera et al., 2025) for word representations in English, as it is designed as a multilingual embedding model that is both lightweight and powerful (and also supports Turkish), that has also been trained with semantic similarity as a training task.

Speaking of Turkish support, we’ll come back to this topic in a little bit.

Frontend logic

The frontend is responsible for serving the generated puzzles, as well as general responsiveness of the game through subtasks such as category-based filtering, puzzle state management, and session management between visits to the site.

It has a whole bunch of important features for improving user experience and quality of life including

• Hint support, validation of user word inputs
• Easier navigation through puzzles using the grid system
• Internationalization of the platform (and the puzzles) in the currently languages
• Tutorials and explanations on game logic

…and so on.

The frontend arguably took more time and effort to polish it into its current state compared to the generation pipeline and definitely deserves more attention, but the heart of the game is about word semantic similarity.

What about Turkish, though?

Turkish is very different from English for a multitude of reasons. There are no gendered pronouns, subject-object-verb ordering is different, the rules are have a bunch of gotchas about Turkish support for application support for developers, however we’ll focus on one main detail here: it is highly agglutinative. What this means is that words are encoded all sorts of meanings primarily through various suffixes that can be chained together to further enrich their meaning. As an example, some words that look relatively normal to my eyes include

farklılaştırmak

fark-lı-laş-tır-mak

meaning “to turn something into something different”

kalabilseydiler

kal-a-bil-se-(y)-di-ler

meaning “if they were able to stay”

…and so on. Of course, one can talk about the limits of suffix chaining with the examples from this page on longest Turkish words such as

Muvaffakiyetsizleştiricileştiriveremeyebileceklerimizdenmişsinizcesine

but that’s just to demonstrate the capabilities of the language.

The point is that words have many forms depending on their suffixes, and these words are very frequent, which messes up our usual flow in a couple of ways.

Just use EmbeddingGemma?

Let’s recall that we specifically chose EmbeddingGemma because it is multilingual (with Turkish support as well), so can’t we use it here? It’s not that easy unfortunately, due to some problems that are progressively harder to fix.

First of all, there’s the issue of overcrowding the vocabulary due to grammatically similar words. For both nouns and verbs, there is a tendency to include words that are a result of simple suffix chaining.

Take two common words such as “ev” (house) and “koşmak” (to run). Here are some common forms:

Although these are all semantically related to the target word, we don’t want to pollute the vocabulary or the rankings with slight modifications of the target word.

Most of these should be trimmed or normalized (simple tenses, some plural forms), but some of these derived words are standalone concepts and are essential to the language. We have to get our hands dirty here.

Preprocessing. We have to implement some form of normalization in the pipeline so that our vocabulary isn’t crowded by common forms. wordsim uses zeyrek here, which is a Python port of the zemberek-nlp Turkish NLP project. At a high level, we process each candidate word in the list of common Turkish words by analyzing each word. We remove those that contain suffixes about plurality/tense/possessiveness, remove proper nouns, convert verbs into infinitives, and some other steps.

Our next problem is quite interesting.

Even without knowing a bit of Turkish, it’s apparent what the system is doing: it’s listing words that look/sound similar to parts of the original word. What gives?

Tokenization. Tokenization is the act of transforming an input word, sentence or document into a series of chunks called “tokens” that the model learns to understand. Tokens come in many shapes and sizes, they can be characters, subwords, entire words (or patches of images — see vision transformers, honestly anything can be a token), and through the process of training the model learns to extract meaning from tokens and subsequent sequences of tokens. See OpenAI’s tokenizer for an interactive example.

One likely guess is that the model does not learn the tokens of sin and cap correctly in Turkish. In particular, EmbeddingGemma’s multilingual support means tokens for sin and cap have multilingual context (they are also English words themselves), composing them together might not suggest to the model that they together form the Turkish word meaning “squirrel”. It could also just be that the model doesn’t have good representation geometry of Turkish words in general, due to its rarity compared to English.

This general issue is apparently something that other researchers have noticed such as in (Bayram et al., 2026). Trying out their Turkish-aware embedding model, EmbeddingMagibu-200m, definitely did seem to improve embeddings in this direction, but ultimately to a limited extent for this project in my experience.

One final problem in the Turkish version that I experienced during development that I’d like to show is the following:

The “-siz/-sız” suffix encodes the meaning of “without” in Turkish; the words listed are not really related except the “without” meaning added to each word.

I’m not particularly confident about this one. It might be the case of tokenization again, but it feels as though this is more about retrieving similar words based on the wrong kind of semantic similarity.

Solution: The simplest solution for Turkish is to go back to our trusted baseline (word2vec embeddings), and search for a better solution in the process. Recently researchers have shown that on the specific task of semantic similarity, a word2vec model with the skip-gram method can perform better than BERT-based models (Sarıtaş et al., 2024) for the Turkish language. Following this result, the Turkish version currently uses word2vec embeddings with skip-gram training.

What this means for Turkish puzzles is that with the current approach we are much better in removing suffix-induced semantic neighbors, improving neighbor diversity at the cost of a slightly suboptimal list of close words in the neighboring word list, which makes the puzzles a little bit harder.

Similar to the English version, we can also look at UMAP projections of Turkish word embeddings as well. You might recall this figure from the teaser:

We see that overall, the category trends from the English version are also present in the Turkish version. It is quite interesting to see that the verbs are so far removed from other categories. This also makes sense as to why the closest words to a target puzzle verb are generally verbs. One more thing is that these are currently from word2vec embeddings, while English is from a transformer-based Embedding model; in fact I expected less similarity between them.

I’m still working towards improving the Turkish model for a more enjoyable experience for all.

What is next for wordsim?

I want to analyze some parameters that control the entire pipeline. For example, we haven’t talked about the embedding dimension, which is the dimensionality of the space in which word embeddings reside in. For English we have $d_{\mathrm{embed}}=768$, meaning 768-dimensional representations of words. However, what is the effect of increasing or decreasing this dimensionality?

• Does decreasing dimension hurt the representational capability of the embedding model for semantic tasks?
• When increasing the dimensionality, how is the system affected by the curious behavior of Euclidean distance in high dimensional space?

There are many more questions about this parameter and other ones in the process (e.g., raw embedding vs prompt-guided embeddings)

There’s also the obvious next task of improving the Turkish version of the game with better semantic understanding. I also want to add new languages so that it is more accessible for more people.

Want to join the effort? The code is available on GitHub. Feel free to contribute, propose ideas, or report bugs and issues that you might find while playing.