Human heights make a bell curve. City populations do not — they make a shape with a colossal tail, where one city holds more people than the next fifty combined. The number of emails arriving in the next hour follows a third shape entirely. These are not arbitrary formulas that somebody chose ; each shape is the fingerprint of a mechanism. Heights are a sum of many small effects, so they are bell-shaped. City sizes grow by multiplication and attract more growth the bigger they get, so they have a heavy tail. Emails arrive independently at some rate, so their counts follow a third law. This post is built around that single idea : if you can describe how the randomness is produced, you can predict which distribution you will see — before collecting a single data point. We will derive the main ones from their mechanisms, see where each shows up in the world, and finish with the cases where the familiar tools quietly stop working.

PART I — THE GRAMMAR
what a distribution is, and the three numbers that describe one

1. A distribution is a budget

Start with the simplest possible picture. You have one unit of probability — total certainty, the number $1$ — and you must spread it over everything that could happen. A distribution is nothing more than the rule saying how you spread it.

If the outcomes are countable — a dice roll, a number of customers, a count of defects — you can hand out the budget in lumps.

For a discrete random variable, the probability mass function gives the lump sitting on each outcome : $$ p(x) = \mathbb{P}(X = x), \qquad p(x) \ge 0, \qquad \sum_{x} p(x) = 1. $$

If the outcomes form a continuum — a height, a waiting time, a temperature — something strange happens. There are infinitely many possible values, so any single one must get zero budget. The probability that a person is exactly $1.800000\ldots$ metres tall, to infinite precision, is exactly zero. What survives is not a probability per outcome but a density : probability per unit of length.

For a continuous random variable, the probability density function $f$ satisfies $$ \mathbb{P}(a \le X \le b) = \int_a^b f(x)\,\mathrm{d}x, \qquad f(x) \ge 0, \qquad \int_{-\infty}^{\infty} f(x)\,\mathrm{d}x = 1. $$ Probability is area under the curve, never the height of the curve.

In plain words : a mass function is a set of coins placed on outcomes ; a density is butter spread over bread. Asking “how much butter is at this exact point ?” is meaningless — you have to ask “how much butter between here and there ?” This is why a density can exceed $1$ without anything being wrong : a density of $5$ per metre over a stretch of $0.1$ metres is a probability of $0.5$.

There is one object that works for both cases and never breaks, and it deserves more attention than it usually gets.

The cumulative distribution function is $$ F(x) = \mathbb{P}(X \le x). $$ It always exists, always increases from $0$ to $1$, and completely determines the distribution.

Figure 1 - The same idea in three costumes. On the left the budget sits in lumps, and a bar's height is a probability outright. In the middle it is spread as butter, and only an area is a probability — the height alone means nothing. On the right the cumulative function turns that area into a vertical gap, which is why it works for both kinds and why every quantile, percentile and simulation goes through it.

The CDF looks like bookkeeping, but it is the workhorse. It answers “what fraction is below this ?” directly, it is what a quantile inverts, it is what a survival curve complements, and in section 9 it turns out to be the machine that generates every other distribution on a computer.

2. The three numbers that describe a shape

Before cataloguing shapes, we need a vocabulary for comparing them. Three numbers do most of the work.

Where does it balance ? The expectation is the centre of mass of the distribution — put the density on a plank and find the point where it does not tip :

\[\begin{equation} \mathbb{E}[X] = \sum_{x} x\,p(x) \qquad\text{or}\qquad \mathbb{E}[X] = \int_{-\infty}^{\infty} x\,f(x)\,\mathrm{d}x. \label{eq:exp} \end{equation}\]

How spread out is it ? The variance is the average squared distance from that balance point, and its square root, the standard deviation, brings us back to the original units :

\[\begin{equation} \operatorname{Var}(X) = \mathbb{E}\big[(X - \mu)^{2}\big] = \mathbb{E}[X^{2}] - \mu^{2}. \label{eq:var} \end{equation}\]

The second form is the one you actually compute with, and it drops out in two lines : expand the square, $\mathbb{E}[X^2 - 2\mu X + \mu^2] = \mathbb{E}[X^2] - 2\mu\mathbb{E}[X] + \mu^2 = \mathbb{E}[X^2] - \mu^2$.

Which way does it lean ? A symmetric shape has mean $=$ median. When a distribution has a long tail to the right, the tail drags the mean above the median — which is exactly why average income is so much larger than typical income. That gap is not a statistical trick, it is the signature of a right-skewed distribution, and section 7 explains where such shapes come from.

A warning to keep for Part III : these three numbers are not guaranteed to exist. There are perfectly ordinary-looking distributions whose mean is undefined and whose variance is infinite — and when you meet one, nearly every habit built on averages fails at once. Section 12 shows the smallest such example.

PART II — THE MECHANISMS
each shape derived from the process that produces it

3. Counting successes : the binomial

The mechanism. You repeat the same yes/no experiment $n$ times. Each attempt succeeds with the same probability $p$, and the attempts do not influence one another. You count the successes.

That is the whole specification, and the formula follows from it without any cleverness. One particular sequence with $k$ successes and $n-k$ failures has probability $p^{k}(1-p)^{n-k}$ — multiply, because the trials are independent. Then count how many such sequences exist : you are choosing which $k$ of the $n$ slots succeeded, so there are $\binom{n}{k}$ of them.

The binomial distribution with $n$ trials and success probability $p$ : $$ \mathbb{P}(X = k) = \binom{n}{k} p^{k}(1-p)^{n-k}, \qquad k = 0, 1, \dots, n, $$ with $\mathbb{E}[X] = np$ and $\operatorname{Var}(X) = np(1-p)$.

In plain words : “what are the chances of this exact pattern” times “how many patterns look like that”. The mean is obvious — flip a fair coin 100 times and expect 50 heads. The variance is largest at $p = 0.5$, which matches intuition : a coin that lands heads 99 % of the time is very predictable, and so is one that lands heads 1 % of the time. Uncertainty peaks when the outcome is a genuine toss-up.

Figure 2 - One formula, three faces. With a small success probability the mass piles up near zero ; at $p = 0.5$ it is symmetric and at its widest, because that is where a trial is least predictable ; and with enough trials the binomial is already indistinguishable from the bell curve drawn in dashes — the central limit theorem at work on a sum of yes/no trials.

Where it shows up. Conversion rates in an A/B test. Defective items in a batch. The number of heads, obviously. In genetics, how many of $n$ offspring inherit a given allele. Any time you have a fixed number of independent, identical, two-outcome trials, this is your distribution, and it is exact rather than approximate.

Where it fails. The three assumptions are load-bearing. If the trials are not independent (customers influence each other), or $p$ drifts (your conversion rate changes over the week), or the trials are not identical, the binomial will be too narrow — your data will show more spread than $np(1-p)$ allows. That mismatch is itself a useful diagnostic.

4. Counting rare events in time : the Poisson

The mechanism. Now drop the fixed number of trials. Events happen at some average rate $\lambda$, independently of one another, and you count how many land in a fixed window of time or space. There is no natural $n$ — a meteor could strike at any instant.

Here is the derivation that makes the Poisson distribution feel inevitable rather than arbitrary. Chop the window into $n$ slices so tiny that at most one event can fall in each. Each slice then contains an event with probability $\lambda/n$, and the count is binomial :

\[\mathbb{P}(X = k) = \binom{n}{k}\left(\frac{\lambda}{n}\right)^{k}\left(1 - \frac{\lambda}{n}\right)^{n-k}.\]

Now let the slices get infinitely thin. Rewrite the right-hand side as

\[\underbrace{\frac{n(n-1)\cdots(n-k+1)}{n^{k}}}_{\to\; 1} \cdot \frac{\lambda^{k}}{k!} \cdot \underbrace{\left(1 - \frac{\lambda}{n}\right)^{n}}_{\to\; e^{-\lambda}} \cdot \underbrace{\left(1 - \frac{\lambda}{n}\right)^{-k}}_{\to\; 1},\]

where the first factor is a product of $k$ terms each tending to $1$, and the third is the classic limit defining $e$. What survives is :

As $n \to \infty$ with $np = \lambda$ held fixed, the binomial converges to $$ \mathbb{P}(X = k) = \frac{\lambda^{k} e^{-\lambda}}{k!}, \qquad k = 0, 1, 2, \dots $$ with $\mathbb{E}[X] = \operatorname{Var}(X) = \lambda$.

In plain words : the Poisson distribution is what a binomial becomes when there are enormously many chances for something to happen and each chance is tiny. That is why it describes rare events — not rare in the sense of unlikely to happen at all, but rare per opportunity (Poisson, 1837).

Mean equals variance is worth pausing on, because it is a testable signature. Compute both from your count data and divide. A ratio near $1$ is consistent with a Poisson process. A ratio much larger than $1$ — overdispersion — means your events are clustering rather than arriving independently, which is one of the most informative things count data can tell you.

import statistics

def dispersion(counts):
    """Poisson has variance = mean, so this lands near 1.
    Much above 1 means the events are clustering."""
    v = statistics.variance(counts)
    return v / statistics.fmean(counts)

Where it shows up. Radioactive decays per second. Photons hitting a sensor pixel. Calls arriving at a switchboard. Mutations per genome per generation. Typos per page. Goals in a football match. The canonical first example is von Bortkiewicz’s 1898 study of Prussian soldiers killed by horse kicks, which fits a Poisson distribution almost perfectly across ten army corps and twenty years (von Bortkiewicz, 1898) — an unpromising dataset that became famous precisely because the mechanism is so clean : many soldiers, each with a tiny independent chance.

You have already used this limit without naming it. In the Bloom filter post, the probability that a given bit stays $0$ after $kn$ hashes is $(1 - 1/m)^{kn} \approx e^{-kn/m}$. That approximation is the Poisson limit above, with $\lambda = kn/m$ — many hashes, each with a tiny chance of hitting this particular bit.

5. Waiting for the next one : the exponential

The mechanism. Same process as section 4 — events arriving independently at rate $\lambda$ — but now ask a different question. Not how many in a window, but how long until the next one.

The answer follows from the Poisson formula in one step. The waiting time $T$ exceeds $t$ precisely when no event occurs in $[0, t]$. That window contains $\lambda t$ events on average, so by the Poisson formula with $k = 0$ :

\[\begin{equation} \mathbb{P}(T > t) = e^{-\lambda t} \quad\Longrightarrow\quad F(t) = 1 - e^{-\lambda t} \quad\Longrightarrow\quad f(t) = \lambda e^{-\lambda t}. \label{eq:expo} \end{equation}\]

The mean waiting time is $1/\lambda$, which is exactly what it should be : if buses arrive at $4$ per hour, you wait $15$ minutes on average.

Now the property that makes this distribution special, and occasionally shocking.

For an exponential waiting time, and only for it among continuous distributions, $$ \mathbb{P}(T > s + t \mid T > s) = \mathbb{P}(T > t) \quad\text{for all } s, t \ge 0. $$ Proof.   By the definition of conditional probability and $\eqref{eq:expo}$, $$ \mathbb{P}(T > s+t \mid T > s) = \frac{\mathbb{P}(T > s+t)}{\mathbb{P}(T > s)} = \frac{e^{-\lambda(s+t)}}{e^{-\lambda s}} = e^{-\lambda t} = \mathbb{P}(T > t). $$

$\square$

In plain words : having already waited twenty minutes tells you absolutely nothing about how much longer you will wait. The process does not “owe” you an event, and the bus does not become due. The clock resets every instant.

That property is the sharpest modelling test in this whole post, because it is either true of your system or obviously false.

• True : a radioactive atom does not age. An atom that has survived a billion years is statistically identical to a fresh one, which is why decay times are exponential and half-lives are constant.
• False : a machine bearing absolutely does age. A part that has run for ten years is more likely to fail in the next hour than a new one. Using an exponential here will badly underestimate late-life failures, and the standard fix is the Weibull distribution (Weibull, 1951), which adds a shape parameter letting the hazard rate rise (wear-out), fall (infant mortality), or stay flat (which recovers the exponential exactly).

If you want the wait until the $k$-th event rather than the first, you are summing $k$ independent exponentials, and the result is the Gamma distribution with shape $k$ and rate $\lambda$. Same process, third question.

Figure 3 - One process, two distributions. The orange ticks are events arriving independently at a steady rate. Ask how many fall in each window and the answer is Poisson ; ask how much space sits between consecutive ticks and the answer is exponential. They are not two models of two things — they are two questions about the same picture, which is why the two formulas derive from one another in a single line.

Figure 3 is the one to remember, because a single process generates two distributions depending on what you choose to measure. Count the ticks inside fixed windows and you get Poisson. Measure the gaps between ticks and you get exponential. They are not two models — they are two views of the same thing.

6. Adding many small things : the normal

The mechanism. A quantity is the sum of a large number of small, independent contributions, none of which dominates. That is it — and the remarkable part is that the shape of the individual contributions does not matter.

This is the central limit theorem, which the statistical inference post states precisely and leans on throughout. Here we care about it as a generative statement rather than a tool : it tells you when to expect a bell curve in the world.

The normal distribution with mean $\mu$ and standard deviation $\sigma$ : $$ f(x) = \frac{1}{\sigma\sqrt{2\pi}}\, e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}. $$

Why that particular formula ? The honest answer involves the Fourier transform, and it is worth a paragraph because it explains the bell curve rather than merely asserting it. When you add two independent random variables, their densities convolve. Convolution is awkward — but as the Fourier transform post shows, convolution in one domain is plain multiplication in the other. So work with the Fourier transform of the density, $\varphi(t) = \mathbb{E}[e^{itX}]$ : adding $n$ independent copies turns the messy repeated convolution into a simple power $\varphi(t)^{n}$. Expanding $\log\varphi$ near $t = 0$ and rescaling, every distribution with a finite variance flattens into $e^{-t^{2}/2}$, and the function whose Fourier transform is a Gaussian is a Gaussian. The bell curve is the fixed point of addition, and the $e^{-x^{2}}$ is exactly the shape that survives being added to itself.

Where it shows up. Human height : hundreds of genes plus nutrition plus a dozen environmental factors, each contributing a small additive nudge. Measurement error, which is the historical origin — Gauss introduced the curve to model errors in astronomical observations (Gauss, 1809). Blood pressure, exam scores, and above all any average of many observations, which is why it dominates statistical practice even when the raw data is not normal at all.

The normal distribution is a consequence, not a default. Reaching for it because it is familiar is the most common modelling error there is. The next two sections cover the two mechanisms that most often produce data people wrongly force into a bell.

7. Multiplying many small things : the log-normal

The mechanism. Identical to the last one with a single word changed. A quantity is the product of many small independent factors rather than their sum. Instead of “add a nudge”, each step is “grow by a few percent”.

That one word changes everything, and the reason is immediate. If

\[X = X_0 \cdot (1+\varepsilon_1)(1+\varepsilon_2)\cdots(1+\varepsilon_n),\]

then taking logarithms turns the product into a sum :

\[\begin{equation} \log X = \log X_0 + \sum_{i=1}^{n} \log(1+\varepsilon_i). \label{eq:logn} \end{equation}\]

The right-hand side is a sum of many small independent terms — so by the central limit theorem it is normal. Which means $X$ itself is the exponential of a normal variable.

$X$ is log-normal when $\log X$ is normal. Its density on $x > 0$ is $$ f(x) = \frac{1}{x\,\sigma\sqrt{2\pi}}\, e^{-\frac{(\log x - \mu)^{2}}{2\sigma^{2}}}, $$ with median $e^{\mu}$ and mean $e^{\mu + \sigma^{2}/2}$ — the mean strictly larger, always.

In plain words : additive noise gives you a symmetric bell ; multiplicative noise gives you a shape that cannot go below zero, bunches up on the left, and trails off to the right forever. And it explains the mean-above-median gap from section 2 mechanically — you cannot lose more than 100 % but you can gain 500 %, so the upside has more room than the downside.

Figure 4 - The same fourteen random walks, with one word changed. On the left every step adds a small amount, and the endpoints pile into a symmetric bell. On the right every step multiplies by a small factor, and the endpoints pile into a shape that cannot cross zero and trails off to the right — because you can lose at most everything but you can gain without limit. Growth is multiplicative, which is why so much of the natural world is log-normal rather than normal.

Where it shows up. Almost everywhere that things grow. Income and wealth. Stock prices, whose returns compound multiplicatively. The size of particles after repeated fragmentation. Concentrations of substances in biological tissue. File sizes, city sizes, the duration of tasks, the survival time after a diagnosis, species abundances. Limpert and colleagues surveyed the sciences and found the log-normal to be at least as common as the normal, and far more common than the literature acknowledges (Limpert et al., 2001).

The practical consequence. If your data is positive, right-skewed, and generated by growth, take logarithms first and then apply everything from the statistical inference post. On the log scale the machinery works exactly ; on the raw scale a mean and a symmetric confidence interval will both mislead you.

8. No typical size : power laws

The mechanism. Two different processes produce this shape, and both are common. The first is rich-get-richer : the probability of gaining more is proportional to how much you already have — a web page with many links attracts more links, a big city attracts more migrants. The second is self-similarity : the system looks statistically the same at every scale, so there is no characteristic size to speak of.

$X$ follows a power law above $x_m$ when its tail behaves as $$ \mathbb{P}(X > x) = \left(\frac{x}{x_m}\right)^{-\alpha}, \qquad x \ge x_m, $$ with the exponent $\alpha > 0$ controlling how heavy the tail is.

The defining property is not the formula but what it implies.

For a power law, the ratio $$ \frac{\mathbb{P}(X > cx)}{\mathbb{P}(X > x)} = \frac{(cx)^{-\alpha}}{x^{-\alpha}} = c^{-\alpha} $$ does not depend on $x$.

In plain words : "twice as big is $c^{-\alpha}$ times as rare" holds equally whether you are comparing towns of 1000 and 2000 people or cities of 1 million and 2 million. There is no scale at which the rule changes — and therefore no such thing as a "typical" value.

That is the deep contrast with the bell curve. A normal distribution has an unmistakable typical value and everything sits near it : nobody is twice the height of the average adult, ever. A power law has no typical value at all, and the largest observation in your sample is usually of the same order as the sum of all the others.

Where it shows up. City populations and word frequencies, both catalogued by Zipf (Zipf, 1949). Wealth, in Pareto’s original observation that a small fraction of the population held most of the land (Pareto, 1896) — the ancestor of the “80/20” folklore ¹. Earthquake energies, through the Gutenberg–Richter law (Gutenberg & Richter, 1944), which is why there is no such thing as a typical earthquake. The number of links pointing at a web page, explained by preferential attachment (Barabási & Albert, 1999). Citations, file sizes, war casualties, blackout sizes.

Why this matters more than it looks. Recall from $\eqref{eq:var}$ that variance needs $\mathbb{E}[X^{2}]$ to be finite. For a power law, $\mathbb{E}[X^{m}]$ exists only when $m < \alpha$. So :

• if $\alpha \le 2$, the variance is infinite — and the central limit theorem, which requires finite variance, no longer applies ;
• if $\alpha \le 1$, even the mean does not exist.

Many measured exponents in the wild fall between $1$ and $3$, squarely in the dangerous zone. Section 12 shows what that does to ordinary statistical practice.

Figure 5 - The tail test, on log-log axes where a power law becomes a straight line. All three distributions look broadly similar in a histogram ; here they separate completely. The exponential has effectively vanished by $x = 10$, the log-normal by $x = 70$, while the power law is still going at $x = 1000$. This one plot is the fastest way to find out whether your averages are trustworthy.

Be sceptical of claimed power laws. A straight-ish line on a log-log plot is weak evidence : log-normal distributions look almost straight over one or two decades too, and both arise from plausible growth mechanisms (Mitzenmacher, 2004). Distinguishing them needs several decades of data and a proper test rather than a fitted line — Clauset, Shalizi and Newman re-examined two dozen famous "power laws" and found the evidence convincing for only a handful (Clauset et al., 2009). The useful takeaway is usually "this tail is heavy, so averages are unreliable", which holds for both families, rather than a precise exponent.

9. The uniform, and the trick that makes all the others

The uniform distribution on $[0, 1]$ spreads the budget perfectly evenly : $f(x) = 1$ on the interval and $0$ outside. It is what you use when every value in a range is equally plausible and you know nothing more.

It is also the raw material for everything else, through a result that is short, exact, and quietly powerful.

Let $F$ be a continuous, strictly increasing CDF and let $U$ be uniform on $[0,1]$. Then $X = F^{-1}(U)$ has cumulative distribution function $F$.

Proof.   Because $F$ is increasing, $F^{-1}(U) \le x$ exactly when $U \le F(x)$. And for a uniform variable, $\mathbb{P}(U \le u) = u$. Therefore $$ \mathbb{P}(X \le x) = \mathbb{P}\big(F^{-1}(U) \le x\big) = \mathbb{P}\big(U \le F(x)\big) = F(x). $$

$\square$

In plain words : feed uniform randomness into the inverse of any CDF and out comes that distribution. Your computer’s random number generator only knows how to produce uniform numbers ; every exponential, normal or Poisson draw you have ever made was manufactured from them by some version of this idea.

For the exponential it is a one-liner. Invert $F(t) = 1 - e^{-\lambda t}$ to get $F^{-1}(u) = -\log(1-u)/\lambda$ :

import math, random

def sample_exponential(rate=1.0):
    """Inverse transform of F(t) = 1 - exp(-rate * t)."""
    u = random.random()
    return -math.log(1.0 - u) / rate

gaps = [sample_exponential(rate=4) for _ in range(1000)]
sum(gaps) / len(gaps)          # about 0.25 = 1 / rate

This is also why the CDF deserved its own definition in section 1. It is not bookkeeping — it is the bridge between pure randomness and any shape you want.

PART III — CHOOSING, AND BEING WRONG
a decision guide, the family tree, and the cases that break averages

10. Which one should I use ?

The question people usually ask is “which distribution fits my data ?”. It is the wrong question, or at least the second one. The first is “what process produced this data ?” — because if you can answer that, the distribution is implied, and you get a model that will keep working on data you have not seen yet. Fitting shapes without a mechanism gives you a curve that describes today and predicts nothing.

Figure 6 - The lookup table worth memorising, keyed on the mechanism rather than on the shape of your histogram. Read it left to right : describe how the randomness is produced, and the distribution — and with it the assumptions you have just signed up to — follows.

Read it as a set of questions to ask about the mechanism, not about the histogram.

• Is there a fixed number of independent yes/no trials ? Binomial. Ask whether $p$ is really constant and the trials really independent — those are the assumptions that break.
• Are events arriving independently at a steady rate, and you are counting them ? Poisson. Check variance $\approx$ mean.
• Same process, but you are measuring the wait ? Exponential for the next event, Gamma for the $k$-th. Check memorylessness : if the thing ages, use Weibull instead.
• Is the quantity a sum of many small independent effects ? Normal.
• Is it a product of many small independent effects — that is, does it grow by percentages ? Log-normal. Take logs and work there.
• Does the process reward size, or look the same at every scale ? Power law. Expect averages to misbehave.
• Is it a proportion, living strictly between 0 and 1 ? Beta, which is flexible enough to be a U, a bell or a ramp on $[0,1]$ and is the natural partner of the binomial.
• Do you genuinely know nothing except the range ? Uniform.

11. They are all related

The distributions above are not a list of unrelated formulas. They are one connected family, and knowing the connections saves a great deal of memorising.

relationship	result
one Bernoulli trial, repeated $n$ times	Binomial
Binomial with $n \to \infty$, $np = \lambda$ fixed	Poisson
gap between Poisson events	Exponential
sum of $k$ Exponentials	Gamma (shape $k$)
number of failures before the first success	Geometric — the discrete exponential, also memoryless
sum of many independent things, finite variance	Normal
exponential of a Normal	Log-normal
sum of $k$ squared standard Normals	Chi-square with $k$ degrees of freedom
Normal divided by $\sqrt{\text{Chi-square}/k}$	Student’s $t$ with $k$ degrees of freedom

That last row closes a loop worth noticing. The $t$-distribution that the statistical inference post introduces for small samples is not an ad-hoc correction — it is precisely what you get when you divide a normal estimate by an independently estimated spread. Its fat tails are the mathematical price of not knowing $\sigma$.

The geometric distribution has a nice appearance elsewhere on this blog. HyperLogLog works by recording the longest run of leading zeros seen in a stream of hashes. The number of leading zeros in a uniform random bit string is geometric — $\mathbb{P}(\ge k \text{ zeros}) = 2^{-k}$ — and the whole algorithm is an exercise in estimating a parameter of that distribution from an extreme value.

12. When averaging stops working

Everything comfortable about statistics rests on two assumptions that usually hold and occasionally do not : that the mean exists, and that the variance is finite. Here is the smallest example where both fail, and it looks completely innocent.

The Cauchy distribution has density $$ f(x) = \frac{1}{\pi\,(1 + x^{2})}, \qquad x \in \mathbb{R}. $$ It is symmetric, single-peaked and centred at zero — it looks like a bell curve at a glance.

It is not one. The tails fall off like $1/x^{2}$ instead of $e^{-x^{2}/2}$, and that is enough to destroy the mean :

\[\mathbb{E}\big[\lvert X \rvert\big] = \int_{-\infty}^{\infty} \frac{\lvert x \rvert}{\pi(1+x^{2})}\,\mathrm{d}x = \frac{2}{\pi}\int_{0}^{\infty}\frac{x}{1+x^{2}}\,\mathrm{d}x = \frac{1}{\pi}\Big[\log(1+x^{2})\Big]_{0}^{\infty} = \infty.\]

The integral diverges, so the expectation is undefined — not large, undefined. And the consequence is genuinely startling.

If $X_1, \dots, X_n$ are independent Cauchy variables, then their average $\bar{X}_n$ has exactly the same Cauchy distribution as a single one of them.

In plain words : averaging ten thousand measurements is precisely as accurate as taking one. The $\sigma/\sqrt{n}$ law from the statistical inference post does not merely weaken here — it fails completely, because the $\sigma$ it refers to does not exist.

Figure 7 - The running average of the first $n$ draws, for two distributions that look almost identical when you plot their densities. The normal converges quickly and then stops moving, exactly as $\sigma/\sqrt{n}$ predicts. The Cauchy does not converge at all : every so often a single enormous draw arrives and drags the whole average with it, and averaging two thousand observations is no better than averaging one.

You will rarely meet an exact Cauchy in the wild. You will regularly meet things close enough to matter : financial returns, insurance claims, network traffic bursts, file sizes, city sizes, anything generated by a rich-get-richer process. In those cases :

• the sample mean wanders instead of settling, and the value you compute depends heavily on whether one enormous observation happened to land in your sample ;
• confidence intervals built on $\sigma/\sqrt{n}$ are far too narrow, because they assume a variance that is not there ;
• the median and other quantiles remain perfectly well behaved, which is why they are the right summary for heavy-tailed data ;
• and the interesting question changes. For a bell curve you ask “what is the typical value ?”. For a heavy tail the typical value is nearly irrelevant, and you should be asking “how large can the largest one get, and what happens when it does ?”

13. How to tell which one you have

Some honest practice, in the order it is worth doing.

Plot it, and plot it three ways. A plain histogram, then the same data with a logarithmic $x$-axis, then the survival function $\mathbb{P}(X > x)$ on log-log axes. Each transformation straightens a different family : a log-normal becomes symmetric on a log $x$-axis, and a power law becomes a straight line on log-log. Figure 5 is that last plot, and it separates in seconds what a histogram leaves ambiguous.

Use a QQ plot for the middle of the distribution. Sort your $n$ observations, and plot each against the value the candidate distribution predicts for that rank. If the family is right, the points fall on a straight line ; systematic curvature at the ends tells you the tails are wrong even when the centre looks fine.

Use the cheap numerical signatures. They will not prove anything, but they point.

import math, statistics

def quick_read(xs):
    """Cheap signals that suggest a family. Still plot it."""
    n = len(xs)
    m = statistics.fmean(xs)
    sd = statistics.stdev(xs)
    skew = sum((x - m) ** 3 for x in xs) / (n * sd ** 3)
    return {
        "cv": sd / m,          # ~1 exponential, >>1 heavy
        "var_over_mean": sd * sd / m,   # ~1 Poisson, counts
        "skew": skew,           # ~0 normal, >0 right tail
    }

Test the mechanism, not only the shape. This is the part people skip, and it is the most informative. For a claimed Poisson process, check that counts in disjoint windows are uncorrelated. For a claimed exponential, check memorylessness directly : split the data at some threshold $s$ and verify that the remaining waiting time among those that already exceeded $s$ has the same distribution as the whole. For a claimed binomial, check that $p$ is stable over time. A shape that fits but whose mechanism fails will break the moment your data changes.

Finally, do not over-fit. Real data almost never follows a textbook law exactly, and chasing a perfect fit usually means adding parameters that describe this particular sample rather than the process behind it. The purpose of naming a distribution is to import a mechanism, its assumptions and its known failure modes — not to win a goodness-of-fit contest.

14. Conclusion

The catalogue of distributions looks like something to memorise, and it is not. It is a short list of mechanisms, each with a shape attached :

Repeat a fixed number of independent trials and you get a binomial. Let the number of chances become enormous while each one becomes tiny and it collapses into a Poisson. Turn that same process around and ask about the waiting time instead of the count, and you get an exponential — memoryless, which is either exactly right for your system or obviously wrong, and there is rarely any doubt which. Add up many small independent effects and, whatever their individual shapes, you get a normal. Multiply them instead and you get a log-normal, positive and right-skewed, which is what growth produces and what most people mistakenly force into a bell. Let size beget size, or let the system look the same at every scale, and you get a power law, where the notion of a typical value quietly stops meaning anything.

Two things are worth carrying away. The first is the working method : ask what generated the randomness before you ask what fits, because the mechanism gives you a model that survives new data while a fitted curve only describes the old. The second is a warning. Almost every convenient tool in statistics — the mean, the standard error, the confidence interval, the square-root law — assumes a finite variance, and heavy tails simply do not have one. When you find yourself in that territory, reach for medians and quantiles, look hard at your largest observation, and stop expecting averages to settle down. They will not.

References

1. Barabási, A.-L., & Albert, R. (1999). Emergence of Scaling in Random Networks. Science, 286(5439), 509–512.
   @article{Barabasi1999,
     author = {Barab{\'a}si, Albert-L{\'a}szl{\'o} and Albert, R{\'e}ka},
     title = {Emergence of Scaling in Random Networks},
     journal = {Science},
     volume = {286},
     number = {5439},
     pages = {509--512},
     year = {1999}
   }
   

2. Clauset, A., Shalizi, C. R., & Newman, M. E. J. (2009). Power-Law Distributions in Empirical Data. SIAM Review, 51(4), 661–703.
   @article{Clauset2009,
     author = {Clauset, Aaron and Shalizi, Cosma Rohilla and Newman, M. E. J.},
     title = {Power-Law Distributions in Empirical Data},
     journal = {SIAM Review},
     volume = {51},
     number = {4},
     pages = {661--703},
     year = {2009}
   }
   

3. Gauss, C. F. (1809). Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium. Friedrich Perthes and I. H. Besser.
   @book{Gauss1809,
     author = {Gauss, Carl Friedrich},
     title = {Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium},
     year = {1809},
     publisher = {Friedrich Perthes and I. H. Besser}
   }
   

4. Gutenberg, B., & Richter, C. F. (1944). Frequency of Earthquakes in California. Bulletin of the Seismological Society of America, 34(4), 185–188.
   @article{GutenbergRichter1944,
     author = {Gutenberg, Beno and Richter, Charles F.},
     title = {Frequency of Earthquakes in California},
     journal = {Bulletin of the Seismological Society of America},
     volume = {34},
     number = {4},
     pages = {185--188},
     year = {1944}
   }
   

5. Limpert, E., Stahel, W. A., & Abbt, M. (2001). Log-Normal Distributions across the Sciences: Keys and Clues. BioScience, 51(5), 341–352.
   @article{Limpert2001,
     author = {Limpert, Eckhard and Stahel, Werner A. and Abbt, Markus},
     title = {Log-Normal Distributions across the Sciences: Keys and Clues},
     journal = {BioScience},
     volume = {51},
     number = {5},
     pages = {341--352},
     year = {2001}
   }
   

6. Mitzenmacher, M. (2004). A Brief History of Generative Models for Power Law and Lognormal Distributions. Internet Mathematics, 1(2), 226–251.
   @article{Mitzenmacher2004,
     author = {Mitzenmacher, Michael},
     title = {A Brief History of Generative Models for Power Law and Lognormal Distributions},
     journal = {Internet Mathematics},
     volume = {1},
     number = {2},
     pages = {226--251},
     year = {2004}
   }
   

7. Pareto, V. (1896). Cours d’Économie Politique. F. Rouge.
   @book{Pareto1896,
     author = {Pareto, Vilfredo},
     title = {Cours d'{\'E}conomie Politique},
     year = {1896},
     publisher = {F. Rouge}
   }
   

8. Poisson, S. D. (1837). Recherches sur la Probabilité des Jugements en Matière Criminelle et en Matière Civile. Bachelier.
   @book{Poisson1837,
     author = {Poisson, Sim{\'e}on Denis},
     title = {Recherches sur la Probabilit{\'e} des Jugements en Mati{\`e}re Criminelle et en Mati{\`e}re Civile},
     year = {1837},
     publisher = {Bachelier}
   }
   

9. Weibull, W. (1951). A Statistical Distribution Function of Wide Applicability. Journal of Applied Mechanics, 18(3), 293–297.
   @article{Weibull1951,
     author = {Weibull, Waloddi},
     title = {A Statistical Distribution Function of Wide Applicability},
     journal = {Journal of Applied Mechanics},
     volume = {18},
     number = {3},
     pages = {293--297},
     year = {1951}
   }
   

10. Zipf, G. K. (1949). Human Behavior and the Principle of Least Effort. Addison-Wesley.
    @book{Zipf1949,
      author = {Zipf, George Kingsley},
      title = {Human Behavior and the Principle of Least Effort},
      year = {1949},
      publisher = {Addison-Wesley}
    }
    

11. von Bortkiewicz, L. (1898). Das Gesetz der kleinen Zahlen. B. G. Teubner.
    @book{Bortkiewicz1898,
      author = {von Bortkiewicz, Ladislaus},
      title = {Das Gesetz der kleinen Zahlen},
      year = {1898},
      publisher = {B. G. Teubner}
    }
    

1. The folklore is a special case, and pinning it down is instructive. For a Pareto tail with exponent $\alpha$, the richest fraction $q$ of the population holds a fraction $q^{(\alpha-1)/\alpha}$ of the total. Setting “the top 20 % hold 80 %” gives $0.2^{(\alpha-1)/\alpha} = 0.8$, hence $\alpha = \log 5 / \log 4 \approx 1.16$. Note where that lands : below $2$, so this distribution has infinite variance, and only just above $1$, so its mean barely exists at all. The cheerful business-school ratio is describing a genuinely wild distribution. ↩