The purpose of this paper is simply to provide another categorical treatment of entropy of probability distributions, which turns out to be computed in terms of a rig homomorphism; our treatment also generalises to cross entropy (and thus to Kullback–Leibler divergence). What is particularly interesting about the treatment outlined here is that we can somewhat “visualise” the notion of entropy in terms of sizes of coding schemes (cf. Section 6). Not only that, but classical entropy is only homomorphic in the product of distributions, whereas the notion that we describe here is homomorphic in both the product and the sum.

A brief outline of this paper is as follows:

This section is simply a brief summary of content from [1], repeated here for the convenience of the reader.

Following [1], we can think of Dirichlet polynomials as functors ${\mathtt{FSet}}^{\mathrm{o}\mathrm{p}}\to \mathtt{FSet}$, where $\mathtt{FSet}$ is the category of finite sets. Indeed, given a natural number $n\in \mathbb{N}$, the exponential $n^{\mathrm{y}}$ can be thought of as the Yoneda embedding of the set with n elements, i.e., where $\underline{n}=\{1,\dots ,n\}$. (For typographical convenience, we sometimes use the notation n and $\underline{n}$ interchangeably. In particular, we write e.g., $d\left(0\right)$ instead of $d\left(\underline{0}\right)$). Then the addition of exponentials corresponds to the coproduct of the corresponding representable functors (and so multiplication by a natural number $a_n$ corresponds to the $a_n$-fold coproduct of the representable functor with itself). This means that evaluating a Dirichlet polynomial at some natural number n corresponds to evaluating the corresponding functor on the finite set $\underline{n}$.

Note that $0^{\mathrm{y}}$ is not the initial object 0, since i.e., $0^{\mathrm{y}}\ne 0$.

When we think of Dirichlet polynomials as functors ${\mathtt{FSet}}^{\mathrm{o}\mathrm{p}}\to \mathtt{FSet}$, addition is given by the coproduct (disjoint union of sets), and multiplication by the product (Cartesian product of sets). This means that working with Dirichlet polynomials in $\mathtt{Dir}$ really is like working with polynomials, in the sense that addition and multiplication are exactly “as expected”.

There is a more geometric interpretation of objects of $\mathsf{Dir}$ as set-theoretic bundles, i.e., (isomorphism classes of) functions $E\to B$, where $E,B\in \mathtt{FSet}$, given as follows: to the Dirichlet polynomial $d:{\mathtt{FSet}}^{\mathrm{o}\mathrm{p}}\to \mathtt{FSet}$, we associate the function ${\pi }_d=d(!):d\left(1\right)\to d\left(0\right)$ induced by the unique function $!:\underline{0}\to \underline{1}$. (This vague statement can be upgraded to an equivalence of categories: ([1], [Theorem 4.6]) for more details).

For example, to the polynomial $d\left(\mathrm{y}\right)=4^{\mathrm{y}}+4·1^{\mathrm{y}}$, we associate the bundle

Note that bundles also form a rig, where the sum is given by disjoint union of sets, and the product is given by the Cartesian product of sets, both on the base and the total space. Further, the equivalence between Dirichlet polynomials and bundles respects the rig structures (cf. [1], Theorem 4.6). Because of this, we often switch freely between thinking of Dirichlet polynomials as functors $d:{\mathtt{FSet}}^{\mathrm{o}\mathrm{p}}\to \mathtt{FSet}$, as bundles ${\pi }_d:d\left(1\right)\to d\left(0\right)$, and simply as functions of the form ${\sum }_{j=0}^n{a}_j·j^{\mathrm{y}}$.

Using the fact that the sum of bundles is given by the disjoint union of sets, we can use this above definition to write any Dirichlet polynomial d as (where ∑ is the coproduct in $\mathtt{FSet}$).

Given the correspondence between Dirichlet polynomials and bundles, we might rightly ask why we should prefer to work with the former over the latter. For one possible answer to this, see Section 6.

The interpretation of Dirichlet polynomials as bundles helps us to understand how they relate to probability theory. Imagine flipping a coin eight times and observing five heads and three tails; we refer to “heads” and “tails” as outcomes, and each of the eight flips as draws; every draw has an associated outcome.

Consider some bundle ${\pi }_d:d\left(1\right)\to d\left(0\right)$. We can think of $d\left(0\right)$ as the set of outcomes, and $d\left(1\right)$ as the set of draws; the fibre ${\pi }_d^{-1}\left(x\right)$ over an outcome $x\in d\left(0\right)$ corresponds to all the draws that lead to the outcome x, and so we obtain a probability distribution on $d\left(0\right)$ by setting $\mathbb{P}(X=x)=\frac{|{\pi }_d^{-1}\left(x\right)|}{\left|d\right(0\left)\right|}$. Conversely, any rational distribution (i.e., a distribution such that all probabilities are rational numbers ) on a finite set arises in this way: take the finite set as the set of outcomes; take the least common multiple of the denominators of all the probabilities as the cardinality of $d\left(1\right)$; and then take $\mathbb{P}(X=x)·\left|d\right(1\left)\right|$ many elements of $d\left(1\right)$ to be in the fibre of $x\in d\left(0\right)$.

Under this interpretation of Dirichlet polynomials as empirical distributions, multiplication $d·e$ corresponds to taking the product distribution.

The fact that $\mathsf{Rect}$ is indeed a rig follows from the fact that its multiplication distributes over its addition:

With this definition, along with Proposition 1, we see that

Readers might recognise $H\left(d\right)$ as being the Shannon entropy of the corresponding probability distribution (cf. [5]). The convention for naming the sides of a rectangle is from [6] (Figure 1).

Note that, in the above example, even though both $L\left(d\right)$ and $W\left(d\right)$ have non-integer values, the formula implied by our choice of nomenclature still holds: the area $A\left(d\right)$ is equal to the length $L\left(d\right)$ times the width $W\left(d\right)$. This leads us to our main theorem. It says that the Shannon entropy, which is only homomorphic in products of distributions, can be computed in terms of the width and area, which together are homomorphic in both sums and products of distributions. We will explain this in more detail in Section 6.

We have mentioned many times that Dirichlet polynomials are equivalent to set-theoretic bundles, so the natural question to ask is “why, then, should we work with the former instead of the latter?”. One answer to this is question is the fact that entropy does not respect bundle morphisms: we cannot functorially assign a morphism between entropies to morphisms, since we are working with arbitrary morphisms of bundles. (If, however, we restrict to only morphisms given by pushforward, then [7] tells us (via Faddeev’s theorem) that the only possible functorial definition of entropy is given by the relative entropy, i.e., the difference of the entropies of the source and the target). This makes it seem rather bad to work with a category (such as that of bundles) instead of simply a rig (such as that of Dirichlet polynomials). Of course, this isn’t an entirely satisfactory answer, since we do care about the notion of morphisms for Dirichlet polynomials (for example, Corollary 2 tells us that the width can be expressed in terms of the number of certain morphisms). In light of Theorem 1, however, we might consider the following possibility: both area and length can be expressed in terms of $d\left(0\right)$, $d\left(1\right)$, and $d\left[i\right]$ (for $i\in d\left(0\right)$), and we could define the width by $W\left(d\right)A\left(d\right)/L\left(d\right)$.

A better answer to this question might be the following: the rig homomorphism $h:\mathsf{Dir}\to \mathsf{Rect}$ is incredibly simple, since it just maps $n^{\mathrm{y}}$ to $(n,n)$; from this computationally simple homomorphism, however, we can recover entropy (as $log\left(A\right(d)/W(d\left)\right)$), without making any reference to the classical equation that defines it (“negative the sum of probabilities of the log of the probabilities”), but instead relying on the fact that $\mathsf{Rect}$ encodes the weighted geometric mean. That is, $H\left(d\right)$ is only homomorphic in the product of distributions, whereas the pair $\left(A\right(d),W(d\left)\right)$ is homomorphic in both the product and the sum.

Now, the entropy $H\left(d\right)=logL\left(d\right)$ can be understood (via Huffman coding, cf. [8]) as the average number of bits needed to code a single outcome (over a long enough message). What is also true, however, is that the width (which is obtained purely “algebraically”, i.e., from the rig homomorphism $h:\mathsf{Dir}\to \mathsf{Rect}$) gives similar information: by Theorem 1, combined with the previous sentence, $logW\left(d\right)$ is the average number of bits needed to code the draw, given an outcome (in the same Huffman coding as before). This answers the question of “what is special about the bundle defined by the width and length” with “it describes the optimal encoding of draws, given outcomes”.

As for the picture in Example 6, we can now understand the hand-wavy explanation a bit better (but still just as hand-wavy-ly): we take our original “half-filled” rectangle $d\left(1\right)$ and pour its contents into a new rectangle, of length $L\left(d\right)$, and then “slosh the contents around” until they lie flat, and then put a lid on it; the rectangle will be perfectly filled up, and the placement of the lid will be given by $W\left(d\right)$.

We also mentioned, in Corollary 3, that the width $W\left(d\right)$ of any Dirichlet polynomial d is an algebraic number, but the actual result is slightly more interesting that this: Corollary 2 tells us that $W{\left(d\right)}^{A\left(d\right)}$ is equal to the cardinality of the set ${\mathtt{Dir}}_{/d\left(0\right)}(d,d)$. We already know how to understand endomorphisms of d that fix $d\left(0\right)$ as endomorphisms of $d\left(1\right)$ that fix the outcome; we can understand $W{\left(d\right)}^{A\left(d\right)}$ as maps from $A\left(d\right)$ to $W\left(d\right)$; roughly speaking, such a map $f:A\left(d\right)\to W\left(d\right)$ determines the remaining ambiguity in determining a draw, given its outcome.

Everything above can be viewed as a specific example of the analogous cross notions. That is, given two Dirichlet polynomials, we can define their cross area, cross width, etc. as follows.

By definition, $X(d,d)=X\left(d\right)$ for $X\in \{A,W,H,L\}$. That is, just as cross entropy is a generalisation of entropy, the notions of cross width, etc., generalise the notions of width, etc.