Moment Generating Functions

---

6.1.3 Moment Generating Functions

Here, we will introduce and discuss moment generating functions (MGFs). Moment generating functions are useful for several reasons, one of which is their application to analysis of sums of random variables. Before discussing MGFs, let’s define moments.

Definition . The n th moment of a random variable $X$ is defined to be $E[Xn]$ . The n th central moment of $X$ is defined to be $E[(X-EX)n]$ .

For example, the first moment is the expected value $E[X]$ . The second central moment is the variance of $X$ . Similar to mean and variance, other moments give useful information about random variables.

The moment generating function (MGF) of a random variable $X$ is a function $MX(s)$ defined as

$$MX(s)=E[esX].$$

We say that MGF of $X$ exists, if there exists a positive constant $a$ such that $MX(s)$ is finite for all $s\in [-a,a]$ .

Before going any further, let’s look at an example.

---

Example
For each of the following random variables, find the MGF.

1. $X$ is a discrete random variable, with PMF PX(k)={13k=123k=2
2. $Y$ is a $Uniform(0,1)$ random variable.

• Solution
  • 1. For $X$ , we have $$MX(s)=E[esX]=13es+23e2s.$$

       which is well-defined for all $s∈R$ .
       

    2. For $Y$ , we can write $$MY(s)=E[esY]=\int 01esydy=es-1s.$$ Note that we always have $MY(0)=E[e0\cdot Y]=1$ , thus $MY(s)$ is also well-defined for all $s\in R$ .

---

Why is the MGF useful? There are basically two reasons for this. First, the MGF of $X$ gives us all moments of $X$ . That is why it is called the moment generating function. Second, the MGF (if it exists) uniquely determines the distribution. That is, if two random variables have the same MGF, then they must have the same distribution. Thus, if you find the MGF of a random variable, you have indeed determined its distribution. We will see that this method is very useful when we work on sums of several independent random variables. Let’s discuss these in detail.

Finding Moments from MGF:

Remember the Taylor series for $ex$ : for all $x\in R$ , we have

$$ex=1+x+x22!+x33!+...=\sum k=0\infty xkk!.$$

Now, we can write

$$esX=\sum k=0\infty (sX)kk!=\sum k=0\infty Xkskk!.$$

Thus, we have

$$MX(s)=E[esX]=\sum k=0\infty E[Xk]skk!.$$

We conclude that the $k$ th moment of $X$ is the coefficient of $skk!$ in the Taylor series of $MX(s)$ . Thus, if we have the Taylor series of $MX(s)$ , we can obtain all moments of $X$ .

---

Example
If $Y\sim Uniform(0,1)$ , find $E[Yk]$ using $MY(s)$ .

• Solution
  • We found $MY(s)$ in Example 6.3, so we have $$MY(s)=es-1s=1s(\sum k=0\infty skk!-1)=1s\sum k=1\infty skk!=\sum k=1\infty sk-1k!=\sum k=0\infty 1k+1skk!.$$ Thus, the coefficient of $skk!$ in the Taylor series for $MY(s)$ is $1k+1$ , so $$E[Xk]=1k+1.$$

---

We remember from calculus that the coefficient of $skk!$ in the Taylor series of $MX(s)$ is obtained by taking the $k$ th derivative of $MX(s)$ and evaluating it at $s=0$ . Thus, we can write

$$E[Xk]=dkdskMX(s)|s=0.$$

We can obtain all moments of $Xk$ from its MGF:

$$MX(s)=\sum k=0\infty E[Xk]skk!,$$ $$E[Xk]=dkdskMX(s)|s=0.$$

---

Example
Let $X\sim Exponential(\lambda )$ . Find the MGF of $X$ , $MX(s)$ , and all of its moments, $E[Xk]$ .

• Solution

  • Recall that the PDF of $X$ is

    $$fX(x)=\lambda e-\lambda xu(x),$$

    where $u(x)$ is the unit step function. We conclude

    $$MX(s)=E[esX]=\int 0\infty \lambda e-\lambda xesxdx=[-\lambda \lambda -se-(\lambda -s)x]0\infty ,fors<\lambda =\lambda \lambda -s,fors<\lambda .$$

    Therefore, $MX(s)$ exists for all $s<\lambda$ . To find the moments of $X$ , we can write

    $$MX(s)=\lambda \lambda -s=11-s\lambda =\sum k=0\infty (s\lambda )k,for|s\lambda |<1=\sum k=0\infty k!\lambda kskk!.$$

    We conclude that

    $$E[Xk]=k!\lambda k,fork=0,1,2,...$$

---

Example
Let $X\sim Poisson(\lambda )$ . Find the MGF of $X$ , $MX(s)$ .

• Solution
  • We have $$PX(k)=e-\lambda \lambda kk!,fork=0,1,2,...$$ Thus, $$MX(s)=E[esX]=\sum k=0\infty eske-\lambda \lambda kk!=e-\lambda \sum k=0\infty esk\lambda kk!=e-\lambda \sum k=0\infty (\lambda es)kk!=e-\lambda e\lambda es(Taylorseriesforex)=e\lambda (es-1),foralls\in R.$$

---

As we discussed previously, the MGF uniquely determines the distribution. This is a very useful fact. We will see examples of how we use it shortly. Right now let’s state this fact more precisely as a theorem. We omit the proof here.
Theorem Consider two random variables $X$ and $Y$ . Suppose that there exists a positive constant $c$ such that MGFs of $X$ and $Y$ are finite and identical for all values of $s$ in $[-c,c]$ . Then,

$$FX(t)=FY(t),forallt\in R.$$

---

Example
For a random variable $X$ , we know that

$$MX(s)=22-s,fors\in (-2,2).$$

Find the distribution of $X$ .

• Solution
  • We note that the above MGF is the MGF of an exponential random variable with $\lambda =2$ (Example 6.5). Thus, we conclude that $X\sim Exponential(2)$ .

---

Sum of Independent Random Variables:

Suppose $X1$ , $X2$ ,…, $Xn$ are $n$ independent random variables, and the random variable $Y$ is defined as

$$Y=X1+X2+\cdots +Xn.$$

Then,

$$MY(s)=E[esY]=E[es(X1+X2+\cdots +Xn)]=E[esX1esX2\cdots esXn]=E[esX1]E[esX2]\cdots E[esXn](sincetheX{i}^{\prime }sareindependent)=MX1(s)MX2(s)\cdots MXn(s).$$

If $X1$ , $X2$ ,…, $Xn$ are $n$ independent random variables, then

$$MX1+X2+\cdots +Xn(s)=MX1(s)MX2(s)\cdots MXn(s).$$

---

Example
If $X\sim Binomial(n,p)$ find the MGF of $X$ .

• Solution

  • We can solve this question directly using the definition of MGF, but an easier way to solve it is to use the fact that a binomial random variable can be considered as the sum of $n$ independent and identically distributed (i.i.d.) Bernoulli random variables. Thus, we can write

    $$X=X1+X2+\cdots +Xn,$$

    where $Xi\sim Bernoulli(p)$ . Thus,

    $$MX(s)=MX1(s)MX2(s)\cdots MXn(s)=(MX1(s))n(sincetheX{i}^{\prime }sarei.i.d.)$$

    Also,

    $$MX1(s)=E[esX1]=pes+1-p.$$

    Thus, we conclude

    $$MX(s)=(pes+1-p)n.$$

---

Example
Using MGFs prove that if $X\sim Binomial(m,p)$ and $Y\sim Binomial(n,p)$ are independent, then $X+Y\sim Binomial(m+n,p)$ .

• Solution
  • We have $$MX(s)=(pes+1-p)m,MY(s)=(pes+1-p)n.$$ Since $X$ and $Y$ are independent, we conclude that $$MX+Y(s)=MX(s)MY(s)=(pes+1-p)m+n,$$ which is the MGF of a $Binomial(m+n,p)$ random variable. Thus, $X+Y\sim Binomial(m+n,p)$ .

---

← previous

next →

---

The print version of the book is available on Amazon.
**Practical uncertainty:**Useful Ideas in Decision-Making, Risk, Randomness, & AI