कुल्बैक-लीबलर दो गामा वितरण के बीच विचलन

15

गामा वितरण parameterize करने के लिए चुनना $\Gamma(b,c)$ पीडीएफ द्वारा $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ के बीच Kullback-Leibler विचलन $\Gamma(b_q,c_q)$ और $\Gamma(b_p,c_p)$ द्वारा [1] के रूप में दिया जाता है

\begin{aligned} K L_{G a} (b_{q}, c_{q}; b_{p}, c_{p}) & = (c_{q} - 1) Ψ (c_{q}) - \log b_{q} - c_{q} - \log Γ (c_{q}) + \log Γ (c_{p}) \\ + c_{p} \log b_{p} - (c_{p} - 1) (Ψ (c_{q}) + \log b_{q}) + \frac{b_{q} c_{q}}{b_{p}} \end{aligned}

$\begin{align} KL_{Ga}(b_q,c_q;b_p,c_p) &= (c_q-1)\Psi(c_q) - \log b_q - c_q - \log\Gamma(c_q) + \log\Gamma(c_p)\\ &\qquad+ c_p\log b_p - (c_p-1)(\Psi(c_q) + \log b_q) + \frac{b_qc_q}{b_p} \end{align}$

मेरा अनुमान है कि कि $\Psi(x):= \Gamma'(x)/\Gamma(x)$ है digamma समारोह ।

यह बिना किसी व्युत्पत्ति के दिया गया है। मुझे ऐसा कोई संदर्भ नहीं मिल रहा है जो इसे प्राप्त करता हो। कोई मदद? एक अच्छा संदर्भ पर्याप्त होगा। एक गामा पीडीएफ के खिलाफ मुश्किल हिस्सा को एकीकृत कर रहा है $\log x$ ।

[१] डब्ल्यूडी पेनी, केएल-डायवर्जेंस ऑफ नॉर्मल, गामा, डिरिचलेट और विशरट डेंसिटीज , यहां उपलब्ध: www.fil.ion.ucl.ac.uk/~wpenny/publications/pensities.ps

kullback-leibler gamma-distribution exponential-family

— इयान लैंगमोर
स्रोत

2

के संबंध में पीडीएफ के व्युत्पन्न ले रहा है

द्वारा प्रस्तुत की कारक

आप देख रहे हैं: इसलिए digamma शो।

c

$c$

l o g (x)

$log(x)$

— व्हिबर er

यदि आप पियरे बल्दी और लॉरेंट इट्टी (2010) में "बिट्स एंड वोज़्स: ए बायेसियन थ्योरी ऑफ़ एप्लीकेशन" के साथ आश्चर्यचकित होते हैं, तो न्यूरल नेटवर्क्स 23: 649-666, आप पाएंगे कि समीकरण 73 दो गामा pdfs के बीच केएल विचलन देता है। ध्यान रखें, हालांकि, ऐसा लगता है कि सूत्र गलत-मुद्रित है।

— मिस्टर क्लारनेट

मैं एक ही समस्या का समाधान खोज रहा है और इस लगता है एक उपयोगी है।

— यी यांग

15

केएल डाइवर्जेंस फॉर्म के अभिन्न अंग का अंतर है

$$ \ eqalign {I (a, b, c, d) & = \ int_0 ^ {\ infty} \ log \ left (\ frac {e ^ {- x / a} x ^ {b-1}} {a ^ b \ Gamma (b)} \ right) \ frac {e ^ {- x / c} x ^ {d-1}} {c ^ d \ Gamma (d)} dx \

& = - \ frac {1} {a} \ int_0 ^ \ infty \ frac {x ^ de ^ {- x / c}} {c ^ d \ Gamma (d)} \, dx - \ log (^ b) \ Gamma (b)) \ int_0 ^ \ infty \ frac {e ^ {- x / c} x ^ {d-1}} {c ^ d \ Gamma (d)} \ _, dx \ & \ quad + (b) 1) \ int_0 ^ \ infty \ log (x) \ frac {e ^ {- x / c} x ^ {d-1}} {c ^ d \ Gamma (d)} \, dx \

& = - \ frac {cd} {a} - \ log (a ^ b \ Gamma (b)) + (b-1) \ int_0 ^ \ infty \ log (x) \ frac {e ^ {- x / c } x ^ {d-1}} {c ^ d \ Gamma (d)} \, dx}

हमें बस दाहिने हाथ के अभिन्न अंग से निपटना है, जिसे देखने से प्राप्त होता है

\begin{aligned} \frac{\partial}{\partial d} Γ (d) = & \frac{\partial}{\partial d} \int_{0}^{\infty} e^{- x / c} \frac{x^{d - 1}}{c^{d}} d x \\ = & \frac{\partial}{\partial d} \int_{0}^{\infty} e^{- x / c} \frac{(x / c)^{d - 1}}{c} d x \\ = & \int_{0}^{\infty} e^{- x / c} \frac{x^{d - 1}}{c^{d}} \log \frac{x}{c} d x \\ = & \int_{0}^{\infty} \log (x) e^{- x / c} \frac{x^{d - 1}}{c^{d}} d x - \log (c) Γ (d) . \end{aligned}

$\eqalign{ \frac{\partial}{\partial d}\Gamma(d) =& \frac{\partial}{\partial d}\int_0^{\infty}e^{-x/c}\frac{x^{d-1}}{c^d}dx\\ =& \frac{\partial}{\partial d} \int_0^\infty e^{-x/c} \frac{(x/c)^{d-1}}{c}\,dx\\ =&\int_0^\infty e^{-x/c}\frac{x^{d-1}}{c^d} \log\frac{x}{c} \,dx\\ =&\int_0^{\infty}\log(x)e^{-x/c}\frac{x^{d-1}}{c^d}dx - \log(c)\Gamma(d). }$

जहां से

\frac{b - 1}{Γ (d)} \int_{0}^{\infty} \log (x) e^{- x / c} (x / c)^{d - 1} d x = (b - 1) \frac{Γ^{'} (d)}{Γ (d)} + (b - 1) \log (c) .

$\frac{b-1}{\Gamma(d)}\int_0^{\infty} \log(x)e^{-x/c}(x/c)^{d-1}dx = (b-1)\frac{\Gamma'(d)}{\Gamma(d)} + (b-1)\log(c).$

Plugging into the preceding yields

I (a, b, c, d) = \frac{- c d}{a} - \log (a^{b} Γ (b)) + (b - 1) \frac{Γ^{'} (d)}{Γ (d)} + (b - 1) \log (c) .

$I(a,b,c,d)=\frac{-cd}{a} -\log(a^b\Gamma(b))+(b-1)\frac{\Gamma'(d)}{\Gamma(d)} + (b-1)\log(c).$

The KL divergence between $\Gamma(c,d)$ and $\Gamma(a,b)$ equals $I(c,d,c,d) - I(a,b,c,d)$ , which is straightforward to assemble.

Implementation Details

Gamma functions grow rapidly, so to avoid overflow don't compute Gamma and take its logarithm: instead use the log-Gamma function that will be found in any statistical computing platform (including Excel, for that matter).

$\Gamma^\prime(d)/\Gamma(d)$ $\Gamma,$ $\psi,$ the digamma function. If it's not available to you, there are relatively simple ways to approximate it, as described in the Wikipedia article.

Here, to illustrate, is a direct R implementation of the formula in terms of $I$ . This does not exploit an opportunity to simplify the result algebraically, which would make it a little more efficient (by eliminating a redundant calculation of $\psi$ ).

#
# `b` and `d` are Gamma shape parameters and
# `a` and `c` are scale parameters.
# (All, therefore, must be positive.)
#
KL.gamma <- function(a,b,c,d) {
  i <- function(a,b,c,d)
    - c * d / a - b * log(a) - lgamma(b) + (b-1)*(psigamma(d) + log(c))
  i(c,d,c,d) - i(a,b,c,d)
}
print(KL.gamma(1/114186.3, 202, 1/119237.3, 195), digits=12)

— whuber
स्रोत

2

Good answer. Thanks! I believe that there is a sign error however in the fourth equality. Also, your gamma pdf should have an extra factor of 'c' in the denominator. Would you like me to edit it?

— Ian Langmore

@Ian You're right; I usually write the measure as

d x / x

$dx/x$ and by not doing that I omitted that extra factor of

c

$c$ . Good catch on the sign mistake. If you would like to make the edits, feel free!

— whuber

2

I made the corrections.

— Ian Langmore

10

The Gamma distribution is in the exponential family because its density can be expressed as:

\begin{aligned} f (x ∣ θ) & = \exp (η (θ) \cdot T (x) - g (θ) + h (x)) \end{aligned}

$\begin{align} \newcommand{\mbx}{\mathbf{x}} \newcommand{\btheta}{\boldsymbol{\theta}} f(\mbx \mid \btheta) &= \exp\bigl(\eta(\btheta) \cdot T(\mbx) - g(\btheta) + h(\mbx)\bigr) \end{align}$

Looking at the Gamma density function, its log-normalizer is

g (θ) = \log (Γ (c)) + c \log (b)

$g(\btheta) = \log(\Gamma(c)) + c\log(b)$ with natural parameters

θ = [\begin{matrix} c - 1 \\ - \frac{1}{b} \end{matrix}]

$\btheta = \left[\begin{matrix}c-1\\-\frac1 b\end{matrix}\right]$

All distributions in the exponential family have KL divergence:

\begin{aligned} K L (q; p) & = g (θ_{p}) - g (θ_{q}) - (θ_{p} - θ_{q}) \cdot \nabla g (θ_{q}) . \end{aligned}

$\begin{align} KL(q; p) &= g(\btheta_p) - g(\btheta_q) - (\btheta_p-\btheta_q) \cdot \nabla g(\btheta_q). \end{align}$

There's a really nice proof of that in:

Frank Nielsen, École Polytechnique, and Richard Nock, Entropies and cross-entropies of exponential families.

— Neil G
स्रोत

Didn't know this. Just a quick question - the

g (.)

$g(.)$ function, does it have to be the same for

θ_{p}

$\theta_p$ as for

θ_{q}

$\theta_q$ ? So for example, would the above formula be valid for KL divergence of normal pdf from gamma pdf?

— probabilityislogic

1

Yes, this formula is for two distributions in the same exponential family.

— Neil G