6

Given there are two matrices of dimensionality 100x2 with absolute values ranging from -50 to +50. Is it possible to determine the kl-divergence by applying the entropy algorithm from scipy.stats to the two flattened vectors of size 200?

piutu
  • 61
  • 1
  • 2

2 Answers2

2

I think you can. Just normalize both of the vectors to be sure they are distributions. Then you can apply the kl divergence . Note the following: - you need to use a very small value when calculating the kl-d to avoid division by zero. In other words , replace any zero value with ver small value - kl-d is not a metric . Kl(AB) does not equal KL(BA) . If you are interested in it as a metric you have to use the symmetric kl = (Kl(AB) +KL(BA) )/2

Bashar Haddad
  • 2,049
  • 1
  • 15
  • 18
0

Even though it is possible, it probably does not produce a meaningful measure. I would advice however to be careful with these points:

  • A probability distribution should represent probabilities. You can not take any arbitrary vector and consider them as probabilities distributions. The key here is to have a sound explanation for why outcome X has a say .2 (arbitrary) probability. The most simple way of "calculating" probabilities is considering it is the frequency of occurrences in an experiment comprised of N trials.

  • What your dimensions represent? To flatten a matrix only means you now have a vector. Having a vector might be enough for you function to run, but might not give you a meaningful measure.

  • stats.entropy normalizes your vector if it does not sums up to 1. Often you will want to use the same kind of normalization on your pipeline. Be sure to normalize it before passing to entropy() if necessary. Here is the normalization performed by entropy:

    pk = 1.0*pk / np.sum(pk, axis=0)
Rabbit
  • 101
  • 3