In trying to better my understanding of mixed distributions, I’m confused why a sampling of values entirely within a subset of the mixed distribution would result in a relatively low likelihood.
My example…
I create two separate distributions (A_dist, B_dist) as well as a mixed distribution ( mix_dist), where mix_dist results in a sampling from both A and B.
A_means = torch.tensor( [1.75, 1.75])
A_stdevs = torch.tensor( [0.3, 0.3])
B_means = torch.tensor( [-0.5, -0.25])
B_stdevs = torch.tensor( [0.25, 0.25])
AB_means = torch.vstack( [ A_means, B_means])
AB_stdevs = torch.vstack( [ A_stdevs, B_stdevs])
A_dist = torch.distributions.Normal( A_means, A_stdevs)
B_dist = torch.distributions.Normal( B_means, B_stdevs)
AB_dist = torch.distributions.Independent( torch.distributions.Normal( AB_means, AB_stdevs), 1)
mix_weight = torch.distributions.Categorical( torch.tensor( [1.0, 1.0]))
mix_dist = torch.distributions.MixtureSameFamily( mix_weight, AB_dist)
To verify the distributions are as I expect, I’ve created a scatter plot…
A_samp = A_dist.sample( (500,))
B_samp = B_dist.sample( (500,))
mix_samp = mix_dist.sample( (500,))
plt.scatter( A_samp[:,0], A_samp[:, 1], alpha=0.3, label='A')
plt.scatter( B_samp[:,0], B_samp[:, 1], alpha=0.3, label='B')
plt.scatter( mix_samp[:, 0], mix_samp[:, 1], alpha=0.3, label='mix')
plt.legend()
To determine likelihood I am taking torch.exp( DIST.log_prog( SAMPLE))
print( f"Likelihood of A samples within A dist : {torch.exp( A_dist.log_prob( A_dist.sample((5000,))).mean( )):0.4f}")
print( f"Likelihood of B samples within A dist : {torch.exp( A_dist.log_prob( B_dist.sample((5000,))).mean( )):0.4f}")
print( f"Likelihood of A samples within mix dist: {torch.exp( mix_g.log_prob( A_dist.sample((5000,))).mean( )):0.4f}")
print( f"Likelihood of B samples within mix dist: {torch.exp( mix_g.log_prob( B_dist.sample((5000,))).mean( )):0.4f}")
which shows…
Likelihood of A samples within A dist : 0.8089
Likelihood of B samples within A dist : 0.0000
Likelihood of A samples within mix dist: 0.3318
Likelihood of B samples within mix dist: 0.4642
It make sense to me that A samples would have a high likelihood of falling within the A distribution. And that B samples would have low/no likelihood of falling within the A distribution. However, since samples from both A and B fall entirely within the mix distribution, shouldn’t they result in a higher likelihood of falling within the mix distribution?