Test for difference between 2 empirical discrete distributions

8

I have test data where I have several large samples from discrete distributions which I am using as empirical distributions. I am wanting to test whether the distributions are actually different and what the difference in means is for those distributions which are actually different.

Since they are discrete distributions my understanding is that the Kolmogorov-Smirnov test is invalid due to the underlying continuous distribution assumption. Would the Chi-Squared test be the correct test for whether the distributions are actually different?

What test would I use for the difference in means? Would a better approach be to sample from the distributions and take the difference and then perform analysis on the distribution of the difference?

Wallhood

Posted 2014-03-04T19:01:54.363

Reputation: 175

Yes, the $\chi^2$-test is the correct one. The accepted answer to this question elaborates on that. distribution 1 = urn 1 and distribution 2 = urn 2. There, the values of the random variables are colours and in your case probably something else, e.g. discrete numbers.

– Georg – 2014-03-04T19:36:24.793

Thank you for the feedback. Is there a test for what the difference in means for when the chi-squared test confirms that the distributions are different? – Wallhood – 2014-03-04T19:53:11.473

Would a better approach be to sample from the distributions and take the difference and then perform analysis on the difference? – Wallhood – 2014-03-04T19:59:52.647

Answers

10

1) The Kolmogorov-Smirnov can still be used, but if you use the tabulated critical values it will be conservative (which is only a problem because it pushes down your power curve). Better to get the permutation distribution of the statistic, so that your significance levels are what you choose them to be. This will only make a big difference if there are a lot of ties. This change is really easy to implement. (But the K-S test isn't the only possible such comparison; if one is computing permutation distributions anyway, there are other possibilities.)

2) vanilla chi-square goodness of fit tests for discrete data are generally, to my mind, a really bad idea. If the above potential loss of power stopped you using the K-S test, the problem with the chi-square is often much worse - it throws out the most critical information, which is the ordering among the categories (the observation values), deflating its power by spreading it across alternatives that don't consider the ordering, so that it's worse at detecting smooth alternatives -- like a shift of location and scale for example). Even with the bad effects of heavy ties above, the KS test in many cases still have better power (but with the advantage of lowering the Type I error rate by way of compensation).

The chi-square can also be modified to take account of the ordering (partition the chisquare into linear, quadratic, cubic etc components via orthogonal polynomials and use only the low order few terms - 4 to 6 are common choices). Papers by Rayner and Best (and others) discuss this approach, which arises out of Neyman-Barton smooth tests. This is a good approach but if you don't have access to software for it, it may take a while to set up.

Either modified approach should be fine, but if you're not going to modify either approach, it's not necessarily the case that the chi-square will be better than the KS test -- in some situations it might be better ... or it may be substantially worse.

If the ties aren't heavy (i.e. there are lots of different values taken by the data), I'd consider the KS as-is. If they're moderate, I'd look to calculate the permutation distribution. If they're very heavy (i.e. the data only take a few different values), the plain chi-square may be competitive.

Glen_b

Posted 2014-03-04T19:01:54.363

Reputation: 190 866

Thank you for the warning. I'll take that into consideration when I decide to use either the K-S test or the Chi-Squared – Wallhood – 2014-03-04T22:18:04.807