Significant Changes

The reporting on the latest WB Doing Business Report gives an excellent opportunity to illustrate the concept of 'statistical significance', and that sometime changes are no changes.

Interfax yesterday reported that doing business in Ukraine is getting easier as Ukraine moved up to the 145th position, while Ukrinform reported that Ukraine dropped 3 places.

And both are right - if you look at last year's doing business report, you will see that Ukraine was ranked 142nd. And in this year's report it is ranked 145th. Hence a drop of 3 places.

But in this year's report, Ukraine's rank of last year is reported as 147th. Hence, Ukraine improved 2 places.

As so often, one needs to read the small text - at the bottom of this year's table there is a footnote that reads 'Last year’s rankings, shown in italics, are adjusted: they are based on the same 9 topics and reflect data corrections.'


So Interfax and Ukrnews are both correct and one can claim both that doing business has become easier and more difficult in Ukraine!

Lies, Lies and Statistics, you might say. However, a more correct conclusion would be that one should use statistics correctly. Indeed, a statistician will typically hesitate to give an exact answer to a question - he would not say Ukraine's rank is 145th, but rather say something like. The best guess (in statistician's talk, the point estimate) for Ukraine is 145th, but Ukraine's true rank could be anywhere between, say, 140 and 150. That is they will give an interval, rather than one specific number, in statistician's talk, they will give an interval estimate. ( Note that I make up these interval numbers as the report does not have enough information to estimate this interval correctly). This interval makes explicit that typically we measure things with errors and that these errors cause us to observe an estimate of a true value rather than a true value itself. The data corrections the Doing Business report refers to illustrates very well that indeed, even the World Bank, measures things with errors.

If you think in terms of such intervals, it becomes clear that a change in the ranking of a country does not necessarily mean that the true rank of the country has changed. The estimate has changed yes, but not necessarily the true value which still can be between 140 and 150 (or something like that). We can see that correcting last year's errors moved Ukraine 5 places, from 142 to 147. So this year's change from 147 to 145 is smaller than last year's mistake! Hence, we shouldnt make much of a change of 2 places, if simple errors in the data alone can change your rank 5 places.

Conclusion - small changes should not be interpreted as reflecting change - given that the limits to our measurement skills are substantial, small changes are likely to be the results of changes in measurement error and hence should be ignored.

Of course, if journalists would apply this approach, there would be substantially less 'news'.