Data Needs Context

Analyzing data – finding means and medians and standard deviations and other related numbers – is important for making decisions. But often, the whole story of the data also includes comparing your results to the results of others. This could include comparing the results of a study on the efficacy of a new drug with the results of a study run by another group on the same or a similar drug. In the case of a survey study, it is often necessary to compare the results of one study to the results of another group doing the same survey. Here is an example that illustrates this point.

USPS Satisfaction Survey

Like many private and public companies, the United States Post Office (USPS) conducts employee-satisfaction surveys using the Gallup Q12 survey, a short survey that many other organizations use. As reported by the Save the Post Office organization, in a review of a report by the Postal Regulatory Commission (PRC) the “grand mean” (the overall average of scores on the 12 survey questions) was 3.16 on a scale of 1 to 5 (1 being the lowest score and 5 being the highest score). They equate this grand mean to a C (average) grade.

Is that good or bad? Sure, it seems like there is room for improvement, but perhaps 3.16 is good for a large organization. So it is important to know how the results compare to other companies. But the report to the PRC did not include the comparison of results to other companies that Gallup provides as part of its survey services. So Save the Post Office did its own research and found 55 other companies that both used the Gallup Q12 survey and reported the grand mean in its public documents. If the USPS’s 3.16 grand mean was added to the list of the grand means of these 55 companies, the USPS would be second to last. This adds some context to the numbers and seems to indicate there is a lot of room for improvement in employee satisfaction at the Post Office.

University Pass Rate Comparison

Another example of needing context for a better understanding of the results of a study comes from my own experience. I taught a developmental (introductory) mathematics course for a private university for many years. This was a course that I helped develop and implement, and I kept track of our pass rates over quite a few years. We generally had five to seven sections of this class running each semester, with class size around 18 students. Our pass rate for this class was in the neighborhood of 70%. Was that good or bad?

To give ourselves some context and determine whether we were doing okay or needed to do make adjustments to the course, we needed to compare our results to other schools’ results. After doing some research, I found that the pass rate for comparable classes at both two-year colleges and state universities was around 55%. We determined that we were doing rather well and, while we were always trying to improve our pass rate, we were satisfied that a fair number of our students were successful. Another comparison that we could and should have done was with similar classes at other private universities or schools with similar class sizes. Such data are not easy to come by and, at the time, it was not worth the effort to get such data.

Conclusion

So the bottom line of this is that your results do not usually live in a vacuum. To give context and meaning to your results, you usually need to compare them to other results. Determining to whom or what you are going to compare your results could influence the direction you go with your research, so think about these as you develop your own research studies. If you are interested in learning more about research, you may want to consider pursuing a degree in data science.