Many, if not most statistical methods were developed for relatively small datasets.
Big Data means we need to reevaluate how we interpret results. A good examples comes from “the Facebook experiment”
Emotional contagion through social networks Adam D. I. Kramer, Jamie E. Guillory, Jeffrey T. Hancock Proceedings of the National Academy of Sciences Jun 2014, 111 (24) 8788-8790; DOI: 10.1073/pnas.1320040111
https://www.pnas.org/content/111/24/8788
Where in experimental conditions with N = 689,003, the researchers found an effect between the emotional content of posts on a user’s wall, and the sentiments of user’s own posts. It was a study that hit the major news-outlets, creating outrage, both for the ethical aspects of the study, but also for the importance of it.
What was not discussed, was the effect size. The effect size has a Cohen’s d range of between 0.001 to 0.02, which at the very best can be called “very small”, and at the lower end, insignificant.
The graph shows the two distributions, with Cohen’s d of 0.02, where the overlap between the two is in blue. For the lower range, the overlap is greater, (meaning the orange and green parts are even smaller).
A plain language explaining of a difference of 0.02 is given by the website: https://rpsychologist.com/d3/cohend/
With a Cohen’s d of 0, 50.8% of the “treatment” group will be above the mean of the “control” group (Cohen’s U3), 99.2% of the two groups will overlap, and there is a 50.6% chance that a person picked at random from the treatment group will have a higher score than a person picked at random from the control group (probability of superiority). Moreover, in order to have one more favorable outcome in the treatment group compared to the control group, we need to treat 177.1 people on average. This means that if there are 100 people in each group, and we assume that 20 people have favorable outcomes in the control group, then 20 + 0.6 people in the treatment group will have favorable outcomes.
Is this statistically difference of practical interest? I would say no.
For more critique of the study and big data, please also see:
https://www.tandfonline.com/doi/full/10.1080/1369118X.2015.1093525
Galen Panger (2016) Reassessing the Facebook experiment: critical thinking about the validity of Big Data research, Information, Communication & Society, 19:8, 1108-1126, DOI: 10.1080/1369118X.2015.1093525