Z-Estimation System

2022-06-08T00:00:00+00:00

Z-Estimation System was originally published by Kate Hu at Kate Hu on June 08, 2022.

Precision Agriculture

2022-06-08T00:00:00+00:00

Precision Agriculture was originally published by Kate Hu at Kate Hu on June 08, 2022.

Interaction Effect

2022-06-08T00:00:00+00:00

Interaction Effect was originally published by Kate Hu at Kate Hu on June 08, 2022.

Data Collection

2022-06-08T00:00:00+00:00

Data Collection was originally published by Kate Hu at Kate Hu on June 08, 2022.

Bias Correction

2022-06-08T00:00:00+00:00

Bias Correction was originally published by Kate Hu at Kate Hu on June 08, 2022.

What motivates statistics?

2015-10-31T00:00:00+00:00

I finished translating the book Objectivity with my friend Meng, which led me to another exciting book Trust in Numbers by Theodore Porter. Both are a history of science books. I got interested in this work because I think the pursuit of objectivity motivates the field of statistics, which often deals with how to remove human bias through developing scientific methods to collect, analyze, represent, and interpret data.

If you believe my statement that the pursuit of objectivity motivates the field of statistics, then let’s continue to consider the next level question: in turn, what motivates objectivity? A question ultimately tells us what drives the development of statistics. These two books attempt to address this question, and interestingly they hold opposite views. Objectivity believes the evolvement of the scientific self motivates objectivity. In contrast, Trust in Numbers argues that the expanded trade from local to larger geographic areas and the change in how societies operate demand objectivity.

If you asked me who I agreed with seven years ago, I would undoubtedly answer “the arguments in Objectivity “. However, after working in the climate science industry and collaborating closely with scientists and other professionals for six years, I would change my answer to the same question to the opposite. The theory stated in Objectivity does not match my experiences in the industry. I encountered more frictions when selling statistics to scientists than engineers, product developers, and businessmen. Statistical methods developed so far seem to help the latter groups make better decisions than the first group— scientists. I have also observed statistical methods often frustrate scientists in developing their scientific stories. For example, we hear the scientists often complain their hands are tied by the significance test developed by statisticians. From a series of successes and failures in promoting the applications of statistics to various professionals over the years, I have started to doubt objective methods and statistics are probably NOT motivated by the evolvement of the scientific self among scientists. If we look at which subfield of statistics has flourished in the past few decades, it is medical statistics. Yes, medicine is a field of science, but the stakeholders are doctors and pharmaceutical companies who mainly rely on statistical evidence for making decisions. Very few papers in medical statistics integrate scientific knowledge in medicine but evaluate the effectiveness of a drug without using any knowledge of biochemistry behind how a drug works. Statistics seem to serve the pharmaceutical industry more than the sciences of medicine, answering the question of why?.

Scientists usually judge based on a plethora of evidence in all formats, including physical and chemical models, anecdotes, observations, theories, data from small experiments, etc. Such judgment will gradually form a new scientific theory, which is not expected to be permanent. As Poincare pointed out scientific theories change over time. The science community is tolerant of the theory being wrong and will change the theory once a new piece of evidence comes in. Their progress mode does not match the statistical methods of using data solely to “reject” or “not reject” a hypothesis, not to mention a lot of data preferred by statisticians. Scientists can not afford to collect a large amount of data. If I were a scientist, I would be annoyed by statisticians who attacked my argument by wielding a weapon of a significance test. “Your data is not large enough to support your hypothesis!” a statistician says. “Yes, you are right. Your theory is right, but it is not practical! I can not collect that much data, and I have other forms of evidence to support my belief.” I would argue back if I wore the hat of being a scientist.

On the other hand, industry problems often are more straightforward than a scientific question, asking for a decision rather than a causal story. They can collect a massive dataset with the management tools to standardize the data collection. The statistical tools generated so far fit their mode of operation well.

This discovery is bittersweet to me because I was drawn to statistics with a wish to advance sciences with it, but now I find the way statisticians think or how they inherit to think that does not match the needs of scientists. But, on the other hand, maybe knowing this is the beginning of finding the right statistical tools for sciences.

What motivates statistics? was originally published by Kate Hu at Kate Hu on October 31, 2015.

Let's see

2015-10-31T00:00:00+00:00

Statistics is common sense and simple solutions often work. The big data phenomena will drive us farther and farther from being simple. Here is a poem that I believe tells what may happen in the next fifty years in statistics. Let’s see.

All that is gold does not glitter,
Not all those who wander are lost;
The old that is strong does not wither,
Deep roots are not reached by the frost.
From the ashes a fire shall be woken,
A light from the shadows shall spring;
Renewed shall be blade that was broken,
The crownless again shall be king.”

–J. R. R. Tolkien

Let's see was originally published by Kate Hu at Kate Hu on October 31, 2015.