Sociologists discuss sociological methods, physicists discuss physics

The title of this piece is a quote attributed to the great french physicist and philosopher of science, Henri Poincare. To further learn about his perspective on the discipline here is another quote of his :

Nearly every sociological thesis proposes a new method, which, however, its author is very careful not to apply, so that sociology is the science with the greatest number of methods and the least results.

Poincare lived from 1854 to 1912, but in my opinion his quote is still true today. We have been studying social sciences since Ibn Khaldun, or perhaps even before that, yet we have failed to replicate the successes of the natural sciences in these fields. In fact, there is no real consensus as to how the social sciences should be approached methodologically.

What is more, it seems like laypeople know this intuitively and demarcate between “soft” and “hard” sciences. But what is the reason for this cleavage ? What are the inherent differences between the two domains of inquiry that can explain their differences in success ? I will proceed to lay in writing my intuitions about these questions.

The butterfly effect

Everyone has heard about the butterfly effect, that “a butterfly batting its wings in India can cause a tornado in Texas”.

This metaphor illustrates the idea than in certain systems, called dynamical systems, minuscule changes in conditions can lead to massively different outcomes.

Human society is one of those systems. And this is the simple, yet fundamental reason, that predictions about the future state of society, especially in the long term, are impossible.

Black swan events, like the 2008 financial crash or the 2011 Fukushima disaster, are a manifestation of the butterfly effect. They are the tornado that we couldn’t predict because we did not pay attention to the butterfly.

“But what if we did pay attention to the butterfly ?” you might ask. We can’t. Everything in our modern world is interconnected and there would be way more information to take into account to perfectly predict such complex systems than even the world’s most powerful supercomputers could handle.

But it is not only that some historical events are fundamentally unpredictable, it is that history itself is defined by a succession of these black swan events.

In the book The poverty of historicism, Karl Popper makes the following argument :

  1. Technological innovation can completely transform society.
  2. If we knew with precision what technologies we would have in the future we would start building them in the present.
  3. From 2. follows that we cannot know exactly what technologies we will have in the future.
  4. From 1. and 3. follows that we cannot know the future state of society.

This argument is, in a sense, a black swan argument. Small technological innovations can transform the world so much that it becomes unrecognizable. They are rare events with potentially humongous consequences, the definition of a black swan.

And although this is true of the technological innovation, these are not the only type of events that can radically change the face of human society. The extinction of a species of fish, changes in climate, the assassination of a a world leader, these are all events that could make the world of our grandchildren seem foreign to us.

I have to concede that it is not only the social sciences that study complex and dynamical systems, the earth’s climate is one and is being studied by “hard” science professionals. But there is an additional property of social systems that makes them even more difficult to study, that we will proceed to look into.

The Oedipus effect

In his short book The poverty of Historicism Karl Popper gives a name to predictions that influence the system they are trying to predict, the Oedipus effect.

Imagine you are an eminent political scientist and you predict there is going to be a war. World leaders react to your prediction and take the necessary measures to prevent the war. That means your prediction turned out wrong, but what would have happened if you said nothing ? It is impossible to know since we don’t have access to counterfactual worlds.

This phenomenon is ubiquitous is social life and I don’t think we have a real answer as to how to deal with it. An other example could be found in econometrics.

Economist Robert Lucas voiced a critique of the field of macro-econometrics, which lies on the Oedipus effect, in his 1976 paper. He summarizes it as follows :

Given that the structure of an econometric model consists of optimal decision rules of economic agents, and that optimal decision rules vary systematically with changes in the structure of series relevant to the decision maker, it follows that any change in policy will systematically alter the structure of econometric models.

Essentially, you predict an economic problem X is going to happen. You take action. X doesn’t happen. Was your prediction good or bad ?

Systems, like society, which react to the predictions we make about them are called second order chaotic systems or level two chaotic systems.

Large social systems tend to be second order chaotic systems. This is in contrast with a lot of studies phenomena in the natural sciences, your prediction about the trajectory of a ball you throw will not influence it.

The fact that many phenomena in the natural sciences are not affected by the Oedipus effect makes them predictable and possible to engineer. Nonetheless, hard sciences do experience something similar to the Oedipus effect.

Many sub-fields of physics such as electronics, thermodynamics or quantum mechanics experience what are called observer effects. This means the very fact that you are observing a phenomenon can disturb it. The observer effect is slightly different than the Oedipus effect in the strict sense, insofar as it is not the predictions that you make that influence the phenomenon but the fact that you are observing it. Although observer effects can be problematic, I do not believe they are as problematic as strict sense Oedipus effect.

The fundamental difference that I want to underline is that when we study society, we study a system that we are part of. The predictions you make about the system are also part of the system and influence it. The inability to have an outside view of such a complicated system makes it all the more difficult to apprehend and predict.

Can we replicate the success of the natural sciences in the social sciences ?

Perhaps the reason sociologists discuss sociological methods and not sociology itself is because most methods fail to make sociology understandable to us. We don’t know what methods to use to predict dynamical and second order chaotic systems.

There is a field of inquiry which specializes in studying not instances of complex systems, but complexity itself. Complexity theorists try to use the latest technologies and algorithms, such as deep learning algorithms, to try and look inside the black box of complexity.

My personal view is that, if we want to make important strides in the inquiry of social phenomena, we first have to advance the study of complexity itself.

Specialized research centers such as the Sante Fe institue or the New England Complex Systems Institute are trying to achieve just that. We can only hope that they will have important breakthrough to share with us in the future.

But to answer the initial question of whether I think social sciences can be as successful as the hard sciences, my answer is no.

One reason is, as I’ve said a previous article, although I have faith we will better understand complexity in the future, some systems are just to complex to understand. And that also holds true for many social systems.

The second reason is, even if we perfectly understood how society functions, social engineering would still be problematic and undesirable. Indeed, even if we understand what is true of the world, that tells us nothing about what we should do in the world. Epistemic knowledge does not translate into ethics.

Even if we reached that perfect understanding, we would probably still disagree on ethical matters, which would make molding a perfect society impossible still, since perfect means something different to each and every one of us.

I have the intuition that sciences are often judged by what they bring to engineering and technology. And in the case the the social sciences, I have made the argument that even if we understood them perfectly, social engineering would still fail.

A short introduction to the replication crisis and fraud in academia

This last year and a half a phenomenon in academia has caught my attention. A big chunk of the scientific papers published in reputable journals don’t replicate. In this article we will try to explain the reasons behind this crisis, its implications and what we might do about it.

What is replication ?

You could make a solid case for the view that the main goal of science is to find the laws of the world. Indeed, the scientific enterprise has, since its inception, expanded our understanding of our universe. A reason for that is the scientist’s ability to discover patterns or constants in our world. Metals expand when they are heated. That holds true whether you live in ancient Mesopotamia or modern Australia, and it will most likely be true in the future. This property is unaffected by either time or space, which, one could argue, is the definition of a law.

Even in the legal sense, laws should to be applied equally on the jurisdiction for which they are designed and they should be stable trough time. That property demarcates the Rule of Law from arbitrary trials. This gives the citizen a sense of legal security and predictability of the judge, if I do X then law Y will apply.

With regards to scientific laws, they have roughly the same purpose. They should be true universally and make the world more predictable and understandable to us. Since I know that metals expand when heated, I know that if I were to heat a piece of steel in two weeks it will expand. Since I know how the metal will behave in the future, I can use that knowledge to solve problems I might have.

“What does any of that have to to with the problems in academia ?” You might ask. Well, replicating a study means conducting it again, by using the same methods and gathering new data the same way the former study did, or even by re-analyzing the same data a second time.

If our research methods are valid and enable us to find properties of the world that are perennially true, then if I conduct the same study twice, I should get the same outcome twice.

Unfortunately, for a significant portion of the studies in scientific journals, even the most prestigious ones, results do not replicate. And that phenomenon affects almost all of the disciplines, with some being hit harder than others. That includes medicine, psychology, economics, sociology, criminology, neuroscience, artificial intelligence and many more.

Why are scientific journals full of false findings ?

First of all, I have to say that it’s normal for some studies to yield false findings, that is just part of science. Look at it this way, of all the possible hypotheses you can make about the world, only a tiny fraction of them will be true. There are way more molecules that don’t cure headaches than molecules that do. Imagine if you had to test them all to find a cure of headaches.

Let’s say you were to test 100 000 compounds among which only one could cure a headache, and your testing methodology returned a false positive only 1% of the time. Even with that relatively low false positive rate, after having tested every single compound, you should have about 1000 findings that say that their compound works even though it doesn’t.

Of course, when researching a certain subject, you don’t test every single hypothesis, you try to make theory-backed guesses as to what can work and then test these hypotheses that you find plausible (this is called inference to the best explanation, or abduction). Nonetheless, the asymmetry between true and false propositions still holds.

However it seems unlikely that this asymmetry is the sole reason for the epidemic of false results in scientific journals. Let’s take a look at some numbers.

FieldEstimated replicability rate
Economics3060 %
Machine Learning (Recommender systems)~ 40%
Psychology~ 40%
Medecine> 50%
Marketing*~ 40%
Political science*~ 50 %
Sociology*~ 40%
Oncology~ 11 %
Physics*~ 70%
Chemistry*~ 60%
Biology*~ 60%
Estimated replicability rate by discipline

Note that in rows marked with an asterix, the replicability rate has been estimated trough surveys of researchers, not actual replication attempts, whereas for the other fields the studies were actually conducted once again.

As you can see, it is not at all uncommon to find fields with a replicability rate of 50% or below. The problem is severe and it seems like it is worse in over-hyped disciplines such as machine learning or oncology.

Indeed, these findings are the result of perverse incentives created by the science publication system. In order to get their grants renewed, scientists have to publish papers in scientific papers, preferably prestigious ones, otherwise their careers might come to an end. This is called the publish or perish effect.

Moreover, scientific journals tend to only publish novel and positive findings. The reason being that, they are private companies than operate for profit. Five companies publish half of academic research, they lobby universities get them to publish exclusively in their journals and then make other scientists pay to access research. This is a perfect example of rent-seeking, capturing public goods for private gain. Notably, Elsevier is the most profitable company in the world with a 40% profit margin.

These two factors combined create the aforementioned incentives, which drive researchers to produce novel, positive findings at all costs, even if it entails partaking in questionable research practices or downright falsifying results.

Questionable research practices are widespread in academia. It is very hard to gauge the extent to which they are, since, almost by definition, the individuals who engage in them try to conceal them.

Nonetheless, we do have some numbers. In a survey of biomedical post-doc students, 27% of them said they were willing to select or omit data to improve their results in order to secure funding. Note that, as far as I know, that survey was not even anonymous ! What is more, an anonymous survey of psychology researchers found that the majority of researchers have engaged in questionable research practices.

We can add to this body of evidence this testimony by a young social psychology researcher who was outright fired from her degree for refusing to engage in p-hacking. She also reported that her fellow researchers would engage in p-hacking to further their left-wing political agenda.

Yet another testimony by an economics researcher makes several concerning accusations. She reports that senior economists silence opinions that diverge from theirs, take credit for work that is not theirs, discriminate against some minorities and more.

Richard Thaler, an eminent researcher know for his contributions to behavioral economics and ex-president of the American Economics association, reportedly tried to discredit valuable research because it contradicted his views. Among this research is a paper reporting that only 33% of economics research can be replicated without contacting the original authors, which I used in my table above to estimate the replicability rate in economics.

Machine learning, over-hyped as it is, suffers from an obsession on devising new state of the art algorithms that achieve high scores on benchmark datasets. This obsession is fueled by scientific journals, who reward solely these types of studies, while refusing to publish research aiming to solve practical, real world problems albeit while using simpler algorithms. There also seems to be a bias towards “mathiness”, with reviewers reportedly asking authors to add mathematical formulas to make their papers seem more “sciency” and marketable.

To top it off, we might observe that there is no correlation between a paper replicating and its number of citations. This could denote several issues that plague academia. It has been observed that researchers will sometimes refuse to cite colleagues with whom they compete for a grant and that they will form citation alliances sometimes referred to as citation rings.

What can we do about it ?

There are several initiatives that could be implemented in order to mitigate this situation.

First of all, science should be freely accessible, since it is funded with our tax money. This reform is necessary, but could be very difficult to implement due to scientific journals’ important lobbying power.

There are dozens open access journals, and many scientists choose to only publish in those. Nonetheless, early career scientists have a strong incentive to publish in renowned journals in order to advance their careers and possibly get tenure. In many universities, tenure is conditional on publishing in these outlets.

Secondly, more replication studies should be undertaken. Online repositories for replication studies are beginning to emerge in order to host this type of studies, which doesn’t get much love from the oligarchs of scientific publishing.

Other than that, scientist should submit their data and code along with their papers, not only to detect fraud but open data and code make replication a lot easier. As some say, “In God we trust, all others bring data”.

Finally, I am personally of the opinion that we should completely ditch peer review as there is scant evidence that it can even beat random screening. It is likely that in the future, statistical models will be devised to rate the quality of a paper and its probability of being replicated. In order to extract the necessary features for such a model, one could turn to natural language processing models.

These latter models are also being used in another way, Brian Uzzi, professor at Northwestern university, trained a NLP model to detect elements of language that indicate fraud or low confidence in the findings, rather than trying to use the measurements and metrics of the study.

Closing thoughts

Hopefully this piece will have fulfilled its purpose by giving an thorough yet brief introduction to some of the major problems facing academia currently. It is regrettable that such a noble pursuit has become so corrupt, discouraging many youths to pursue a career in academia, myself included.

Despite it all, I am still optimistic, insofar as academia does not have a monopoly on science, far from it. Private companies and institutes have been responsible for many scientific breakthroughs in the past two centuries. The most recent notable example would be Google’s quantum supremacy. The private sector is particularly proficient in the advancement of applied, practical science, in other words : technology development.

I encourage everyone interested in science but critical of academia to not get disheartened with science as a whole. If you consider yourself a humanist, perhaps solving people’s everyday problems trough knowledge is more important than theoretical progress. After all, don’t we pursue science to better our lot ?