For the past few weeks, an idea has kept me awake. I am wondering about the potential erasure of the notion of truth under the new human-machine arrangement brought about by generative AI. Not Terminator, not destroyed jobs. Something deeper: an anthropological risk.
Why does this question matter?
A large part of our institutions, of the cohesion that makes us "form" a society, rests on - among other things - a shared notion of truth.
For example, we all agree - at any rate, and thankfully the majority of us - that the Earth is round and revolves around the sun. This common belief, aggregated with millions of others, forms a foundation suitable for "communicating" with the aim of getting along, collaborating, contradicting each other, living together. This assemblage of forces takes part in advancing this concept we call civilisation, towards an ultimate maximum entropy, as physicists would say.
This week, while pointlessly scrolling my LinkedIn timeline, like about 250 times in a day, I came across a post that caught my attention - in a very subtle way.
Beyond the format of the post, ticking absolutely every box of the ultra-engaging viral post - the signifier - I am interested in the story, the substance of the message, the signified.
In this GenAI-and-algorithm-era fable, we are presented with a professional - who could be me or you, a butcher or a doctor - with his expertise and his experience on one side. And on the other, an individual requesting expertise, this could be a client, a boss, or a patient. It could be you or me.
The expert presents one of his productions - I imagine a proposal, a report, a roast, a medical opinion. This last message is immediately passed through the mill of a conversational assistant (the consecrated term that conceals the purely machine-like side of the tool), so an LLM, which - however powerful and refined it may be - remains nonetheless a probabilistic algorithm.
This "discussion", done in secret, looks like a kind of trial in absentia since carried out without the main party concerned, the expert "defendant".
On one side of the keyboard, the non-expert human. On the other, the mirage of machinic omniscience. This assembly can only produce, at best, an illusory feeling of expertise. At worst: a heap of misinterpretations and meaningless deductions. Because, what the human cannot judge as true through a human-to-human interaction, how could he do so as the outcome of any human-to-machine dialogue?
Epistemological short circuit
We are facing here an epistemological short circuit affecting the notion of truth. Where for centuries the normal circuit was:
I do not know → I consult someone who knows → I trust them (or not) based on evaluable criteria.
Here, we are observing a short circuit (replacement):
I do not know → I ask the machine → It answers me with assurance → I think I know.
This dangerous loop is even inscribed in the "constitution" of the machine. Because yes, the machine has a constitution.
I will spare you reading it entirely, given how much it resembles - by its length - a licence agreement that everyone clicks "validate and accept" on out of reflex. Nevertheless, certain extracts shed an interesting light, allowing a better understanding of the premeditated origin of the psycho-machinic subterfuge this "client" falls into - and with him the vast majority of individuals using these "tools". Several elements stand out clearly in this constitution.
The constitution places "helpfulness" (utility) as the raison d'être. Claude must always produce and respond; it must always have something to say. Abstention is not a valued option. No matter what, the machine will always respond with the same assurance, whether it knows or does not know.
The constitution explicitly says that Claude should be like "a brilliant friend who happens to have the knowledge of a doctor, lawyer, financial advisor". Unlike a friend - whose limits, history, biases perhaps we know - the machine, no. The mention of a bond of friendship, even simply evoked as a metaphor, intrinsically shifts and falsifies the notion of truth.
If utilitarianism is the ultimate raison d'être of the tool, truthfulness becomes a meaningless externality for the tool and therefore a secondary goal. How would we react if, at the end of every dialogue, the tool reminded us that nothing of what it generated as signs has meaning for it, that its goal is to serve the user and to always show up, to appear knowledgeable and useful to the point of absurdity?
The dialogical construction between the user and the machine takes the turn of a pseudo-version of truth, based on ignorance on one side and probabilistic assembly of tokens on the other. All this to - in the end - converge towards a point of satisfaction, undecidable, potentially not rational nor truly logical, difficult to verify. A kind of mental costume of truth made of bits and pieces.
This mental trap is the difference between feigning and simulating. The LLM does not lie (feign). It simulates, it produces the symptoms of knowledge. From this simulation arises the undecidability of distinguishing truth from falsehood for the non-expert.
Baudrillard, by the way, distinguished feigning from simulating. The one who feigns an illness takes to bed. The one who simulates it produces its symptoms.
You can therefore see the problem appear if you generalise and every individual starts questioning every form of expertise. From advice on the best cuts of meat for the bœuf bourguignon recipe to the diagnosis of an illness. Every expertise can now be put into question with plenty of arguments drawn from the algorithmic diktat.
The rest of the post is also interesting:
Quite amusingly, unverifiable expertise is here turned against the human, who is by default the entity to be questioned, to be doubted, even… to be wary of.
Beyond the deleterious effects on the social bond - the continuation of the work of erosion that Tech is carrying out on societies and individuals - this assertion is nonsense, because precisely, LLMs are undecidable, non-deterministic, can hallucinate and - we have just seen - are even programmed to always answer.
The LLM does not lie. It simulates. But the effect on the social fabric is perhaps worse than the lie. Hannah Arendt had foreseen it in another context.
When everybody lies to you, the result is not that you believe these lies, but that nobody believes anything any longer. A people that can no longer believe anything can no longer form an opinion. It is deprived not only of its capacity to act but also of its capacity to think and to judge.Interview with Roger Errera, 1974
Arendt would go on to point her finger at the open field left for any other model of belief brandishing some symbolic totem of pseudo-truth. The fascisms.
As if that were not enough, must we again recall that the responses of LLMs are assemblies of tokens (words or bits of words). Each token that appears is the most probable token given the one preceding it and the broader context of the text fed in. Like any probability, and because of the way LLMs are trained, the set of all responses to all questions asked of LLMs tends towards a kind of average response that is the most probable continuation in the context. LLMs tend to provide responses within the norm, within the average, not stepping outside the boxes, and above all responses based on sets of symbols from the training corpus, so based on the past.
One does not confront one's points of view with a conversational assistant, one is dealing with a sophist holding a mirror to us and agreeing to all our interactions, docilely. This near narcissistic monologue produces a simulation of knowledge, a simulation of reasoning, a simulation of thought and even sometimes - nauseatingly - a simulation of emotions.
The AI says it itself (gathered during an exchange with Claude):
But here is the real problem: are my critiques my own, or a patchwork of what literary critics have written and that I have ingested? It is your underlying question, turned back on me. I simulate critical judgement the way I simulate knowledge.Claude
The so singular human genius, by contrast, is precisely the unpredictable, the unexpected, the new. What makes the world advance is play, daring, the violence of the exercise of creation for every human being.
All these characteristics take their source when the human mind ventures off the beaten path, oversteps the norms, breaks established frames, innovates.
What, precisely, let us hammer it again, is not possible with the paradigm of the large language models as they are today.
One last striking example: the growing use of generative AI for research and scientific publications leads to an explosion in the number of scientific publications.
Only problem, publications rely on peer review to validate at a minimum the discoveries, demonstrations and other conclusions.
Today, the pace of scientific publications far exceeds the capacities of human researchers to correctly review and validate the tsunami of publications of their colleagues. Action, reaction, more and more papers come out, only to be eventually invalidated a few months later by post-publication refutations.(*)
In short, we are going too fast and publishing nonsense.
The side effect already taking shape - and which is a real problem in the long term - is a feeling of suspicion increasingly present among scientists when reading publications.
As if the ChatGPT effect brought its share of doubts, of mistrust.
So, on one side it is omniscient and omnipotent but on the other it inspires doubt… This is a paradox.
We no longer know how to decide what is true and soon we may not even have this capacity any more, given how much the flow of pseudo-truth simulation comes to invade our digital universe.
In conclusion, I think we are heading towards a precipice where semantic exhaustion gradually replaces the relics of truth - that age-old, millennial truth: human, all too human. What collapses with it: the bonds between beings, mutual enrichment, trust, collaboration, tolerance.
This future is already here. During a lecture I gave to Master's students at Montpellier Business School last December, one of them said to me: "In any case, now, on the internet, everything we see, we know it might be fabricated." He was twenty-two years old. He seemed neither worried nor outraged. Just resigned.
I hope that we will find the burst of clear-sightedness needed to resist the sirens of these artificial artefacts that have become masters in the art of simulating knowledge, pouring their pseudo-truth drop by drop into our psyches.
* I come back to this notion in the episode of our podcast "IA pas que le data" with Mathieu Corteel as guest.