Edition 16
Information keeps moving between inboxes. The question is whether any understanding moves with it.
Last year, I released the TRUST-IN/Nirmata (trustinbharat.org) cybersecurity framework around this time. I asked people for reviews, and a fellow practitioner sent one 2 days later. At the risk of boasting, this framework is a maturity model with 12 security pillars, 5 levels, and the self-assessment itself is about 191 questions. It is not conceivable to me that a person can write a meaningful review in 48 hours. It took me months to put it together, drawing on 34 years of industry experience, my CISSP knowledge, and my study of ISO 27001, 42001, GDPR, and DPDPA. So I asked him the obvious question. Did you read it, or ask ChatGPT to review it? He admitted that he used ChatGPT. I told him never to send me a review if he has not created it himself after studying the framework. I also told him that I felt humiliated and belittled by what he did after my painfully hard work.
If someone asks for a summary of the Foundation series by Asimov, or the Three-Body Problem by Liu Cixin today, the AI will do a good summary. The nuances, the little details, the ideas, the humour, the human sensitivities. All of these are lost. And the whole experience of living through the books is completely lost. Vicariousness is an essential part of our lives. When I used to read Robert Ludlum, Alistair Maclean, Sidney Sheldon, and other fiction writers (also Asimov, Sagan, Herbert, etc.) back in high school, I learned about much of Europe and the USA. This gave me a picture of the Alps, Germany, the UK, Italy, Spain, the USA, the Moon, Mars and the universe. I had never set foot outside India at that time. If I had read a summary of those books, I would have never experienced anything vicariously. The whole fun of reading would have been taken out of it. Similarly, even though I have never touched a mainframe in college, reading books about mainframes and the early computing era filled me with awe and a basic understanding of how the technology evolved. Later, when I used a personal computer, I could relate to what I had read and therefore understood. Similarly, when I visited the countries I had read about, I could connect to many things. Places, people, culture and so on. Reading is not just putting letters together and understanding the word. It is a lot more than that.
The writer goes through a lot of revisions and pains to make sure that this intended effect is achieved by the reader. Just putting words together does not mean a thing, nor can it ever be useful. Writing requires thinking and bringing out the emotions in the reader. Similarly, making movies, music, or art is the same. It needs to connect with the users. When you delegate to AI, it works by providing the next probabilistic word for every word, based on its training. It is without emotion or thought. That is the crux of AI slop and why it is instantly recognisable. When AI reads, it is using those patterns to make sense of the prose. The writer takes pains to make sure that the emotion is carried over. AI does not have emotions to understand this. And the summary will likewise be emotionless. Summarising the writing with AI is a disservice to the writer.
Information Has Moved. Has Knowledge?
This is where I think we are making a mistake in how we use AI. Someone does not want to write but yet wants to create a lot of followers on social media, so AI writes for them. We used to have human ghostwriters in the past. They have now lost the battle with AI, for those who do not value the human element that goes into writing. The document reaches another person, who does not want to read the whole thing, so AI summarises it for them. The writer may have never spent enough time to understand what he is writing and what the argument he has to make is about. The reader might not spend enough time to understand the document, or the argument it makes. Somehow, the document has been created, circulated, summarised and perhaps acted upon. This is what is going on with communication in the last couple of years. I am not sure if knowledge was transferred between us. Communication is defined as passing on your ideas to another person. In this case, the very definition of communication is defeated.
Over time, since industrialisation, we have been moving towards becoming a metric-driven society. By those standards, everything was good in the case above. A report was created, distributed, circulated, summarised, and presented. A hundred-page document was converted into five or ten bullet points. It might seem efficient to you now. Sometimes it could be genuine. But honestly, information moving from one email inbox to another is not really communication. Knowledge is lost in the ether space.
An AI summary can tell me that a document has twelve recommendations and that three of them are critical. That may save me a lot of time, but sadly I am not sure whether those twelve recommendations are relevant or whether the three are critical to me. And, if the document contains the reasoning behind those recommendations, the circumstances under which they apply, the exceptions, the doubts and the evidence, I have not understood any of that by reading three bullets. I have read the words, but not understood the impact of what I am reading. It is the same as the difference between hearing and listening.
Writing Was Never Just Typing
As many of you are aware, I have been writing for a while now. What you see is rarely the first draft of what I wrote. I have an idea, and I start writing about it. Midway through writing, I realise that I need to research a particular subject a little more. And then I set off on a tangent of learning and questioning my writing. I eventually come back and finish the draft. I let it rest for a day and then read it myself. I tell myself, more often than not, that I can do better. I then rewrite the parts I am not happy about. I then look at it from a totally newbie perspective and ask if I have assumed anything that I should not have. And this leads to another draft.
Along the way, I delete sentences, paragraphs, and sections, then add new paragraphs, ideas, and sentences. I remove items that just don’t work well with the rest of the article. And sometimes I discover that a specific sentence is a new idea in itself. So off it goes, and into a second document to grow its legs and become an article of its own.
All of that is part of my writing process. I am sure others have their own process. Sometimes, the thinking does not always happen before the writing starts. A lot of it happens while writing. This is why I have a problem when somebody says AI can save all that effort. The obvious question is: which effort really is saved when AI is writing?
There are some use-cases where AI can genuinely be useful. The language in which you write may not be your first language, in which case a little help with grammar or synonyms would be good. Honestly, you can use other tools for this too. You can use it to organise the notes. And there are tools for this as well. One useful assistance it can provide today is to read your document and find gaps or flaws in reasoning. But if AI does the part where I have to decide what I actually think, then it is no longer helping me write. It is doing some of the thinking for me. And I have a huge problem with that.
While that may still produce an excellent document, it does not mean I understand what was written. The bigger implication is that when someone challenges what was written, I have no real answer to it because I did not think through the writing. Weighing what was written and the probable arguments helps you prepare when you are asked to substantiate.
The finished document can look extremely impressive. And by now, many of us have learnt to recognise AI writing. Therefore, the writer’s ownership tends towards zero. Mind you, I am not talking about readability, grammar, spelling or productivity here.
We Are Also Becoming Lazy Readers
It would be easy to blame writers for delegating too much. Readers are doing exactly the same thing, and I am sometimes guilty of this too. A 200-page report lands in front of you, and you are tempted to ask your favourite AI tool to help summarise. Sometimes that is exactly what you should do. There is no point reading two hundred pages when I need one paragraph from page 146, especially when you don’t know where exactly that paragraph is. There is no point struggling through a badly written report if I only need to know whether it contains something relevant to what I am working on.
AI could be very useful for this. You can ask the AI tool to identify which pages have information specific to you and the related pages. You then read those pages and understand them. If that raises additional questions, go back to the AI.
The problem is when we do it with everything.
Give me the five key points.
Tell me the conclusion.
Tell me what matters.
Tell me what I can ignore.
Does the author agree with what I already think, or, in other words, is there a confirmation bias here?
At that point, I have stopped using AI to help me read. I am now asking it to decide what I should have read. And I view my future reading of the report through a coloured lens.
The material is supposed to make us think. A research paper is not only its abstract and conclusion. The devil is in the details. A strategy document is not only its recommendations. It could be flawed in the execution plan. A novel is certainly not only its plot. The nuances and details are the most important bits. The parts between the beginning and the end are often where the value sits.
Reading takes time, and sometimes that is the objective of reading. A machine can tell us where the author started and where he arrived. When we read the AI summary, it means we didn’t travel with the author vicariously and experience the journey. We miss out on all the little details. We get to the destination, but miss the excitement of the journey altogether.
What Happens Inside Companies?
Now think about how this can work inside an organisation. A senior leader asks for a strategy paper. Someone in the team uses AI to generate most of it. The paper goes back to the senior leader, who uploads it to another AI tool and asks for a summary. The five points are copied into a presentation. They are discussed in a meeting. An AI note-taker records the meeting and creates another summary. That summary becomes the record.
This is a very plausible scenario today. I see a lot of AI note-takers trying to join my meetings, and I kick them out. Every stage in the paragraph above produces an artefact. And yes, there is an audit trail for the entire workflow. What is missing is evidence that knowledge is indeed exchanged. And that is not something an audit trail will prove.
The same thing can happen with risk assessments, board papers, audit responses, policies, research reports, proposals, investment notes and technical documents. We may reach a point where companies have more documentation than ever before and less understanding of what their own documents actually mean.
There is one question I ask when people ask me what to use AI for: which part of this work actually requires somebody to understand it?
And my advice for those people is simple. If nothing requires them to understand the mechanics or the reasoning behind it, automate it. In all other cases, refrain from automating it. Approving work done by AI is more than likely not the same as understanding it. Signing is not understanding.
Then the Machines Start Reading Each Other
There is another problem that follows from this. AI writes an article. Another AI summarises it. Someone publishes the summary. A search engine indexes it. A different AI retrieves it six months later and produces another explanation from it. Someone else asks for a summary of that explanation. The information keeps circulating while the distance from whoever originally knew something becomes larger.
Take a simple example. An engineer encounters a very unusual problem in production and spends two days figuring it out. They write a detailed explanation. That is original knowledge. An AI rewrites the article to make it cleaner. Another website summarises it. A company uses the summary to create internal documentation. Months later, an internal assistant answers another engineer’s question using that documentation.
Everything may still be correct, but imagine the original engineer mentioned one small condition that caused the failure. It looked unimportant during the rewrite. It disappeared in the summary. The internal documentation never contained it. The next engineer encounters exactly that condition, and the AI cannot retrieve something that stopped existing three transformations earlier.
The prose improved. The knowledge became thinner.
This is the part that interests me more than the usual discussion about whether AI-generated content is good or bad. We should also ask how far the material has travelled from the person who actually knew something and what may have been lost along the way.
The Person Who Was There
The more synthetic material we create, the more valuable the original source may become. The engineer who actually saw the failure, the doctor who actually saw the patient, the scientist who actually ran the experiment, the journalist who actually spoke to the people involved, the reviewer who actually used the product and the traveller who actually went to the place all have something that a later summary does not.
They have experience.
AI can create endless explanations of what happened. It cannot create another instance of the event having happened. This matters because repetition can create a false sense of confidence. If fifty websites repeat the same statement, it can look like fifty sources even if all fifty copied one source. If twenty AI-generated articles describe the same thing slightly differently, it can look like broad agreement even if every one of them comes from the same original paragraph.
Volume is not evidence. We knew that before AI. AI simply makes the volume much easier to manufacture.
That means provenance will matter more. Where did this come from? Who originally knew it? Was it observed, inferred or generated? How many times has it been rewritten? Those questions sound tedious until the answer actually matters.
There Is a Technical Version of This Too
Researchers have already looked at what happens when AI models repeatedly learn from material generated by other models. The phrase commonly used is model collapse. If one AI model produces synthetic data, another AI model learns from that output, and the process continues without enough original data being retained, parts of the original distribution can gradually disappear. Rare examples are particularly vulnerable. A lot can be lost in translation, as such.
The model does not necessarily become useless overnight. It can simply become narrower. That interests me because it resembles what happens when humans repeatedly summarise something. The first summary loses a little detail. The next loses a little more. Eventually we are left with the things that everybody at each stage thought were important. Anything unusual is at risk of disappearing.
The technical problem can be worked on. Researchers are already looking at ways of mixing synthetic and real data, tracking provenance and filtering generated material more carefully. The human version is harder to fix because it involves behaviour. If we stop reading the original, the original might as well not exist for us.
The Closing Signal
I am not arguing that we should stop using AI to write or summarise. That would be silly. I use AI where it is useful. It can help me find mistakes, question an argument, point out something I have missed, search a long document, explain something differently or remove hours of mechanical work. But it is not going to be able to write an article or articulate my arguments well.
But I would never ask AI to read a novel or a paper for me. I do not want somebody to send me a review of my work that they never actually reviewed. And I do not want to reach a point where two machines exchange information while the humans on either side merely approve the outputs.
Sometimes the effort is wasted and should be removed. Sometimes the effort is the learning. We need to know which is which. Writing is thinking, and more often than not learning as well. Reading is learning and understanding. You can read fiction set in a different country without having ever travelled there. A technical document can give the student an understanding of a hypothetical machine, or a machine the student has never seen or touched. An articulate argument can make somebody reconsider something they thought they were certain about. All of those things take time. That does not make the time inefficient. Time is NOT WASTED.
We now have software that can write things we do not want to write and summarise things we do not want to read. But if the writer has not thought about what was written, and the reader has not thought about what was read, I ask you:
The Closing Signal
Who has gained the knowledge?