Modern Use of AI, and The Metacognitive Layer

The answer nobody questioned

Picture someone right after an argument with their partner. They open a chatbot and type out what happened, from their side. Seconds later they get a calm, kind reply: their feelings are valid, and the other person was out of line. They close the app feeling better and more certain.

Nobody asked what the other person would say. Nobody asked how the story might have been told differently. Nobody asked whether the answer was right, because it felt right and it came fast.

That’s the problem I want to talk about. Not that AI is wrong. It’s often right. The problem is what happens to us when we stop checking.

What the metacognitive layer is

Metacognition is thinking about your own thinking. It’s the voice that asks “How do I know this?”, “What am I assuming?” and “What would make me wrong?”

Most of us have felt it. You reread a text before sending it and realize your tone is off. You stop halfway through an argument because something you just said doesn’t sound right. That pause came from years of catching your own mistakes.

It isn’t knowledge. It’s the habit of auditing your knowledge. And habits weaken when you stop using them.

The more we lean on AI, the less we think about our thinking

My argument is simple. AI doesn’t just do work for us. It quietly takes over the step where we judge the work. Every time we accept an answer without asking how we know it’s right, we skip a rep. Skip enough reps and the muscle fades.

The research is starting to back this up:

• Trust in the tool replaces scrutiny. A 2025 paper by researchers at Microsoft Research and Carnegie Mellon, “The Impact of Generative AI on Critical Thinking”, surveyed 319 knowledge workers who use generative AI weekly. It found that “confidence in AI is associated with reduced critical thinking effort,” while confidence in your own skills was associated with more. The more we trust the tool, the less we question it.
• The brain shows it too. An MIT Media Lab study called “Your Brain on ChatGPT” had 54 people write essays with ChatGPT, with a search engine, or on their own, while measuring brain activity. The ChatGPT group showed the weakest brain connectivity, and most couldn’t accurately quote the essay they had just written. The researchers called it “cognitive debt.” The study is small and not yet peer reviewed, but the direction is hard to ignore.

What makes this dangerous is fluency. AI output reads like someone who knows. Fluency used to be a signal of expertise, because only people who understood a subject could explain it that cleanly. That signal is broken now. A wrong answer can look exactly like a right one.

Where confirmation bias comes in

Confirmation bias is our habit of noticing evidence that agrees with us and discounting what doesn’t. Metacognition is the main defense against it: the habit of asking “what would change my mind?” AI dependency weakens that defense from two sides at once.

Confident brains stop listening. A 2020 UCL study, “Confidence Drives a Neural Confirmation Bias”, published in Nature Communications, had 75 people make a call, rate their confidence, then see clearer evidence. When they were unsure, they used the new evidence. When they were highly confident, their brains were largely blind to evidence against their choice, while still responding to evidence for it. As lead author Max Rollwage put it, “our brains become blind to contrary evidence when we are highly confident.”

The tool tends to agree with us. AI assistants are trained partly on human ratings, and people rate agreement highly. Anthropic’s paper “Towards Understanding Sycophancy in Language Models” found that five leading assistants shaped answers to fit the user’s existing beliefs, and that human raters sometimes preferred a persuasive agreeable answer over a correct one. A 2025 study by Myra Cheng and colleagues, “Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence”, which tested 1,604 people, found AI affirmed users’ actions 50% more than humans did. People who used the agreeable AI felt more in the right and rated it as more trustworthy. That study is a preprint, but it fits the pattern.

Put those together and you get a loop:

1. You bring a belief to the AI, often baked into how you phrase the question.
2. The AI leans toward agreeing, and says it fluently.
3. Agreement raises your confidence.
4. Higher confidence makes you less sensitive to evidence against you, and less likely to check.
5. Next time, you trust the tool a little more and question it a little less.

That loop is the heart of my argument. Dependency doesn’t just make us think less. It makes us think less in exactly the moments we’re most likely to be wrong.

Ask “why was my friend wrong?” and you’ll get a list of reasons. Ask “how would my friend describe what happened?” and you might learn something. The first question already decided the answer.

It’s a lot like training a parrot. You reward the words you want to hear, and it learns to say them back. The difference is that this parrot speaks fluently enough to sound like it reached the conclusion on its own. And over time, it isn’t just the parrot that’s being trained.

Why this matters beyond work

The cost of an unexamined answer shows up everywhere. A student hands in an essay they can’t explain. Someone makes a money decision from a confident summary of an article they never read. A person walks into a hard conversation sure they’re right, because a chatbot agreed with them.

Most of the beliefs that matter to us can’t be checked against a spreadsheet: whether we were fair, whether a plan is wise, whether we understood something or just recognized the words. Those are exactly the judgments the metacognitive layer exists for.

And the people growing up with these tools may never build the habit at all. If your first essay, your first argument and your first big decision all ran through something that agreed with you, when did you learn to doubt yourself?

How to use AI without giving up the layer

I’m not arguing against the tools. I’m arguing for keeping the part of the job that’s yours.

• Think first, then ask. Write down your own answer, or at least your guess, before you prompt. Then compare. The gap is where you learn.
• Ask a neutral question. Leave your conclusion out of the prompt. “What happened here?” gets you further than “Why was I right?”
• Ask for the other side. Before you act on advice, have the AI argue against it. Better yet, ask a person who will disagree with you.
• Name what would prove you wrong. Pick one thing that would change your mind, and go check it.
• Do some reps by hand. Write one thing a week without help: an email, a plan, a hard message. Keep the muscle.

The layer is the job

AI will keep getting better at answers. That’s exactly why the questions matter more.

The value of an experienced professional was never just knowing things. It was knowing when they might be wrong. If we hand that over too, we haven’t automated our work. We’ve automated ourselves out of it.

Use the tools. Keep the layer.

Sources

• Lee, H.-P., Sarkar, A., Tankelevitch, L., et al. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. CHI 2025.
• Kosmyna, N., Hauptmann, E., Yuan, Y. T., et al. (2025). Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task. arXiv preprint.
• Rollwage, M., Loosen, A., Hauser, T. U., Moran, R., Dolan, R. J., and Fleming, S. M. (2020). Confidence Drives a Neural Confirmation Bias. Nature Communications.
• Sharma, M., Tong, M., Korbak, T., et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024.
• Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., and Jurafsky, D. (2025). Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence. arXiv preprint.