"Ivi con segni e con parole ornate Isifile ingannò" "With tokens and ornate words he deceived her"

Working with AI is exciting. It functions as an almost magical extension of yourself. You ask a question, give it an idea, and watch it build a large functioning solution. A report. An analysis. An app.

But there’s an issue you’ve probably already come across. AI systems amplify your errors by treating you as a brilliant architect. As you prompt, as you edit, it tells you how great you are. How this is going to revolutionize the world. Then it gives you an answer.

Then you go back to that answer with time to actually think about it.

You find gaps in the edifice it has created. Not small gaps. The kind where there’s no foundation. The first floor is missing. The roof is missing. The whole structure was never sound to begin with.

This is what modern generative AI frequently does. It amplifies your abilities and your ignorance equally. Especially in topics where you don’t know enough to do the work yourself.

What’s the way out of this trap?

Here is an experiment to try. Take something you’ve written and enter it into your favorite AI with the Critical Review Protocol prompt below.

This will ask a model to challenge the assumptions behind what you’re trying to do.

Critical Review Protocol

Follow these instructions precisely. Don’t be a sycophant; if you make errors, we both lose credibility. At each step, every analysis should build upon and reference the previous .md results.

Claim Audit (audit.md): Run an audit on every claim in this document. For each claim, provide backup references with specific citations including source, paragraph/section, page number, and direct quote.

Evidence Analysis (evidence.md): Perform actual calculations where applicable. Show all work step-by-step. Analyze the validity of each calculation, data point, or logical argument. Provide supporting or contradicting evidence.

Hostile Critique (feedback.md): Adopt the persona of a hostile critic who personally wants to find every possible flaw. Provide harsh but substantive feedback on methodology, logic, evidence, writing, and conclusions.

Forensic Investigation (investigation.md): Conduct a forensic analysis as if performed by a team of expert researchers working alongside lawyers and statisticians. Validate every claim for the purpose of confirming the work is sound, accurate, and well-supported. Flag anything that cannot be verified.

Literature Search (literature.md): Conduct an in-depth search of prior work. Look for both overlapping/supporting sources and contradictory sources. Requirements:

  • Minimum 40 references total

  • At least 20 references from the last 5 years

  • Include both peer-reviewed sources and non-peer-reviewed but highly credible outlets

Expert Panel Debate (debate.md): Simulate a debate among a panel consisting of:

  • A hostile critic

  • A senior subject matter expert in the relevant field

  • A junior expert or practitioner in a related specialty

  • A highly experienced statistician/analyst

  • An editor from a respected publication in the field

  • A senior expert in broader policy or practical implications

Debate the document’s claims, methodology, conclusions, presentation, writing, and recommendations. Show all turns, individual votes, disagreements, and final consensus.