A couple weeks ago, I was watching a guest lecture from Mark Caulfield from the University of Washington, and this post is for reflecting my thoughts on his insights.

In his guest lecture, Mike describes generative AI as like a submarine. It works the best when the water (or the data it’s trained on) is deep, but when it’s shallow, virtually nothing works. AI, especially chatbots, such as ChatGPT can be an incredibly powerful tool when given the right instructions.

In Mike’s case, he’s aiming to use to automate his SIFT (Stop, Investigate, Find and Trace method), which he presented an AI-powered interpretation of in the lecture.

Mike showed off a custom version of ChatGPT called the Toulminator, which uses the philosophy of Henry Toulmin, (shown below) to break down the claim of text from screenshots. I’ll have to do a bit more research myself on the Toulmin model to fully grasp how it works, but from what I understand, it breaks down a claim and its surrounding evidence, then determines through reasoning whether it’s true.

Above: an example of a tweet that Mike Caulfield’s Toulminator analyzed taken straight from the guest lecture. Below: Toulminator later verified the claim to be false based on the argument it presented.

I was genuinely surprised by the results when I saw Toulminator in action, and I think tools like it would come in quite handy in the age we live in now. Search engines excel at finding information tailored to your specific worldview and make you feel that you’re right. If you find a piece of information that helps strengthen one of your statements, it means someone else already has its counterargument, curated by their personalization of the algorithm. Whilst GenAI makes a lot of mistakes stating facts, it can be great at checking preexisting facts with the right prompting, as stated by Mike Caulfield himself.

If Toulminator does become a mainstream tool one day, I would use it to check anything I’d find suspicious every chance I get.