How to judge a critical thinking workshop for transfer

A team can spend a session naming the ways a confident person fools themselves, and still approve the recommendation they brought into the room. The vocabulary is new. The call is not. If you are about to pay for that session, a list of traps the room can recite is a weak receipt.

The booking that earns its cost is the one you can see on a later decision of the same kind. Someone slowed a claim, asked for the evidence that would change it, and had enough knowledge of the work to tell a fluent answer from a sound one. A completion record will show that people sat through the labels. It will not show whether the labels travelled.

Why the label does not travel

Daniel T. Willingham, a professor of cognitive psychology at the University of Virginia, set the problem out in American Educator in the summer of 2007. The essay is Critical Thinking: Why Is It So Hard to Teach?. He says people have treated this kind of thinking as a skill like riding a bicycle, something you learn once and then apply anywhere. The research he reviews says the processes of thinking are intertwined with what is being thought about. You can remind someone to look at an issue from more than one side. If they do not know much about the issue, they cannot do it. A maxim can be memorised. Without background knowledge and practice, it usually stays a maxim.

He is plain about the scope. Critical thinking, as he puts it, is not a set of skills that can be deployed at any time, in any context. Even three-year-olds can engage in it, and trained scientists can fail at it. He treats it as thinking that avoids the usual traps and is not mere memory of a solution already shown. The person doing it has to be calling the shots. He would not give much credit if a teacher prompted every step.

The studies he uses ask whether the thinking shows up on a new problem. That is the question that matters if you are about to pay. Novick and Holyoak, in 1991, had people read a garden problem whose solution turned on the least common multiple. Later they saw a school band that had to march in even rows. The mathematics was the same. The surface was not. Just 19 percent saw that the band problem could use the garden solution. Told the two were similar, 35 percent solved the band problem, against 19 percent with no hint. Most people, even with the hint, still could not. Knowing you ought to look past the story is not the same as seeing the structure under it.

How specific the knowledge is

Chen, Mo and Honomichl, in 2004, show how specific that knowledge can be. A treasure hunter has to get back out of a cave, and the useful move is to leave a trail of sand. About 75 percent of the American college students solved it. About 25 percent of the Chinese students did. The experimenters suggest the Americans knew Hansel and Gretel, so the structure was already familiar. A second puzzle, built on a Chinese folk tale, reversed which group solved it. The Americans were not better at puzzles in general. They were better at the one whose structure they knew.

What a separate course tends to change

Willingham then looks at programmes built to teach thinking apart from any subject, with patterns, mysteries or everyday problems, on the claim that a set of skills will travel. He is specific about why the cheerful studies are weak. Many test people only once. Many have no comparison group, or compare the programme with schooling left as it was, so a keen teacher could be the cause. Few ask whether people can use the new thinking on material that does not look like the exercises. As that test moves further away, he writes, the apparent effect drops quickly.

What those programmes do seem to teach is a reminder. Look under the story. Consider both sides. Do not settle on the first answer that sounds reasonable, and do not seek only the evidence that confirms you. A person who has heard it often enough is more likely to notice the moment. The reminder does not supply the knowledge the moment requires. You can know that you should refuse the first reasonable-sounding answer, and still not know how to produce a better one or how to weigh it. That takes knowledge of the domain, and practice at using it. His line is that knowing one should think this way is not the same as being able to.

He quotes a National Research Council committee, in Taking Science to School in 2007. Teaching content alone is not likely to lead to proficiency in science, nor is engaging in inquiry experiences devoid of meaningful science content. Chen and Klahr, in a 1999 study he cites, taught children in second, third and fourth grade to keep every factor still except the one under test, using materials such as springs. Seven months later, with different materials and a different experimenter, the children could still apply the strategy, and the older children transferred it more reliably. Recognising that they were designing an experiment brought the strategy back. Holding the other factors still depended on knowing which factors might matter.

Where training still fails

Training already in hand is not a blanket. Physicians, he writes, are known to discount or misinterpret patient data that conflicts with a diagnosis they already have in mind. Scientists with a doctorate are open to faulty reasoning when the problem sits in an unfamiliar context. Familiarity with one kind of problem does not cover the next kind.

The work the day has to use

Bring a decision the team is about to make, in work they know. A fictional case trains the surface of that case. The people in the band problem were not failing at the arithmetic in the abstract. They were failing to see it inside a story about a parade. A scenario about an invented product teaches sharpness about invented products. The next real call has a different surface. Do not expect the sharpness to travel unless they practised on that surface, or on several that share its structure.

Ask who in the room can tell a wrong step from a bold one. Willingham says the effect of those programmes depends on the teacher, because a written course cannot anticipate every idea the room will produce. In a company that person knows the work well enough to say a step does not follow, and why. A facilitator with a slide of named traps is not that person. If they are absent, the room can congratulate a fluent comment a specialist would throw out. When every step is prompted, he would not give the credit. The team is watching someone else think.

Read the agenda for the material. If the hours are named for traps, puzzles and a discussion of a case the provider brought, you are buying the kind of programme he found does not travel. If the hours are named for one live decision, the documents that sit under it and a pass where people have to produce the other side themselves, you are closer to what he says can be taught. The reminder still belongs. People should hear, out loud, that the first reasonable answer is not the end of the job. They should then do the job on their own material, with a response from someone who knows it.

What a later decision should show

Do not score the day by a quiz on the labels. His line is that you learn to think critically in the ways in which you practise. A quiz repeats the practice of the quiz. A later decision of the same kind is the test that matches the claim you paid for. Ask the manager, on a real call made after the session, what was challenged, what knowledge made the challenge possible and whether the person making it was calling the shots or reading a prompt. If they can name the trap and cannot name the knowledge, the reminder travelled and the thinking did not.

A decision on work this group does not know is the dissimilar material on which those programmes faded. If you need the same standard there, the practice has to happen there. Another pass over the same labels will not supply it. He is also clear this is not reserved for people who already enjoy a debate. The difficulty is recognising the moment, and knowing enough to meet it. If the people who will sign the decision stay outside, the session has inverted the problem.

What you are willing to reopen

Name the decision, the documents and the specialist before you book. Name one claim the room is allowed to reopen, including a claim a senior person has already blessed. If every exercise leaves that claim untouched, the day will teach people to be sharp about things that do not matter. The novelty he describes is the right standard. Applying a remembered checklist to a conclusion nobody is allowed to move is memory. It is not the thinking he is talking about.

Read the reopened claim back before anyone leaves, in a sentence the specialist will sign. A page of reflections written afterwards, by the person who enjoyed the discussion, will drift toward the labels. If the group cannot point at a claim that moved, or at a claim that survived because the evidence held, the session is not finished. Survival is allowed. An unexamined survival is the old decision with a new vocabulary.

Collective Campus is building high-performance culture and teams for the AI world. The critical thinking workshop is for a team that can bring a live decision and a person who knows that work well enough to say the reasoning is wrong. What changes when a polished draft is already on the table is in what a critical thinking workshop should change. Where this skill sits against a literacy course is in why critical thinking beat AI literacy. Teaching people to challenge a model output, rather than to comply with a rule, is in from compliance training to judgment.

What to bring

Bring the decision, the documents under it and the name of the person who can tell a fluent step from a sound one. If the decision cannot be reopened, you do not have a practice. You have a lecture about other people's mistakes.