Why some people never learn

I’m a test-prep tutor for graduate exams, and I’m a good one. I have hundreds of satisfied clients, many 5 star reviews, and my salary as a tutor is above the mean, for whatever that’s worth. Most people do well with me, and get the score they want. Some people do very well with me, and get a score that’s better than they could have dreamed.

And some people do poorly. Their score remains flat, even after working hard. This is a post about the last group of people.

This was something that mystified me for a while. It was never surprising that people who didn’t do the work didn’t learn. After all, I can lead a horse to water, but I can’t make it drink. But it was surprising that some people who did work hard (did drink the water?) still didn’t learn.

They weren’t dumb. They were like the rest of my clients for graduate school and business school exams: college-educated adults, with white collar jobs and good career prospects.  They weren’t dyslexic, or innumerate. They had taken classes before, and gotten good grades. So why did they fail to learn?

The traditional answer was just that “test preparation doesn’t improve scores, period”. But, that line of reasoning has since been abandoned. Besides, it’s hard to jive that with my students who jumped, in some cases, 20 percentiles while working with me. I have had students who studied hard on their own and took months of prep classes, made no improvements, then improved from just 10 hours of working with me.

So what’s with these students who don’t improve at all? Well, I think I have the answer. It’s reflectivity.

Reflective learning

The exams that I teach (the GMAT, GRE, and LSAT) are concrete and limited.

By concrete, I mean that there’s no fuzziness on answers. Give me a question from any of those exams, and I can reliably tell you the answer that the test-maker intends to be considered correct. This is true whether it’s a question from a test 20 years ago, or from a year from today.

By limited, I mean that there are only a certain amount of topics and subtopics that can be included. There are only a limited number of facts, formulae, and concepts that a student is expected to understand. The test makers strive to create a test that can be mastered without any experience in a specific culture, field of study, or environment.

One way to think about these tests is this: they are similar to sports, but not similar to art.

With enough video cameras, nobody doubts who won a basketball game. The goals and rules of a basketball game are known in advance, and there are no surprises with regards to either during a match.

No amount of video cameras can definitively answer the question of “who is the best musician” in a battle of the bands. And the existence of 4’33 means that nobody can have any expectations of everything to expect at a concert, or even to define what a concert is.

Because these exams are concrete and limited, like sports and unlike art, they are therefore ideal for mastery learning. In my definition (which is somewhat different from the traditional definition), mastery learning requires several steps. It requires a student to


1. Do questions
2. Recognize when they get questions wrong
3. Reflect as to why and how they got the questions wrong
4. Master the concept and similar concepts in order to not get a similar question wrong
5. Test their mastery on the same question and similar questions

In practice, I use an error log to force my students to go through this process. I assign them homework, and whenever they get a question wrong, they put it in the error log. From there, I have them log their subsequent attempts at the question. If I’m not sure they’re really understanding the concepts behind the question, I ask them probing questions to make sure they’re really mastering them. I don’t want them to just parrot back the answers or explanations.

This process is painful and frustrating, but it’s a crucial part of learning concrete and limited topics. To go back to my sports analogy, you can see the same process at work in Kobe Bryant’s practice schedule.

Kobe works tirelessly, practicing every single situation that could come up in basketball. He improves his dribbling and his shooting by playing endless one-on-ones, improves his cardio with early morning workouts, and watches film (and forces his teammates to watch film) to reflect on his overall performance.

Example of Kobe’s deliberate practice: he used to regularly play one-on-one games up to 100 with his teammates.

Kobe finds these practices rewarding because basketball has the same feedback loop as test-prep. He puts himself in a situation where he can make an error; he notices and reflects on any errors he makes; he improves and tests out his improvements.

You can even see it at work in the LeBron meme. You know, the one where LeBron got dunked on by Tatum, and was overheard saying the following:

“He got me,” LeBron said of Tatum’s dunk over him. “That f***ing Tatum boomed me.”
LeBron added, “He’s so good,” repeating it four times.
LeBron then said he wanted to add Tatum to the list of players he works out with this summer.

Notice the end of it: Tatum dunks over LeBron, so LeBron is determined to get dunked on by Tatum over and over again until he can stop him.

In order to learn, a student needs to do something wrong, realize it, reflect, master the concept, then test their mastery. I’m almost tempted to say this is the whole of learning. The rest is just commentary.

Let’s go back to my students who work very hard and never get anywhere. Why don’t they learn?

Well, it is almost always that they fail to follow this process. They work, but they don’t work right.  They might work hard, for instance, but they avoid the questions they tend to get wrong, because it makes them feel dumb. Or, they might simply grind through questions, but never reflect on why they get the questions wrong that they do.

I’ve actually spent a lot of effort on designing my teaching and teaching tools (like the error log) to avoid these exact issues. As a result, the proportion of people who don’t improve from my tutoring has gone way down, although I still have a few.

What’s the role of a tutor in all of this?

One natural question that one might ask is then, “If learning is about mastery, then what’s the role of a teacher?”

A possible answer, to be honest, is not much. Studies have found that the role of a teacher can explain between 5% and 20% of learning achievement. It’s behind many other factors in student learning: the school itself, the student’s background, the student’s innate abilities.

In my mind, the role of a tutor is 3 fold: motivation, advice on resources, and specialized knowledge.

Many people come to me as a tutor just to force them to work. They’ll actually tell me that, too. They’ll say, “I know I can do this on my own, but I need someone to tell me to do it”. Well, I can do that, although it does make me think that I could also just be replaced by their mom or an app.

Second, a tutor can provide advice on resources. In order to get useful questions (within the bounds of the test) and master concepts, a student needs some resources. After all, unless you’re Newton, you’re not deriving calculus on your own. A tutor can point to useful resources, and explain how to use them (i.e. through mastery learning).

Last, a tutor can provide specialized knowledge. If there are concepts that are poorly explained in the existing resources, or the concepts are best explained person-to-person (like shooting mechanics in basketball), a tutor can provide the explanation needed.

However, I’d be remiss if I didn’t say that it’s entirely possible for people to learn on their own. After all, that’s how I learned.

Conclusion

Putting in the work is necessary for learning, but it’s not sufficient. The most common reason that people fail to learn is that they either avoid trying things where they might make an error, or they are not reflective when they do make an error.

Given that our rapidly changing economy requires us to “adapt or perish”, the skill of learning new skills, specifically in concrete and limited topics, is vital. Unfortunately, this is not a skill that is taught in schools; in fact, self-teaching is often discouraged in schools. Each must learn this for themselves.