Open Letter to Khan Academy about Basic Probability
Statistics is not beautiful (sniff)
26 August 2013How to study statistics (Part 1)
9 September 2013Khan academy probability videos and exercises aren’t good either
Dear Mr Khan
You have created an amazing resource that thousands of people all over the world get a lot of help from. Well done. Some of your materials are not very good, though, so I am writing this open letter in the hope that it might make some difference. Like many others, I believe that something as popular as Khan Academy will benefit from constructive criticism.
I fear that the reason that so many people like your mathematics videos so much is not because the videos are good, but because their experience in the classroom is so bad, and the curriculum is poorly thought out and encourages mechanistic thinking. This opinion is borne out by comments I have read from parents and other bloggers. The parents love you because you help their children pass tests. (And these tests are clearly testing the type of material you are helping them to pass!) The bloggers are not so happy, because you perpetuate a type of mathematical instruction that should have disappeared by now. I can’t even imagine what the history teachers say about your content-driven delivery, but I will stick to what I know. (You can read one critique here)
Just over a year ago I wrote a balanced review of some of the Khan Academy videos about statistics. I know that statistics is difficult to explain – in fact one of the hardest subjects to teach. You can read my review here. I’ve also reviewed a selection of videos about confidence intervals, one of which was from Khan Academy. You can read the review here.
Consequently I am aware that blogging about the Khan Academy in anything other than glowing terms is an invitation for vitriol from your followers.
However, I thought it was about time I looked at the exercises that are available on KA, wondering if I should recommend them to high school teachers for their students to use for review. I decided to focus on one section, introduction to probability. I put myself in the place of a person who was struggling to understand probability at school.
Here is the verdict.
First of all the site is very nice. It shows that it has a good sized budget to use on graphics and site mechanics. It is friendly to get into. I was a bit confused that the first section in the Probability and Statistics Section is called “Independent and dependent events”. It was the first section though. The first section of this first section is called Basic Probability, so I felt I was in the right place. But then under the heading, Basic probability, it says, “Can I pick a red frog out of a bag that only contains marbles?” Now I have no trouble with humour per se, and some people find my videos pretty funny. But I am very careful to avoid confusing people with the humour. For an anxious student who is looking for help, that is a bit confusing.
I was excited to see that this section had five videos, and two sets of exercises. I was pleased about that, as I’ve wanted to try out some exercises for some time, particularly after reading the review from Fawn Nguyen on her experience with exercises on Khan Academy. (I suggest you read this – it’s pretty funny.)
So I watched the first video about probability and it was like any other KA video I’ve viewed, with primitive graphics and a stumbling repetitive narration. It was correct enough, but did not take into account any of the more recent work on understanding probability. It used coins and dice. Big yawn. It wastes a lot of time. It was ok. I do like that you have the interactive transcript so you can find your way around.
It dawned on me that nowhere do you actually talk about what probability is. You seem to assume that the students already know that. In the very start of the first video it says,
“What I want to do in this video is give you at least a basic overview of probability. Probability, a word that you’ve probably heard a lot of and you are probably just a little bit familiar with it. Hopefully this will get you a little deeper understanding.”
Later in the video there is a section on the idea of large numbers of repetitions, which is one way of understanding probability. But it really is a bit skimpy on why anyone would want to find or estimate a probability, and what the values actually mean. But it was ok.
The first video was about single instances – one toss of a coin or one roll of a die. Then the second video showed you how to answer the questions in the exercises, which involved two dice. This seemed ok, if rather a sudden jump from the first video. Sadly both of these examples perpetuate the common misconception that if there are, say, 6 alternative outcomes, they will necessarily be equally likely.
Exercises
Then we get to some exercises called “Probability Space” , which is not an enormously helpful heading. But my main quest was to have a go at the exercises, so that is what I did. And that was not a good thing. The exercises were not stepped, but started right away with an example involving two dice and the phrase “at least one of”. There was meant to be a graphic to help me, but instead I had the message “scratchpad not available”. I will summarise my concerns about the exercises at the end of my letter. I clicked on a link to a video that wasn’t listed on the left, called Probability Space and got a different kind of video.
This video was better in that it had moving pictures and a script. But I have problems with gambling in videos like this. There are some cultures in which gambling is not acceptable. The other problem I have is with the term “exact probability”, which was used several times. What do we mean by “exact probability”? How does he know it is exact? I think this sends the wrong message.
Then on to the next videos which were worked examples, entitled “Example: marbles from a bag, Example: Picking a non-blue marble, Example: Picking a yellow marble.” Now I understand that you don’t want to scare students with terminology too early, but I would have thought it helpful to call the second one, “complementary events, picking a non-blue marble”. That way if a student were having problems with complementary events in exercises from school, they could find their way here. But then I’m not sure who your audience is. Are you sure who your audience is?
The first marble video was ok, though the terminology was sloppy.
The second marble video, called “Example: picking a non-blue marble”, is glacially slow. There is a point, I guess in showing students how to draw a bag and marbles, but… Then the next example is of picking numbers at random. Why would we ever want to do this? Then we come to an example of circular targets. This involves some problem-solving regarding areas of circles, and cancelling out fractions including pi. What is this about? We are trying to teach about probablity so why have you brought in some complication involving the area of a circle?
The third marble video attempts to introduce the idea of events, but doesn’t really. By trying not to confuse with technical terms, the explanation is more confusing.
Now onto some more exercises. The Khan model is that you have to get 5 correct in a row in order to complete an exercise. I hope there is some sensible explanation for this, because it sure would drive me crazy to have to do that. (As I heard expressed on Twitter)
What are circular targets doing in with basic probability?
The first example is a circular target one. I SO could not be bothered working out the area stuff so I used the hints to find the answer so I could move onto a more interesting example. The next example was finding the probability of a rolling a 4 from a fair six sided die. This is trivial, but would have been not a bad example to start with. Next question involve three colours of marbles, and finding the probability of not green. Then another dart-board one. Sigh. Then another dart board one. I’m never going to find out what happens if I get five right in a row if I don’t start doing these properly. Oh now – it gave me circumference. SO can’t be bothered.
And that was the end of Basic probability. I never did find out what happens if I get five correct in a row.
Venn diagrams
The next topic is called “Venn diagrams and adding probabilities “. I couldn’t resist seeing what you would do with a Venn diagram. This one nearly reduced me to tears.
As you know by now, I have an issue with gambling, so it will come as no surprise that I object to the use of playing cards in this example. It makes the assumption that students know about playing cards. You do take one and a half minutes to explain the contents of a standard pack of cards. Maybe this is part of the curriculum, and if so, fair enough. The examples are standard – the probability of getting a Jack of Hearts etc. But then at 5:30 you start using Venn diagrams. I like Venn diagrams, but they are NOT good for what you are teaching at this level, and you actually did it wrong. I’ve put a comment in the feedback section, but don’t have great hopes that anything will change. Someone else pointed this out in the feedback two years ago, so no – it isn’t going to change.
This diagram is misleading, as is shown by the confusion expressed in the questions from viewers. There should be a green 3, a red 12, and a yellow 1.
Now Venn diagrams seem like a good approach in this instance, but decades of experience in teaching and communicating complex probabilities has shown that in most instances a two-way table is more helpful. The table for the Jack of Hearts problem would look like this:
| Jacks | Not Jacks | Total | |
| Hearts | 1 | 12 | 13 |
| Not Hearts | 3 | 36 | 39 |
| Total | 4 | 48 | 52 |
(Any teachers reading this letter – try it! Tables are SO much easier for problem solving than Venn diagrams)
But let’s get down to principles.
The principles of instruction that KA have not followed in the examples:
- Start easy and work up
- Be interesting in your examples – who gives a flying fig about two dice or random numbers?
- Make sure the hardest part of the question is the thing you are testing. This is particularly violated with the questions involving areas of circles.
- Don’t make me so bored that I can’t face trying to get five in a row and not succeed.
My point
Yes, I do have one. Mr Khan you clearly can’t be stopped, so can you please get some real teachers with pedagogical content knowledge to go over your materials systematically and make them correct. You have some money now, and you owe it to your benefactors to GET IT RIGHT. Being flippant and amateurish is fine for amateurs but you are now a professional, and you need to be providing material that is professionally produced. I don’t care about the production values – keep the stammers and “lellows” in there if you insist. I’m very happy you don’t have background music as I can’t stand it myself. BUT… PLEASE… get some help and make your videos and exercises correct and pedagogically sound.
Dr Nic
PS – anyone else reading this letter, take a look at the following videos for mathematics.
And of course I think my own Statistics Learning Centre videos are pretty darn good as well.
Other posts about concerns about Khan:
Another Open Letter to Sal ( I particularly like the comment by Michael Paul Goldenberg)
Breaking the cycle (A comprehensive summary of the responses to criticism of Khan
20 Comments
Hurrah! Well said.
Somehow KA receives great reviews among my colleagues. For a couple years now KA was the only video source allowed through our district internet filter. I really think no one making decisions ever looked at the stuff.
There is no way I would use any of their History stuff in the classroom. It’s just too awful.
The powers that be are dropping most of our filter just this week. Not because of the endless requests I’ve made to use sites other than KA. More because Administration realized they were fighting a losing battle.
I’ll now be able to recommend your videos to my “numbers” colleagues.
Thanks Willis.
I just watched the 60 Minutes programme and it was so unbalanced. The idea of Khan academy is fair enough, but the quality is poor. I think part of the problem is that there is so much stuff that no one has time to review it.
I thought the history stuff would be bad.
I have an issue with people who spam the ANZSTAT list. I have put a comment in your thoughts section, but don’t have great hopes that anything will change.
You might like to ask the list how many feel that my weekly reminder of a blog is spam? Quite a few people really like it. I presume that is what you are implying. And maybe it would be more polite to use your real name?
I like it, and find it a welcome change to all the generic job ads. Your posts often make me actually think!
Thanks, Kathleen. I appreciate the endorsement.
It is interesting, Nicola, that you raise the idea of being polite. Some would consider it impolite to post a reminder every week as you do – given that people do not have a choice to accept, or reject, what has become a regular email. Spam-victim is clearly one. I think it is your place to ask the list who wants the weekly reminder and then it is up to you to contact them weekly (via their personal email) and not contact everyone; that would seem (to me at least) a common sense way to do it. Maybe three times a year you could post to the list to invite new listers to come and read your thoughts.
You say ‘quite a few’ really like it. I would have thought a %, in this instance, would be a useful way to measure the list’s desire.
I offer this in the spirit of constructive criticism, just as you have done in your open letter to Mr Khan.
Hi Anthony
Here is a suggestion. You can put my email address in your spam filter and then you need never see any email from me again. Unlike malicious spammers, I do not change my email address. Problem solved!
Nicola,
The point here is not whether or not I, personally, want to receive your weekly reminder. The point is what is broadly acceptable practice on a professional email list. I am on a number of professional lists and your type of email is the only one like it I receive. Most professionals are capable of remembering/decided, without a weekly reminder, to visit a blog and read a post. Or, if they like the blog, they will subscribe to it and get the information that way.
Anyone can choose to put your email in their SPAM filter, of course, but then if you contribute something to the list that is a comment about an issue that is worth reading and thinking about it will be missed – kind of defeats the purpose of the list.
Also, why should an individual have to act when they did not initiate the process. Do you really feel it is OK for you to make multiple people do something that one person (you) can fix?
Once again, this is offered in the spirit of constructive criticism.
[…] 1 September 2013 | 11:41 pm Khan academy probability videos аחd exercises aren't good еіtһer Dear Mr Khan You һаνе created аח amazing resource tһаt thousands of people аlƖ οvеr thе world ɡеt a lot of һеƖр from. Well done. Some οf уoυr materials are&9999;… http://learnandteachstatistics.wordpress.com/2013/09/02/khan-academy-probability/ […]
Here, here.
I feel that KA fanbois are like Harry Potter groupies, both can’t deal with open, honest and constructive comment on their deity.
You drill to the heart of the matter – teaching a subject with a deep and experienced understanding of specific pedagogy is poles apart from being able to narrate a one dimensional explanation of facts, often with debatable rigour and production values.
As a teacher I find that most online video resources are lacking in some way – not to say they are totally flawed, but just that most are only 75 to 85% well balanced and complete. This is OK for science and maths at K12 level, but for statistics the often conflicting explanations of “what the numbers actually mean” is off-putting for learners and teachers alike. And this is the heart of what I find challenging in KA..
Just because you can KNOW how to do something (say, make pizza dough) does not mean that you are equipped and competent (and dare I say arrogant) enough to make a video explaining it to others.
Sure make the video, publish it on the web. For some, you way of explaining will work. For others it will fall on fallow ground, and for some, it will confuse the heck out of them. Heck, thats the world of teaching we inhabit. But please, when someone criticises you, accept it as part of the job – and actually acknowledge that others better schooled and experienced in pedagogy might just know better.
Dr Nic a thoughtful critique of KA – personally, I always offer a “health warning” to all my students that KA is not a panacea of knowledge and indeed some things are plain wrong.
Rather retro, but I still recommend actual printed text books – at least they have gone through some editorial control before being committed to paper.
Keep up the good work.
Glen
Its funny that you write about the way Khan Academy is not properly teaching people in the right way but more of a regurgitation of what many public schools teach. My opinion is that I do not see too many people out there trying to make learning an engaging experience. I have had an issue throughout my life of not being able to understand what has been taught to me, from grade school up to and beyond college. School has been arranged almost in a militaristic way to force a sort of eidetic memory of sorts without the understanding of why things work together. I am not writing this as a “Fanboi” but more from my own perspective of being a student having an issue with just remembering and regurgitating information without knowing what the information actually means.
I believe I have learned more from Khan Academy’s math section in 8 hours of use than I had taking the equivalent four quarters of math at the college I went to. For me this is a big step and a preferred method of learning without having my corpse picked clean of all its monetary value. for me just watching videos online does not cut it as I learn better by having the chance to figure out problems myself than just being force fed boring information.Some people learn differently and require different approaches to absorb information. For me Khan Academy works quite amazingly and even the gimmick of fancy graphics, learning badges, Energy Points, and structured Mind Map, all help me to strive forward into learning as much as possible.
You are doing a wonderful job teaching others in your own way and I applaud you for that.
I found this discussion by searching the internet “sal khan voice irritating” and khan academy videos so boring I can’t stand to watch.” I just wanted to know that I was not the only one. I hesitate to recommend the site to kids because eventually his voice will bore them to utter tears. ty
[…] rather than the subject of my post. This is borne out by the fact that a more recent post: “Open Letter to Khan Academy about Basic Probability” gets considerably less traffic. I suppose there are not many people who want to know about the […]
[…] about the probabilities, which is confusing when we start discussing models. If you look at the Khan Academy videos (but don’t) you will find trivial examples about coloured balls or sweets or strangely complex problems […]
Couple points I disagree with:
– You complain that he doesn’t use the proper terminology to present subjects (complementary events): in real life, problems don’t present themselves to you labeled with their terminology; they present themselves exactly as Khan presents them. Presenting terminology before a person has a good, working knowledge of the subject is like teaching toddlers grammar before they learn how to speak.
– You criticize Khan’s use of coins and dice as examples, rather than including “the more recent work on understanding probability.” He is obviously trying to use examples everyone can immediately relate to. You don’t need examples out of astronomy when the goal is to teach probabilities. Cards and coins suffice. (You also seem to object because of your feelings about gambling, as if there were no other games with playing cards that don’t involve gambling. I think it’s safe to say that almost everyone, even the most pious Christian, is familiar with the makeup of a deck of cards.)
As a college student in Brazil, I often find myself referring to KA when I need a clear, working understanding of the subject I’m on. This entire open letter reads a bit as if you resent KA’s popularity as opposed to yours. But because there just may be a chance that your videos are superior to Khan’s as you seem to suggest, I will certainly be checking them out.
Hi
Thanks for caring enough to comment. I didn’t complain that Sal didn’t use correct terminology. I suggested that putting it in the title might help someone who was having difficulties with complementary events to find this video.
I said that coins and dice are big yawn – which they are. And in this day and age of computer games, we are finding that quite a few students do not play card games and do not know how many cards are in each suit. There are people from some religious backgrounds who will not allow cards in the home. I would try to make a learning experience inclusive for these people also.
The other problem with coins and dice is that they perpetuate the misconception that all probabilities are equal, such that if there are 6 outcomes, the probability of each must be one sixth.
I do resent the lack of quality in Khan’s offerings. When he is as popular as he is, I believe he has a responsibility to get it right.
Please do check out my videos. You can see the play list on probability here:
[…] thought I’d better take a look at what Khan Academy did to random variables. I started watching the first video and immediately got hit with the flipping coin and rolling […]
Im sorry to see all these stories… let me sugget a bit of the meta-approach…
for there is an unwritten rule in life – DO your own story/product/promotion WITHOUT putting the Other down! Can you not convince the world out there, obviously in need of good educational videos on this topic, without criticising the competitor so bluntly? You could have followed that ‘more stylish’ way of promoting your own product without wasting your time to even mention the Other (unless saying something positive)… simply let your own quality convince users… get out of the mud, after all you and your competitors are on the same teaching-side and share the same mission…teach what you know and allow for plurality, so that any individual user could chose according to their own preference…
Remember, “Coffee, tea, or me?” …. Before we studied Probability there was only one way to answer this humorous question. But in probability it’s confusingly different: It’s P(coffee or tea or me) = P(coffee) + P(tea) + P(me) (assume intersection is zero). This what I think is called the inclusive or, which is not the meaning of the word ‘or’ that we use in everyday life.