Reimplementing Bem's Feeling the future

Photo from yd on flickr

Daryl Bem has been making headlines with his series of studies that claim to demonstrate ESP and precognition, or in this case, retrocognition.  Two of the studies showed that studying words AFTER you recalled them improved how well you did.  That is, study a list of words. Then recall as many as you can.  Then, half of them are chosen, and you are asked to study them.  Apparently, he found that the words you later studied were recalled better (by a tiny bit).  

For whatever reason, the critiques offered are like a projective test: each criticism sees the results as a consequence of their own hobbyhorse, whether it is failure peer review, frequentist versus Bayesian reasoning, or whatever.  There are a number of replications underway (see here  and here) which should deal with the critique of replication, but my hobbyhorse is experimental software, so I'll point out that a good replication should be a clean implementation, especially for controversial effects like this.


There are many errors that could have been made, either in the logic of the program (how it labels or saves data), or in the design that could lead to systematic errors or greater variance than would otherwise be expected.  For example, in Bem's study 8, all participants apparently used the same list order.    Well, free recall has large recency effects, which means that the words people recalled  will be highly constrained by the fixed list order. Also, it is very common for canned list shuffling routines to work incorrectly (even if they use a truly random source), and produce new random orders that are correlated with input position.  In theory, these factors could produce the types of results Bem demonstrated in the recall experiments.  I would argue that even if the results can be replicated, his data should not be taken seriously until his source code can be evaluated.  Although his experiment is available for download, his source code is not, and so it is impossible to verify this.  Plus, if you want to download his study, it is sort of difficult to run, and it is like 200 MB (probably because it includes a bunch of videos or something). 

I've implemented an abbreviated version of the test (just the memory test; not the meditation or personality variables) in case people want to try it out.  It takes between five and ten minutes to complete. You can download it at the link below.  It is only 10K--just a few hundred lines of code, and should be pretty simple to use and evaluate.

Instructions for Windows:

1. Download and install PEBL http://sourceforge.net/projects/pebl/files/pebl/0.11/PEBL_setup_0.11.exe/download

2. Download the PEBL implementation of Bem's Study 8 at http://obereed.net/feeling.pbl.  Save the file somewher like My Documents/pebl-exp.0.11
3. Run the launcher, select feeling.pbl, and hit run.  If you use fullscreen but want to exit early, remember that ctrl-alt-shift-\ is the magic escape key combo.
4. Put a unique code in the entry saying 'enter subject code'. 
5. Try your hardest to remember the words, and to study them later.
6. When you are done, email me the file results to me at user name stmuelle at that google-sponsored email service (gmail),  with "BEM STUDY RESULTS" as the subject, or post the line in results.txt in the comments.

If I get enough responses, I'll post the results, with the caveat that this is for entertainment purposes only, and people could be cheating, either one way or the other.