The PEBL Trail-making test: Results

A while back, I blogged about the trailmaking test, and some of the algorithms used to do target path layout to hopefully produce better and more consistent results. The test actually contains two conditions: 1. Moving through points 1..25; and 2. Alternating between letters and numbers: 1 A 2 B 3 C .... I ran fifteen rounds of the test on myself.  Each point configuration is tested twice: once with numbers, and once alternating.  The order is mixed, so you tend to get half of configurations first as numbers only, and half in the mixed condition.  Also, the points are reflected across a vertical axis so that the problem is identical, but the details are a bit different.  I ran this twice: one with the 'path-shortening' algorithms to produce an efficient path (Which appears in the version in the PEBL Test Battery, and one with a random path through the points.  The following compares my mean solution times in seconds for Type A (numbers only)  and Type B (number-letter alternating) trials, and include standard deviations.

-------------------------------------------------------
                            Test                    

Type    Random                Shortened       
--------------------------------------------------------
   A     35.6 s (4.4 s)   13.0 s (2.4 s)
   B     36.8 s (4.6 s)   16.3 s (3.1 s)
  B/A    1.03               1.25
--------------------------------------------------------

The results look promising. First, the time to complete was more than cut in half, while the effect of alternating actually INCREASED, both in absolute and relative magnitude.  The relative difference between A and B was about 25%.  Also, the variability of solutions decreased substantially.  The shortening seems to have done the trick, at least for me, and makes it makes it so you can test about twice as many trials in the same time.



I also wanted to look at whether there were any meaningful order effects.  Remember that type A and Type B use the same configurations, but simply reflect the points across the vertical center:

Does having experience with the same configuration 'help' the second time through?  Personally, it was difficult to notice, even when I knew there are duplicates. On average, there did seem to be an effect of whether it was first or second time through, for both test types, with time decreasing the second time through:


---------------------------------------------------------------
             Numbers    Alternating
--------------------------------------------------------------
 First        25.9  s     28.2 s
 Second    22.9  s     25.2 s
--------------------------------------------------------------

But this should be taken with a grain of salt, because there is an overall effect or order--I might have just gotten better with time even if I didn't remember the problem, and there were no repeats. I did an ANOVA to examine whether the effect of first/second test was reliable:

                  stimate      St. Err. t val. Pr(>|t|)   

(Intercept)      35120.7      963.7  36.442   <2e-16 ***

Noise type      -21535.7      986.0 -21.842   <2e-16 ***

Test Type (A/B)   2292.0      977.2   2.346   0.0226 * 

Test order (1/2)  -106.7      988.2  -0.108   0.9144   

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3784 on 56 degrees of freedom

Multiple R-squared: 0.8978,    Adjusted R-squared: 0.8923

F-statistic:   164 on 3 and 56 DF,  p-value: < 2.2e-16

Results indicate that there was no reliable effect of test order, despite reliable main effects of noise type and test type.  I wouldn't say we are completely in the clear, but it is a good indication that using dual forms of the test is acceptable.

Using the tests twice means that you can directly compare problems, because the have sort of equivalent complexity. This was a big problem with the original Reitan points--the A form was shorter than the B form.  Now, considering just the 'reduced-length' tests, I found that for me, I solved 14 out of 15 of the numbers-only tests faster.  For the one that was slower, I was moving fast on the numbers only problems and didn't realize I missed a point, then had to backtrack 3 or 4 points to get it.  Here is a a plot showing individual solution times for the reduced-noise tests:

Here is another way of looking at the data. The red points are the 'shortened' ones, and the black are the random points.  Lines connect the two versions of the equivalent test (numbers-only versus alternating).  The same test was more consistently slower in the abbreviated test than in the random location path.




So that is a brief look at the data from the PEBL Trail-making task.  I think that having some published norms for the task would be great.  Anyone want to help?  Sign up for and email the pebl-norms list.