Thirty-three adult individuals were recruited for Studies 1 and 2 and those who took part in one study did not participate in the other. Individuals were students or members of staff from the University of Surrey and were all right-handed. Both studies were approved by the University of Surrey’s Ethics Committee and all participants gave written consent prior to participation. Two participants withdrew from the studies (one from Study 1 and one from Study 2), leaving N = 16 for Study 1 and N = 15 for Study 2. Participation was reimbursed with £5 for Study 1 and £25 for Study 2. The latter also included a performance-related cash bonus to encourage motivation. Demographic data for both studies are presented in Table 1.

A computer-based motor task which involves the dexterous manipulation of an adapted vertical mouse was used in Studies 1 and 2. The adapted mouse comprised a commercially available wireless vertical mouse (Penguin Ambidextrous Vertical Mouse, Posturite Ltd., United Kingdom) with a plastic bottle attached to its vertical handle in order to increase task difficulty and counter possible familiarity with vertical mouse use (Figure 1A). The task was based on a freely available online game¹ which allows customization of many of its gaming settings. Two bespoke versions of the game were programmed for Studies 1 and 2.

In Study 1, the game comprised one block of four trials, consisting in a 15-s familiarization trial followed by three 1-min practice trials (Figure 2A). A 10-s countdown preceded the familiarization trial and a 30-s countdown was given prior to commencement of every practice trial. For Study 2, the game consisted of five blocks. The first and last blocks had four trials each, as in Study 1 (one familiarization trial plus three practice trials), while the remaining blocks were comprised of three practice trials each (Figure 2B). A 1-min rest interval spaced blocks 1, 2, 3, and 4 from each other. Blocks 4 and 5 were spaced by a 5-min rest interval. All other parameters were similar to Study 1.

In both versions of the game, the task consisted in hitting circular targets on an 18.5 cm × 20.5 cm computer screen frame by moving a cross-hair controlled by the adapted vertical mouse (Figure 1B). Touching a target with the cross-hair already counted as a hit; no clicking was needed. After each target was hit, another one would immediately pop up on the screen following a pseudorandomised spatial distribution pattern. No penalty was incurred if participants missed the targets and/or moved the cross-hair outside the screen frame. However, these indirectly reduced performance because of time costs. Participants were instructed to hit as many targets as possible during the practice trials.

The adapted mouse was placed on a 50 cm × 42 cm smooth mouse pad (HyperX FURY Pro Gaming mouse pad, Kingston Technology Corporation, United States), which was fixated to a height-adjustable table (Figure 1C). The task was performed in a standing position in order to increase task difficulty by adding postural demands. Table height was individually adjusted so that the participants’ right upper limb would not touch the mouse pad, again to place greater demand on motor control. Individuals were instructed to keep their left upper limb hanging on the side. The cross-hair on the screen was to be controlled by sliding the mouse across the pad without lifting it. Every trial started from a central position, i.e., with the cross-hair and the mouse at the center of the computer screen frame and the pad, respectively. The apparatus was slightly shifted to the right so that participants remained aligned to the center of the computer screen (Figure 1C).

For both studies, the number of targets hit and the average response time (RT), defined as the time in milliseconds elapsed between a target appearing on the screen and being hit, served as outcome parameters for statistical analysis.

The number of hits was higher in Session 2 (M = 78.25, SD = 7.61) than in Session 1 (M = 76.75, SD = 7.59). This difference, 1.50 ± 2.94, 95% CI (-0.07, 3.07), however, was not statistically significant, t(15) = 2.04, p = 0.06, and represented a small-sized effect, d = 0.20. A similar pattern was observed for RT, which was smaller in Session 2 (M = 728.24 ms, SD = 58.98) than in Session 1 (M = 738.86 ms, SD = 60.81). Likewise, this difference, –10.63 ± 24.11 ms, 95% CI (-23.47, 2.22), was not statistically significant, t(15) = 1.76, p = 0.098, and represented a small-sized effect, d = 0.18.

Estimated ICCs for number of hits and RT were 0.91, 95% CI (0.75, 0.97), F(15) = 25.65, p < 0.001, and 0.91, 95% CI (0.75, 0.97), F(15) = 23.69, p < 0.001, respectively, indicating good test–retest reliability for the task. The results are summarized in Table 2.

For both parameters, participants performed increasingly better in Sessions 2–5, when compared to Session 1, and the improvements in performance achieved at the end of the training period were largely sustained over the 1-week long-term retention interval (Session 5 to 6) (Figure 3 and Table 3). These observations were corroborated by a main effect of Session on number of hits and RT. Post hoc comparisons revealed statistically significant improvements in performance from Session 1 to 2, 1 to 3, 1 to 4, and 1 to 5, and retention from Session 5 to 6. The results are summarized in Table 4.

The estimates of SEM and MDC₉₅ were, respectively, 1.88 and 5.21 for number of hits, and 14.42 and 39.97 ms for RT. At the group level, for both number of hits and RT, from 1 to 4 days/sessions of training on the task, the 95% CI of the mean of the differences in performance moved away from the interval corresponding to the range of random measurement error, suggesting a trend toward a real improvement in performance with training, i.e., learning (please refer to Figure 4). Analysis of individual data confirmed this trend by showing an increase in the MDC₉₅ proportion (60–80%) from 1 to 4 days/sessions of training, for both number of hits and RT, hence indicating a motor learning effect for individual participants (please refer to Figure 5).

At the 1-week long-term retention test, at the group level, for both number of hits and RT, the 95% CI of the mean of the differences in performance between Sessions 5 and 6 was within the interval corresponding to the range of random measurement error (please refer to Figure 4). At that same time point, 75% of the participants (9 out of 12) who after 4 days/sessions of training displayed learning, i.e., an improvement in performance that was equal to or greater than the absolute values of the MDC₉₅, showed a change in performance which was within the range of random measurement error (please refer to Figure 5). Altogether, these findings indicate good long-term retention of motor learning from Session 5 to 6.

Table 5 displays the estimates of SEM and MDC₉₅ for both the number of hits and RT data. The respective MDC₉₅ proportions at the end of the training period are also displayed.

According to general guidelines (e.g., Portney and Watkins, 2015), our results suggest good test–retest reliability for the computer-based motor task we designed (Table 2). Although performance improved slightly from test (Session 1) to retest (Session 2), these differences were not statistically significant and represented small effects and, more importantly, did not affect test–retest reliability (Table 2).

Ensuring the test–retest reliability of a task/test not only gives researchers more confidence for attributing potential changes in performance scores to an experimental manipulation but, critically, it also allows for the estimation of the statistics of SEM and MDC, which in turn can contribute to unravel the likely origin of those changes (Beninato and Portney, 2011; Portney and Watkins, 2015).

Motor learning was assessed through four consecutive short-term retention tests administered 24 h after completion of the preceding training session. Such tests corresponded to the first block of the second, third, fourth, and fifth experimental sessions (Figure 3). It has been suggested that 24 h since the last practice session is an adequate minimal interval for the application of retention tests and thereby the assessment of motor learning (see Kantak and Winstein, 2012 for further discussion). The long-term retention of motor learning after 5 days/sessions of training on the task was also investigated through another retention test which was administered 1 week after the last training day/session. This test corresponded to the sixth experimental session (Figure 3).

At the group level, according to the p-value-based analysis of difference, for both hits and RT data, motor learning took place as early as after 1 day/session of training (Session 1 to 2), with improvements in performance increasing with the number of days/sessions of training (Figure 4 and Table 4). The analyses also showed good long-term retention of motor learning 1 week after the end of the training period (Figure 4 and Table 4). However, the analyses based on SEM and MDC revealed a somewhat different scenario. For instance, after 1 day/session of training on the task (Session 1 to 2), both in terms of number of hits and RT, the 95% CI of the mean of the differences in performance overlapped largely with the interval corresponding to the range of random measurement error, that is, the interval spanning between the respective ±MDC₉₅ values (Figure 4 and Table 4). This means that, at that stage, the respective improvements in performance could in fact have been caused mostly by random measurement error, as opposed to by learning. The scenario remained rather similar as training progressed to 2 and 3 days/sessions of training on the task. It was only at the end of the training period, i.e., after 4 days/sessions of practice, that the 95% CI of the mean of the differences practically did not overlap anymore with the interval corresponding to the range of random measurement error—there was no overlap for the hits data and the overlap was minimum for RT (Figure 4 and Table 4). This indicates therefore that it was only at that stage that the respective improvements in performance (Session 1 to 5) were more likely to reflect real improvements, i.e., to have been caused mostly by motor learning, as opposed to by random measurement error. This observation is critical as it could be used to inform future studies as to the minimum amount of training on the task that would be needed in order to produce robust motor learning. For instance, instead of only 1 or 2, or even 3 days/sessions of training, as suggested by the p-value-based analyses alone, adopting a longer training regime, e.g., at least 4 days/sessions of practice, would be a more appropriate conduct. By doing so, one could then be more confident that motor learning would be contributing more than random measurement error to any observed improvements in performance.

This seeming discrepancy between p-value- and MDC-based analyses might be due to the fact that the former does not take into account the random error that is associated with the measurement instruments producing the means which in turn are compared by the traditional inferential statistical tests, while the latter is performed directly from an estimation of the size of that error (see the respective formulas in the Supplementary Material). Assuming that measures of motor performance and their change over time are free from random error, irrespectively of statistical significance, is at odds with good research practice and should be avoided. This is a fundamental issue for motor learning studies, which typically test for improvements in performance scores from pre- to post-training assessments. In order to further support our abovementioned findings and arguments, by using the data obtained in Study 1, we simulated 10 hypothetical motor learning studies on a freely available statistical software (ESCI² – Chapters 5 and 6, 2011; see also Cumming, 2012, 2014 for further information on ESCI). Details of how these simulations were performed and their results are available in the Supplementary Material. In short, these simulated studies compared performance scores between a pre- and a post-training assessment, and they all produced two-tailed p-values < 0.05 for a paired-samples t-test with α = 0.05, and effect sizes varying from medium to large (minimum of 0.49 and maximum of 0.95). Although all simulated studies yielded statistically significant results and medium-to-large effect sizes, hence suggesting improvements in performance, i.e., learning, from the pre- to the post-training assessments, in 9 out of the 10 studies the 95% CI of the mean of the differences in performance scores overlapped with the interval corresponding to the range of random measurement error. In four studies, the overlap was total. In only 1 out of the 10 studies the 95% CI of the mean of the differences was completely outside (above) the interval of random measurement error —this particular result mirrors our abovementioned result for the improvement in performance, in terms of number of hits, that took place after 4 days/sessions of training on our task (Session 1 to 5). Overall, the results from these simulated studies lend strong support to the argument that p-value-based analyses of difference alone do not inform as to the likely origin of changes in performance scores, and that even improvements in performance which are found to be statistically significant can sometimes be due mostly to random measurement error, instead of to motor learning.

During the analysis of individual data, for both number of hits and RT, from 1 to 4 days/sessions of training on the task, an increase in the MDC₉₅ proportion was observed (60–80%) (Figure 5), indicating a motor learning effect for some participants at the individual level. Nevertheless, at the end of the training period, i.e., after 4 days/sessions of practice, for both hits and RT, that proportion did not reach 100%. This was due to the fact that, at that stage, 20% of the participants (3 out of 15) displayed a change in performance which was smaller than the absolute values of the MDC₉₅ (Figure 5). These participants could be considered “non-learners,” as their respective change in performance was likely to have been caused mostly by random measurement error, and not by learning. People differ in the way they respond to training and although we did not formally address this issue, it could be speculated for instance that such individual differences in learning might have emerged from variations in people’s motor ability, which according to Schmidt and Lee (2014) may be defined as “a fundamental characteristic of different individuals that tends to underlie particular skills; ability is largely inherited genetically and is not modifiable by practice” (p. 190).