One of the most important and difficult tasks in our society is forecasting a person’s potential. In music, dance, and theater, choices regarding who is cast are made based on a few minutes interaction with potential candidates. Similarly, top-rated reality TV-shows like American Idol or Australia’s Next Top Model attempt to forecast potential based on sometimes very brief performance exposures. In each of these situations decisions are made on the basis of available information and assumptions about how this information relates to future performance. In sport, the entire field of talent identification is grounded in the notion that performance during early stages of development gives some indicator of an athlete’s future potential (Baker et al., 2017).

One of the cornerstones of sport science is understanding the factors explaining exceptional performance (Starkes and Ericsson, 2003; Baker and Farrow, 2015). Although this information is valuable for appreciating the physical, cognitive and psychological capabilities of our species as well as for testing models of how humans respond under intense stress, its primary raison d’être is usually to provide coaches and trainers with information to improve judgments about talent selection and development. Despite its potential importance to sports systems worldwide, there has been very little research considering whether performance parameters measured earlier in development (e.g., in youth) are valid predictors of expertise later in development (e.g., as an adult).

One of the factors limiting research in this area is the difficulty of obtaining high quality youth samples and managing to track them over extended periods of time in order to determine the level of performance they ultimately attain. Moreover, because most talent development systems operate via a ‘de-selection’ process (i.e., most athletes are removed from the system with a select minority being chosen for continued advancement) the number of athletes who start in the system and the number exiting the system at its highest level are considerably different, which increases the administrative, logistical and financial demands on researchers aiming to obtain samples of sufficient size for inferential and comparative analyses. Although demanding, longitudinal studies of talent development have begun to emerge – most notably from Elferink-Gemser, Visscher and their colleagues as well as Till et al. (2013, 2015) and Deprez et al. (2014, 2015a,b,c). These studies have been extremely useful in identifying the varying influence of a range of variables, such as self-regulation, quantity of practice, and early measures of technical and tactical skill on the long-term development of athletes in soccer, field hockey and rugby, among other sports. Within sport science almost all talent prediction research is based on the assumption that there is a ‘formula’ that could identify athletes best suited for specific sports. An extreme example for this was the talent system in the former German Democratic Republic, in which all school children were scanned with various tests to look for students with talent (Kupper and Thieß, 1971). The assumed linearity underpinning these approaches (e.g., someone who performs well at the talent selection stage will perform exceptionally in later years) has been criticized from an ecological perspective (Renshaw et al., 2012), resulting in these approaches having had negligible effects on improving the efficacy of talent selection decisions.

Available evidence of the accuracy of talent decisions by coaches and scouts is not compelling. Koz et al.’s (2012) study of the accuracy of selection decisions for professional sports ‘entry drafts’ suggests that even when these decisions are made late in development (i.e., early adulthood), the level of predictive accuracy is relatively low. However, these types of archival analyses (see also Staw and Hoang, 1995; Berri and Simmons, 2009) may provide negligible information regarding the quality and accuracy of decision-making during early development (e.g., during youth). While there is some suggestion that coaches experience and training may allow them to see something less qualified individuals have difficulty seeing (Christensen, 2009), this notion has not been adequately examined using comparison-based designs.

In the sections below we describe the results of an analysis that evolved over an extended timespan. One of the authors was involved in the initial assessment of a group of youth handball players over 10 years ago and we are now able to determine how these athletes advanced across their athletic careers. The primary aim of this exploratory study was to evaluate the prognostic validity of talent selections by varying groups 10 years after they had been conducted. In a first step of this study, motor test results acquired during a youth talent camp were used to determine a possible model fit. Following Schneider et al. (1993), these tests are often used to measure general motor abilities of young athletes (e.g., Söğüt, 2016) and are widely administered by many sports for talent selection (e.g., Schorer et al., 2012). While often administered, the use of motor tests seems debatable (Lidor et al., 2009), although these tests have had some success in handball (Lidor et al., 2005; Matthys et al., 2013). While Matthys et al. (2013) found some differences between elite and non-elite players in the Yo-Yo intermittent recovery test and on the speed and coordination items, in the study by Lidor et al. (2005) only a handball skill specific test differentiated.

In part two of this study, the original subjective predictions of national and regional coaches who scouted during the tournament of this youth talent camp were evaluated. Determining the long-term accuracy of their predictions is the main focus of our study. In part three of this study, these coach predictions were compared to those made by amateur handball players and novices (i.e., no handball experience) who viewed short videos from the tournament matches. This final element was included to compare the expert decisions of the national and regional coaches with advanced performers (i.e., players) and novices (Abernethy et al., 1993) to determine whether the accuracy of prediction was differentiated by skill level. Because this comparison was conducted after the talent selection occurred, the amount of information given to these groups was limited. Therefore, these groups serve only as a baseline comparison to the performance of the national and regional coaches.

Because much of the research design was determined by serendipity as much as it was by conscious planning, we are cautious in how we present these results; however, given the lack of data on the efficacy of coach/scout decision-making in talent settings, we believe the comparisons (i.e., the motor tests, coach decisions, and players and novice decisions) provide us with the opportunity to explore the prognostic validity of talent selections over an extended period (cf. Vaeyens et al., 2009; Rees et al., 2016).

This study was conducted in three parts. These parts evolved over time and are presented in temporal order below.

This study compared talent selection and forecasting decisions across different levels of skill. Our results indicated only small differences in the classifications between the different models using varying predictors. As expected, the post hoc logistic regression using the motor tests outperformed all other forecast models, presumably because this approach used all available data to create the best model-fit. It needs to be restated here that it can only be considered cautiously, because it is no real prediction, but it provides us with an idea how much variance might be possible to explain. This caution increases even, when you see that the prognostic validity of the predictors seems questionable. From a practitioner’s viewpoint, these would likely not be the tests chosen as the most predictive of later performance because they do not measure handball-specific skills (e.g., handball distance throw). Future studies need to confirm the prognostic validity of these tests (cf. Lidor et al., 2009).

The slightly superior performance of the national coaches was interesting. On the one hand, national coaches should be significantly better at forecasting the future success of the talents they scout. However, on the other hand, they were at the try-out camp and used information collected over the 5 days to make their decision (as well as any personal experience they might have had); moreover, and perhaps more importantly, they may have had a role in influencing future performance by training the selected players in the national team and also identifying them as highly talented players for the professional clubs, which may have resulted in greater quantity and quality of training. The same holds for the regional coaches who are usually ‘active agents’ in the training of some of the athletes. In the light of these confounds, the issue is perhaps not that the well-educated and trained coaches were superior to the performance of the less educated/knowledgeable groups but that the differences between these groups were so small. Future studies are necessary to replicate these results, ideally with greater test power, and determine the influence confounding variables may have had.

In these future studies, some limitations should be overcome if possible. Most significantly, our design limited the information to which the advanced and novice groups had access. While this may have prevented lesser skilled decision-makers from being overwhelmed by the amount of data (see Goldstein and Gigerenzer, 2009), it resulted in a discrepancy in available knowledge between the groups for making their decisions. The coaches, for example, had access to information drawn from observations of the games played, results of the motor tests and through interactions with other coaches at the talent selection camp, not to mention any personal knowledge they may have had about the players. On the one hand, a qualitative study examining ‘how’ coaches make these decisions may be valuable. On the other hand, it may also be possible that these differences in information access could be avoided using more strict laboratory protocols (Marewski et al., 2010; Raab, 2012). It should be noted that while controlling information in lab settings might improve experimental control in the study design, it could decrease the ecological validity and representativeness of the tasks under examination. Collectively, the current results, coupled with those from future qualitative and experimental studies with more precise controls, may do much to explain the practice of talent prognosis.

Three points should be considered from a practical viewpoint. First, although our focus has been on comparing the different levels of assumed decision-making skill to a post hoc model, there may be some inherent value to the use of objective tests that was not explored in the present study. For instance, objective tests have a pedagogical aspect to them. By controlling for specific skills with these tests, every players gets feedback on his or her performance, something that can be very useful in the learning process (Shea and Wulf, 1999). The coach can tell each player where there is room for improvement and in which part of their performance they are already superior.

Second, it seems important to not only look at the percentage of correct classifications. Depending on your position in the sports system, type-I- or type-II-errors have different consequences. For instance, if the prediction model produces more type-I-errors, low achievers are supported and developed although they might not eventually reach elite levels of performance. However, in sport systems under financial constraints (i.e., most sport systems), these types of errors are not affordable. Prediction models that fail to identify an athlete with high potential result in lost talent, which can be particularly damaging if the numbers participating in the sport are low. While the effectiveness of talent development systems has been questioned (Vaeyens et al., 2009), the overall goal of athlete development systems is to facilitate the most talented persons to reach elite status; therefore, it is important to consider which type of error is more relevant for the improvement of the talent identification and development process.

Third, given the amount of error that is found in all predictions, it seems reasonable to create a system in which athletes are not only considered as potential high-performers at a single time point. Therefore, multiple testing or observations should be included in the talent system. For the German Handball federation, for example, this was implemented by having two national try-outs that are a year apart, as well as having the national coaches scout at elite level youth matches (Schorer et al., 2012). A similar point relates to the need for talent selection process to acknowledge the challenge in creating representative tasks that capture the nuanced relationships among skills that may be still developing (e.g., anticipation, decision-making) and performance outcomes (as a youth and as an adult) that may be predicted by different combinations of variables.

The use of longitudinal data for our analyses was a strength of this study. Most studies in sport consider how specific tests (e.g., anthropometric, cognitive or physiological) predict selection at one point in time. Little research has considered how early performance predicts achievement at later points of time (Güllich and Cobley, 2017), with still less considering how to forecast performance after 10 years of talent development, a time period noted as being an important threshold in the development of expertise (Ericsson, 2007; Ericsson and Williams, 2007; Ericsson et al., 2009). Despite this strength, there are some limitations to our analyses. First, the small number of participants resulted in goal keepers and field players being analyzed together despite the clear performance differences between these positions. Future studies should try to increase the number of observations and try to differentiate between playing positions. Additionally, the testing was conducted based on the standards from over 10 years ago and as a result some of the tests from 2001 are no longer considered as ‘state of the art.’

It is noteworthy that two athletes within the sample participating at the talent camp later became National team players. One of them, a goalie, was recognized by all of the coaches. The other, a field player, was not recognized by either of the national coaches and only two of the regional coaches. At the age under investigation here, she seemed to have nothing outstanding one could see. Anecdotally, she reported that she started training quite hard after the first selection camps and subsequently became a national team player. These examples illustrate how difficult the forecasting of future national players might be, because the process of talent development is often non-linear, highlighting again the problematic nature of type-II-errors (Renshaw et al., 2012).

In summary, talent selection is a very difficult task; ideally it is a decision that should be re-considered often, in order to consider performer-specific changes over development. Based on the results of the current study, understanding the foundation on which these decisions are, and should be, made seems an open field for future research in all kinds of talent domains.

The study was conducted over 10 years ago. At this time the institute of the first author did not have an ethic committee. But the study was conducted in accordance with the ethical declaration of Helsinki. We received written informed consent from all participants that made decisions.

JS conceptualized and designed the study. He acquired, analyzed and interpreted the data. He also drafted substantial parts of the manuscript. RR and LF wrote and revised parts of the manuscript. JB conceptualized and designed the study. He also drafted substantial parts of the manuscript. All authors read and approved the final manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer AV and handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.