Smallholder farming systems produce the majority of the food consumed in many lower- and middle-income countries, and contribute significantly to national and local economies. However, a transformation is needed to deliver food security and decent incomes for the farmers themselves, and to feed the growing populations within those countries. This transformation must be environmentally and socially sustainable to be successful in the long term. One obstacle is the lack of good quality, timely, and targeted information.

In this editorial we unpack three key terms from the title of this Research Topic, and use the articles published to illustrate those terms. The key terms are “data-oriented,” “agile,” and “system transformation.” The term “data-oriented” is used to refer to big data, the compilation of data, replicable analysis methods, and the various other developments facilitated by the digital revolution. Dealing with one of the negative features of the digital revolution is a recurring theme: information overload—or “infobesity”—whereby the flood of non-useful information hampers rational decision making. The term “agile” is used to refer to an emerging but not yet clearly defined methodological style, which tends to be enabled by the digital revolution, attempts to deal with problems of infobesity, and attempts to deal with the challenge of conducting outcome-oriented science in complex and uncertain situations. The term “system transformation” refers to efforts to stimulate and facilitate sustainability transitions within the smallholder farming sector. These terms are explored further below. The Research Topic focuses on research tools (tools or methods for knowledge creation) and excludes tools which are primarily geared toward the implementation of farming activities.

Digital and data technologies have far-reaching implications in many sectors, including agriculture (Klerkx et al., 2019). Much of the innovation has been in industrialized farming systems using technology to increase the efficiency of production (Wolfert et al., 2017; Basso and Antle, 2020). In less industrialized farming systems, digital agriculture has been related more to information services, for example extension advice, weather, or marketing information (Malabo Montpellier Panel, 2019). The potential for digital and data technologies to alter agricultural innovation (for better or worse) has been recognized but not received much attention (Fielke et al., 2020). One of the major side-effects of the digital revolution has been massively increased data collection, and practically unlimited data storage. This opens up positive but also negative possibilities—the temptation to record too much non-useful information can lead to infobesity.

The articles within this Research Topic seek to address infobesity in two main ways. The first is to make better use of the extant huge data resources, through improved data management, replicable analyses, and other best practices. The second way is to control our scientific appetite for data, via the creation of agile tools and methods, and the re-orientation of researchers' attitudes.

Gorman et al. conducted a systematic review of recent studies using household-level smallholder survey data and found that in the vast majority of cases best practices were not being followed. Only 14% of the studies made their data accessible. After descriptive statistics, linear regression was the most widely used analysis method (64% of studies); and was generally used inappropriately, to explain context specific and complex associations without adequate reference to that context or complexity. More than half (59%) drew conclusions which extended beyond the scope of their data or analysis. This rather damning analysis points to a lack of coordination which prevents the research community working on smallholder development from building up a coherent body of evidence over time. We should learn from the field of medicine in which standards have been agreed on what data is collected, how impact is evaluated, and how metadata and study context are recorded (von Elm et al., 2007; Field et al., 2014).

Kruseman puts forward a metadata schema to overcome the lack of interoperability in messy socio-economic datasets, borrowing concepts from information science and the development of the World Wide Web. Devare et al. describe a tool for creating digital agronomic field books, in which data is recorded and published according to best practices, including linkages to agronomic ontologies and publication in open access databases. Andrade et al. go further along the data pipeline, beyond data acquisition and organization to deliver analysis and actionable insights to decision makers within agricultural value chains. They note a widening gap between those who can and cannot process the modern forms of data, for whom analysis is a key bottleneck.

The agile methods in this Research Topic were enabled by the digital revolution, and have developed in response to the problems of infobesity. But that is only half of the story. Agile methods are also necessitated by a fundamental challenge faced when applying the scientific method in pursuit of agricultural sustainable development in lower- and middle-income countries. There is a tension between the scientific desire to collect comprehensive, granular, and precise information vs. the practical realities of conducting that research in resource-constrained environments where there is typically poor record keeping, a low level of education, and low institutional capacity. Attempts to record overly-precise or overly-granular information can be counter-productive, undermining data quality, relationships, and taking resources away from other important scientific activities, such as interpretation, publishing, and stakeholder engagement. The application of science in agricultural development fits many of the features of post-normal science (Funtowicz and Ravetz, 1993), whereby facts tend to change depending on the stakeholder viewpoint, place, or time of study; timeliness is key; and outcomes rely on complex negotiations rather than linear logical arguments. Many of the agile methods and tools implicitly recognize this and respond to the situation.

Articles by Eitzinger and by Chelanga et al. present novel data collection approaches. Eitzinger's 5Q approach used interactive voice calls to ask only five questions to over 37,000 respondents. The five questions were selected using a decision tree system, which, through many individual calls, builds up a rich data resource covering a larger number of questions. Chelanga et al. described the Kaznet smartphone app, which was used by pastoralists to monitor livestock, grazing, and market conditions whilst they went around their daily business in Northern Kenya and Southern Ethiopia.

Five articles conducted analyses based on data derived from multiple implementations of agile tools. Milner et al., Marinus et al., and Caulfield et al. each compiled secondary data from previous implementations of the Rural Household Multi Indicator Survey (RHoMIS) (Hammond et al., 2017; van Wijk et al., 2020). Gotor et al. combined data from RHoMIS and the tricot approach (van Etten et al., 2019; Brown et al., 2022), while Teeken et al. combined data from RHoMIS and an application of the 1000 minds tool (Hansen and Ombler, 2008; Balogun et al., 2022). Another three articles collected novel data using the RHoMIS tool and presented analyses based upon that data (Alary et al.; MacLaren et al.; Périnelle et al.). van Vliet et al. and Harris et al. compiled large datasets from more traditionally implemented household surveys. All of this demonstrates the utility of following good practices in data management, and how effective design of data collection tools facilitates enhanced use of that data.

The term “system transformation” is increasingly used in the context of sustainable development and agriculture research (e.g., CGIAR, 2021). Although a somewhat nebulous concept, it usefully articulates the outcome-oriented nature of the research. Analyses of system transformation generally focus on themes of resilience, robustness, rigidity, adaptability, and transformability (Zurlini et al., 2015; Meuwissen et al., 2019). System transformation (or transition) is complex, long-term, unpredictable, involves many sectors and stakeholders, and entails behavior change (Geels, 2002; Markard et al., 2020). Management of system transformations requires foresight and anticipation, the preparation of many necessary “ingredients,” and the setting-up of “guardrails,” so that when the various cumulative stimuli necessitate a transformation, it is more likely to be a favorable transformation.

Agile tools and methods can play a role supporting such transformations. The main route is by enhanced provision of useful and timely information to decision makers at multiple levels. The secondary route is by stimulating behavior change of researchers. The steps involved in the main route are: an improved and more collaborative data environment (Gorman et al.; Devare et al.; Kruseman); quicker and more efficient data collection (Eitzinger; Chelanga et al.); development of common analytics (e.g., for resilience, Alary et al.; or on poverty reduction; Harris et al.; van Vliet et al.; Marinus et al.); and delivery of actionable information to decision makers. While the whole chain is not evident in any one article, all of the constituents appear within this Research Topic collection. Perhaps Andrade et al. come the closest to describing the complete pipeline in a single application.

Different models for information flow are required for decision makers at the farm-level and at finer jurisdictional levels (e.g., for extension services). Périnelle et al. combine participatory agronomy and agile methods in Burkina Faso; Teeken et al. explore social differentiation factors for variety trait selection in Nigeria; and Gotor et al. report on citizen science variety testing in India. Milner et al. apply a spatial clustering technique to account for contextual drivers of dietary diversity in Southern Kenya; Caulfield et al. explore demographic factors and off-farm work to shed light on livelihood dynamics in the northern Andes; and MacLaren et al. explore how demographic features and assets influence decisions to diversify farms in Nigeria and Kenya. These studies all take account of local context to target specific interventions toward specific groups of people for greater impact and efficiency.

As these efforts to finesse development programming mature and the information flows become more routinely used, we should start to monitor the impacts on decision making. It will be important to show how information improves decision-making and contributes to system transformation.

This Research Topic collection provides a robust foundation to support future development of agile research tools to cut through the excess of data and deliver timely and actionable information. This is demonstrated by the emergence of common practices between many of the methods presented within this collection. We distill a list of features which agile research tools contain in differing combinations:

Light-weight compared to traditional alternatives.

We expect that over time, experimentation with agile data-oriented research tools will provide more insights in the relative contribution of the different features to decision making and system transformation. Future studies should document their impact on the quality, timeliness, and granularity of decisions affecting system transformation.

JH wrote the initial draft of the editorial and proposed the Research Topic. All authors contributed to the manuscript and approved it. All guest editors contributed to the conceptualization of the topic and the soliciting and editing of manuscript.