AgTC and AgETL: open-source tools to enhance data collection and management for plant science research - Vargas-Rojas, Luis; Ting, To-Chia; Rainey, Katherine M.; Reynolds, Matthew; Wang, Diane R. (Frontiers in Plant Science, 2024)
Advancements in phenotyping technologies have enabled plant science researchers to gather large volumes of information from experiments, especially those evaluating multiple genotypes. To fully leverage these complex and often heterogeneous data (i.e. those with differing formats and structure), scientists must invest considerable time in data processing, and data management has emerged as a considerable barrier for downstream application. Here, we propose a pipeline to enhance data collection, processing, and management from plant science studies comprising of two newly developed open-source programs. The first, called AgTC, is a series of programming functions that creates comma-separated values file templates to collect data in a standard format using either a labbased computer or a mobile device. The second series of functions, AgETL, executes steps for an Extract-Transform-Load (ETL) data integration process where data are extracted from heterogeneously formatted files, transformed to meet standard criteria, and loaded into a database.There, data are stored and can be accessed for various data analysis-related processes, including dynamic data visualization through web-based tools. Both AgTC and AgETL are flexible for application across plant science experiments without programming knowledge on the part of the domain scientist, and their functions are executed on Jupyter Notebook, a browser-based interactive development environment. Additionally, all parameters are easily customized from central configuration files written in the human-readable YAML format. Using three experiments from research laboratories in university and non-government organization (NGO) settings as test cases, we demonstrate the utility of AgTC and AgETL to streamline critical steps from data collection to analysis in the plant sciences.