The ubiquitous biogenic amines octopamine (OA), serotonin (5-HT) and dopamine (DA) activate neural circuitry to regulate behavior (Libersat and Pflueger, 2004; Bergan, 2015). The phylogenetic distribution of these neuromodulators suggests a deep evolutionary history predating the origin of the nervous system (Gallo et al., 2016). With few structural modifications, monoamines are functionally diverse in insects (Roeder, 1999; Mustard et al., 2005; Blenau and Thamm, 2011). Conserved aminergic circuits (Kravitz and Huber, 2003; Barron et al., 2010; Perry et al., 2016) and patterns of receptor expression (Roeder, 1999; Blenau and Thamm, 2011) control behavior in diverse species across insect orders. However, how monoamine neurotransmitter systems served as preadaptations for the evolution of derived behaviors associated with the transition from solitary life to sociality in insects is poorly understood. Insect colonies show remarkable variation in structure and degree of integration of worker actions that could underscore complex social behavior. Using well-resolved insect molecular phylogenies (Wiegmann et al., 2011; Song et al., 2012, 2015; Moreau and Bell, 2013; Regier et al., 2013; Schmidt, 2013; Misof et al., 2014; Wang et al., 2014), we explore the evolution of neuromodulation of social behavior (Supplementary Table S1) by analyzing patterns of monoamine function in solitary and social taxa (Figure 1; Supplementary Table S2).

Two neural circuits regulate vertebrate decision-making: the social behavior network, controlled by neuropeptides and gonadal steroids, and the mesolimbic reward system, activated primarily by DA (O’Connell and Hofmann, 2011a,b). These circuits act in concert to regulate social interactions and evaluate stimulus valence, respectively, forming the social decision-making network (O’Connell and Hofmann, 2011b, 2012). Insect social decision-making systems are poorly understood in comparison, although behavioral influences of neuromodulators are well known (Supplementary Table S2).

Neurochemical and neuroendocrine analyses of complex social behavior in insects have largely been limited to the species-rich Hymenoptera (>150,000 species), which includes ants, bees and wasps with solitary, presocial and eusocial life histories. Solitary species are composed of individuals that live and forage alone and interact with conspecifics primarily during mating or territorial disputes. Presocial describes life histories that are intermediate between solitary and eusocial (Eickwort, 1981). Eusociality is defined by: (1) reproductive division of labor (the differentiation of fertile [queens and males] and sterile [workers] castes); (2) allomaternal care (cooperative care of immatures by workers); and (3) overlapping generations of reproductive and worker castes (queen longevity allowing coexistence with offspring). Varying degrees of sociality are found in a number of clades. Phase transitions from solitary to gregarious behavior occur in desert locusts (Order Orthoptera; Anstey et al., 2009; Ott and Rogers, 2010), beetles (Order Coleoptera) show multiple occurrences of the evolution of familial sociality, including biparental care (Costa, 2006; Cunningham et al., 2015; Panaitof et al., 2016), and one species of weevil is eusocial (Kent and Simpson, 1992). Solitary life histories predated eusociality in both the Hymenoptera (Wilson, 1971) and Isoptera, which diverged from cockroaches into entirely eusocial forms (Bourguignon et al., 2015). The evolution of a reproductive caste occurred once in ants and multiply in bees and wasps; diversification of workers, particularly in ants, has many independent origins (Oster and Wilson, 1978; Trible and Kronauer, 2017). Eusociality independently evolved in the Order Isoptera (termites, >3000 species; Thorne and Traniello, 2003).

Parental behavior, reproductive competition and foraging and defense strategies in solitary (Field et al., 2006, 2015; Thompson et al., 2014) and eusocial (Tibbetts, 2013) hymenopteran species reflect social decision-making, although neurochemical and neuroanatomical correlates of such systems are poorly understood (Ilies et al., 2015). For example, neural mechanisms underscoring vertebrate-like cognitive abilities, such as individual facial feature recognition in some eusocial wasps, are not known (Gronenberg et al., 2008; Sheehan and Tibbetts, 2011). Decision-making at the colony level is seen in collective (swarm) intelligence (Seeley, 2010; Jeanson et al., 2012; Sasaki and Pratt, 2012; Reid et al., 2015) and in part concerns worker interactions (Greene and Gordon, 2003; Greene et al., 2013) that may be causally related to brain neurotransmitter levels (Muscedere et al., 2012; Kamhi and Traniello, 2013; Kamhi et al., 2015; Hoover et al., 2016). Studies have focused on the aminergic control of worker interactions that contribute to social organization, including responsiveness to social signals and cues that regulate alloparental care, food exchange, nest construction, defensive behavior and foraging (reviewed in Kamhi and Traniello, 2013; Simpson and Stevenson, 2015; Hamilton et al., 2017). Studies have begun to explore genetic and epigenetic underpinnings of task performance and plasticity through state changes in behavior (Lucas and Sokolowski, 2009; Simola et al., 2016) that may involve neuromodulators.

Genes controlling behavior in solitary insects regulate social behavior in eusocial species (LeBoeuf et al., 2013) and affect sensory receptor evolution (Baldwin et al., 2014). Monoamine functions in solitary insects likely reflect this conservation, and appear to have been preadaptive for eusocialty. To understand the evolution of neuromodulatory systems in insects, we organized available data on aminergic control into eight behavioral categories: activity, aggression, development, higher-order sensory integration, nutrition, reproduction, sensorimotor functions and social functions (defined in Supplementary Table S1). Behaviors may span multiple categories, such as parental care and mate selection involving reproduction and derived social functions. Statistical tests showed similar patterns of monoamine function in solitary and eusocial species (Supplementary Figure S1), although small sample sizes constrain inferences. While data on biogenic amine regulation is variable and fragmentary, some patterns emerge suggesting that aminergic circuitry has shifted in function during the transition from solitary to social life. Monoamines have been co-opted for social functions through receptor and circuitry evolution and have gained novel functions to regulate social behaviors. For example, 5-HT (Alekseyenko et al., 2010, 2014; Bubak et al., 2014) and OA (Stevenson et al., 2000; Hoyer et al., 2008; Zhou et al., 2008; Stevenson and Rillich, 2017) increase aggression in solitary insects. In social insects, aggression is associated with the ability to pheromonally distinguish nestmates from non-nestmates (Stroeymeyt et al., 2010; Sturgis and Gordon, 2012), and OA is implicated in improved nestmate recognition (Robinson et al., 1999; Vander Meer et al., 2008; Kamhi et al., 2015). OA may thus enhance sensitivity to pheromonal cues and regulate social interactions similarly in both solitary and social insects.

DA, 5-HT and OA are involved in regulating metamorphosis in solitary insects (Nässel and Laxmyr, 1983; Hirashima et al., 1999). In social insects, monoamines are associated with age-related behavioral changes and collateral physiological and neural development (Schulz et al., 2002; Seid and Traniello, 2005; Cuvillier-Hot and Lenoir, 2006; Wnuk et al., 2010; Giraldo et al., 2016). OA increases with age and is causally related to the transition from nursing to foraging in honey bees (Schulz et al., 2002). In ants, 5-HT, DA (Seid and Traniello, 2005; Cuvillier-Hot and Lenoir, 2006), and OA (Wnuk et al., 2010) increase with age; 5-HT, similar to OA in bees, is correlated with age-related initiation of foraging (Seid and Traniello, 2005) and sensitivity to pheromonal signals underscoring trail communication (Muscedere et al., 2012).

In respect to other behaviors, suppressing DA neurons in Drosophila melanogaster consistently inhibits aversive but not appetitive learning, whereas manipulating OA action produces the opposite pattern (Schwaerzel et al., 2003; Claridge-Chang et al., 2009; Aso et al., 2010). Similar patterns have been found in honey bees (Mercer and Menzel, 1982; Hammer and Menzel, 1998). However, appetitive learning in social insects must be considered in respect to the social context, where foraging is dependent on the nutritional state of the colony rather than the individual (Traniello, 1977; Seeley, 1989). OA increases the likelihood of successful foragers waggle dancing, which communicates information about food location and quality to nestmates; this demonstrates that an amine may be adapted to serve a colony-level function in food collection rather than benefit individual nutrition (Barron et al., 2007).

Biogenic amines appear to have gained new functions associated with the regulation of social organization. DA correlates with increased receptivity and mating in solitary insects (Pastor et al., 1991; Neckameyer, 1998; Chvalova et al., 2014; Brent et al., 2016), and reproductive state in many hymenopterans (e.g., Sasaki et al., 2007). Honey bee and some ant workers are reproductively capable; however, both ant and honey bee queens release a pheromone, queen mandibular pheromone (QMP), that inhibits worker reproduction (Fletcher and Blum, 1981; Hoover et al., 2003) by acting through DA circuitry (Harris and Woodring, 1995; Boulay et al., 2001; Beggs et al., 2007). Aggressive interactions between workers to control reproductive dominance also affect DA levels (Shimoji et al., 2017). These studies suggest that in both solitary and eusocial insects DA regulates reproductive state, and DA additionally may be integral to the maintenance of reproductive division of labor and the resolution of reproductive competition in eusocial species.

We identify four research areas, among several others, that are significant in the study of the neuromodulation of complex eusocial behavior.

Broader sampling is required to gain phylogenetic insight into the evolution of aminergic control systems. Determining patterns of conservation and/or diversification of aminergic regulatory mechanisms of social behavior will benefit from studies of insect genera that include solitary and eusocial species. Despite the widespread activity of biogenic amines, functional patterns appear. 5-HT may control energy expenditure through feeding behavior and circadian rhythms, DA regulates fertility, thus modulating task performance in eusocial species, and OA modulates appetitive learning associated with feeding and nestmate recognition. Advances in epigenetics (Libbrecht et al., 2016), neurogenetics (Friedman and Gordon, 2016), and the integration of sociobiology and neurochemistry (Kamhi and Traniello, 2013) will aid in future research.

JFK, SA and JFAT compiled literature. SA performed statistical analyses and created associated figures, and CSM created the phylogenetic presentation of aminergic control of behavior. JFK, SA and JFAT prepared drafts of the manuscript. All authors contributed to the conception of the perspective, analysis and synthesis of material, manuscript revision and gave final approval for publication.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer VM declared a past collaboration with one of the authors CSM to the handling Editor.