As demand for public health resources often outweighs supply, public health officials and practitioners are increasingly called upon to measure and demonstrate the impact, including value-for-money, of interventions (1). Funders increasingly seek optimization between the financial inputs they provide and the health outcomes that are achieved as a result of funding (2–4). However, quantifying value-for-money of investments in health has remained a challenging endeavor, partly due to the sheer scale of the impacts that can arise from health interventions (5, 6). Traditional economic evaluation methods used in public health such as cost-effectiveness analysis (CEA) or cost benefit analysis (CBA) often fall short in various ways, whether it be their suitability, implementation challenges, lack of comprehensiveness, or other factors (7, 8).

Social return on investment (SROI) analysis is a methodology that has been advanced as a more pragmatic and holistic framework to assess impact of interventions that measures the value generated by an intervention by considering its broader impact on all stakeholders in the local area in which the intervention is implemented. However, its application to public health programs is not yet mainstream (1, 5, 9, 10). SROI analysis is a process for understanding, measuring, and reporting on the social, environmental, and economic value created by an organization, program, or policy (11). It is a mixed methods approach employing both quantitative and qualitative techniques, with an emphasis on stakeholder engagement throughout the process. Compared with CEA which typically measures the value of improved health resulting from an intervention in natural units (e.g., lives saved or cases averted), SROI measures the value of the range of outcomes which result from an intervention, including health, social, economic and environmental outcomes as relevant, and in monetary terms (9). And while CBA also tends to consider a wider range of outcomes and in monetary terms, outcomes in CBA have typically been identified from the perspective of the government or analyst, whereas in SROI outcomes are identified by key stakeholders who are affected by or affect the intervention (9). Furthermore, CBA often lists any benefits that cannot be easily monetized and explains why so, whereas SROI analyses usually deliberately attempt to leverage financial proxies to estimate monetary value of outcomes that cannot be easily monetized (e.g., empowerment, self-confidence) to the extent feasible in SROI’s attempt to measure and account for the full value of an intervention (9). SROI is based on a set of principles, articulated by Social Value International (SVI) (the standard bearer for SROI methodology and assurance), which underlie how SROI should be applied (11, 12).

Similar to CBA, a key result of SROI analysis is the determination of the SROI ratio, which compares the value of the benefits with the investment. For example, an SROI of 3:1 indicates that for every $1 invested in an intervention, it delivers $3 in value.

This article presents the SROI methodology as applied to a public health program and lessons learned from this work, using a mobile pre-school vision program as an illustrative example. Importantly, the SROI analysis process, which can be applied to any public health program, not only produces a measure of impact, but also key insights with respect to strengths of and opportunities for program implementation. The broader implications of this approach to further impact measurement in public health monitoring, evaluation and implementation efforts are explored in the discussion section.

An evaluative retrospective social return on investment (SROI) analysis was conducted to evaluate the broader impacts of the Pre-School Vision Program implemented by the University of California Los Angeles Mobile Eye Clinic (UMEC) over a five-year period. UMEC is an intervention which visits various Southern California community locations annually to deliver free basic vision care (including glasses and referrals to specialists as needed) to pre-school aged children in underserved communities across Los Angeles County. Between 2012 and 2017, UMEC completed vision screenings for 90,000 preschool-aged children with funding support from First 5 LA.

Consistent with SVI guidelines, we followed the six stages of SROI analysis: establishing scope and identifying key stakeholders, mapping outcomes, evidencing outcomes and giving them a value, establishing impact, calculating the SROI ratio from the perspectives of key stakeholders, and reporting (see Table 1) (11–13).

The evaluation found the SROI ratio of this program to be approximately $40:$1, that is, for every $1 invested in the program, approximately $40 in social value was created in the community. Results of the sensitivity analyses conducted suggest that, at the very least the program would have an SROI of approximately $19:$1 and at most an SROI of $59:$1 (see Table 4). The SROI ratio and sensitivity range strongly indicate the value of UMEC exceeds the investment. Of the total impact, the most value was created for pre-school children who were screened, diagnosed as needing glasses, and who received glasses (78%), followed by parents/guardians (19%), followed by schoolteachers (3%) and by other key stakeholders (3%). This distribution of impact is consistent with the primary beneficiaries of the intervention.

For pre-school children who received glasses from UMEC, 16 outcomes were identified, of which 10 were monetized (the remaining 6 outcomes were not included in the value calculation as it was unclear whether the data collected on those outcomes was reliable). During survey interviews with parents, the following outcomes experienced by children were cited most frequently: improved quality of life, increased attentiveness and/or performance in school, and improved confidence and/or self-esteem (see Table 2). Of all the outcomes experienced by the children that were monetized, the three that generated the most value were (a) improved quality of life, (b) increased confidence and/or self-esteem, and (c) less dependency on others. There were several additional outcomes which generated positive value (refer to Table 3). To note, there was also an outcome experienced by the children—increased bullying or teasing (as a result of wearing glasses) – which generated negative value. It is also worth noting that pre-school children who were screened but did not require nor receive glasses also experienced a benefit from the intervention in terms of knowing their health status, though this benefit was not included in the SROI valuation.

For parents/guardians, all outcomes experienced were of a positive value. These outcomes included: (a) parents received benefit of knowledge and feeling of relief of knowing their child’s eye health status due to screening received, (b) parents obtained affordable eye health screening for their child, and (c) increased awareness of eye health information. All of these outcomes were monetized.

For schoolteachers, the key outcomes experienced were: (a) increased awareness of eye health information, (b) increased job satisfaction, and, on the negative side, (c) a disruption to school activities due to the UMEC screenings. All of these were monetized.

For school nurses and family service advocates, one key outcome was experienced: increased job satisfaction, and this was monetized. Of note, school teachers, school nurses and family advocates stated that they felt their job satisfaction increased given the impact they have not only on overall health and development, or scholastic improvement of the child, but also specifically on the child’s eye health as a result of supporting children to be screened and reminding them to wear glasses.

For UMEC staff, two outcomes of program involvement were identified: (a) increased job satisfaction and (b) helping to feel part of the Los Angeles community. The first outcome was not monetized as the activities performed were a central part of their jobs so it was considered that effectively their pay reflects this value and income of implementing staff members is not considered an outcome in SVI methodology. With respect to the latter outcome, one staff member described having moved to Los Angeles recently from another city and that the program helped him feel more connected with his new community. This outcome was monetized.

For EyeDeal Optical staff, one key outcome was experienced: increased job satisfaction. The co-owner of the company shared that participation in this program motivated him to start a similar non-profit organization in his home country of Guatemala to support children’s eye health. This outcome was not monetized as it would have required additional research with Eyedeal Optical staff that was out of scope of this analysis.

Results suggest a very positive impact of the program, with benefits generated for various key stakeholders including pre-school children, parents/guardians, schoolteachers, school nurses and family service advocates, EyeDeal Optical, and UMEC staff. The sensitivity range of the SROI ratio determined in this study is consistent with broader findings on return on investment studies of early childhood and public health interventions, and that an ounce of prevention is worth a pound of cure (1). While $40:$1 may seem relatively high, results are consistent with previous studies which have generally found vision care for preschoolers to be a beneficial and cost-effective investment, including one study which found a benefit-to-cost ratio of visual acuity screening among preschoolers as high as $162:$1 (18, 26, 27). That data was collected from a representative sample size and that SROI results remained robustly positive under sensitivity analysis testing in this study contribute to the quality of the study and its results. As the sensitivity analysis range explicitly acknowledges the inherent uncertainties and judgments involved in an SROI calculation, and that in this case, the low end of the relatively wide range was still highly positive ($19:$1), this range provides essential context for informed decision-making. In addition, evaluators employed a conservative approach to valuation, for example, in only including those material outcomes for which reliable data was collected and in only accounting for outcomes as lasting for 1 year (even though it is very likely their benefits accrue for longer than a year, and in some cases over the life span), both of which demonstrate the application of the SROI principle to not over claim results in this study.

Focus groups and survey interviews highlighted strengths of program implementation including removing barriers to access to care through convenience of the school location, timings and free services and glasses. In addition, parents and school staff cited the benefits of increased health awareness not only for the children, but for the parents and siblings as well. Focus groups also highlighted some recommendations to improve the program implementation including the ability for children to choose different styles of glasses which could enhance self-image and improve compliance, particularly for children who experienced bullying. Bullying has been documented in other similar interventions and key strategies emphasize the importance of educating parents and teachers on the importance of wearing glasses and their critical role in encouraging children to wear them (28–31). Additional opportunities identified during focus groups for future implementation included increased frequency of screenings and hours for parents to interact with UMEC, offering additional replacements beyond the one-time glasses replacement, and faster turnaround time between screenings and glasses delivery. In addition, focus groups revealed a clear need to collaborate with local pediatricians on eye care for this population to improve understanding of eye health, diagnoses, and adherence to wearing glasses for children. Focus groups also identified an opportunity for future research with respect to the prevalence of misdiagnosis of ADHD or other behavioral diagnoses due to undiagnosed vision impairment, as experienced by some of the children. In summary, this study demonstrated that investments in the UMEC Pre-School Vision Program are valuable for various stakeholders in many ways, and that the developmental and educational benefits set children up for long term success and increase their quality of life both immediately and throughout the lifespan.

Beyond evaluation of the program itself, several broader lessons were learned as a result of applying the SROI methodology to this public health intervention. Compared to standard economic evaluation approaches, using an SROI approach presented key advantages. First, unlike traditional return on investment analysis, this research took an inductive approach. It did not test hypotheses formulated ex-ante, but involved stakeholders in identifying the changes they experienced, and the outcomes (both positive and negative, intended and unintended) which ought to be included in the analysis. This allowed the incorporation of those outcomes which were deemed most important to stakeholders themselves, as opposed to pre-defining what ought to be important from the standpoint of others such as program donors or managers. Secondly, the SROI analysis did not incorporate solely what was easy to quantify and value (so-called ‘hard’ economic or health impacts). Rather, it also considered key ‘soft’ impacts which have traditionally been excluded from return on investment analysis (e.g., confidence and social developments of children). Critically, this study found that ‘softer’ (or less tangible) impacts were extremely valuable to stakeholders, often more so than ‘hard’ ones. Of course, SROI methodology also has its challenges. Similar to cost–benefit analysis, the lack of standard financial proxies for outcomes is one of the biggest challenges in SROI (32, 33). Furthermore, depending on the methods used, there may be a high degree of subjectivity with estimation of deadweight and attribution (34, 35).

As suggested by this study, while not without its limitations, SROI analysis can be a useful methodology to practitioners, donors, researchers, and policy-makers who are interested in holistic measurement of impact, including value-for-money, of public health interventions. This study is, to authors knowledge, one of a handful to apply the social return on investment methodology to vision health, mobile health and child health interventions. This study demonstrates that through the direct involvement of stakeholders throughout the SROI process, a better quality of analysis can be achieved compared to traditional methods. Ultimately, the goal of program evaluation is to understand the strengths and weaknesses of the program of interest and provide recommendations to improve the program in order to better serve communities. Traditional economic evaluation methods in public health rely heavily on the insights provided by those conducting the analysis – which extends to the strengths and opportunities of the program, potentially missing key opportunities. Through the heavy involvement of stakeholders in the SROI process, direct feedback from these groups can be collected – in turn, informing the final analysis, results, and recommendations. Such indicators and insights are often missed by traditional economic approaches used in the public health field. If ultimately, what gets measured, gets valued, it is of prime importance to ensure we are measuring all that truly matters to all of those who are affected by an intervention to prove and improve the value of public health interventions.