All the major NGOs working in the Sham Shui Po district were invited to a briefing session about the EFWB project, organized by the Sham Shui Po District Social Welfare Office and the academic partners from the Schools of Public Health and Nursing in December 2011. Thirty NGOs participated and submitted a brief proposal of their programs to the research team before the TTT. Agency staff from these organizations were assigned by their supervisors to participate in the TTT (free of charge) and were given feedback to improve and finalize their proposal to be completed within 2 weeks after the training. Financial support from the FAMILY project to run the programs was given to each participating NGO once its proposal was approved. The community-based interventions started about 1–5 months after the TTT. Most of the organizations had service agencies in this district serving particular client groups, such as low-income groups and new immigrants.

Figure 1 shows how our TTT was changed from the TTT of the HFK Project. First, the length of the training was reduced from 2 days (12 h) to 1 day (7 h), by eliminating contents on healthy cooking and nutrition that were not the focus of the EFWB Project. There is evidence that brief trainings are effective and cost effective (27–29). Brief training limits the burden on the participants and enhances acceptability. Second, the current TTT was disseminated to a larger number of community agency staff, which could have greater sustainability and community impact. Third, the current TTT gave a brief introduction of the general concept and its application to research and evaluation methods, including the Logic Model (30, 31), randomized controlled trials (RCTs), and process evaluation (32), which had not been emphasized and evaluated in previous training (see below for more detail). It helped push the boundaries of community practitioners’ skill sets, as their background and training was oriented toward service delivery rather than theory and research. Understanding the research and evaluation methods was needed to plan, deliver, and evaluate the community intervention program, such as the EFWB Project that involved various community agencies and diverse service targets working under a large project with the same similar theoretical base and outcomes, and required a certain degree of flexibility of the interventions among different agencies for different participants. Fourth, the current evaluation extended the Integrated Model of Training Evaluation and Effectiveness (IMTEE) framework (3) and included more quantitative variables (e.g., the application of the acquired skills beyond the current project) to reflect transfer performance. This evaluative information indicates persistence of effects and is a key to improving the scope and reach of the training. Finally, the evaluation of the current TTT included more focus groups than the previous TTT, and groups were formed by agency staff in the same arm, in order to strengthen the qualitative evaluation.

The training content was designed by a multidisciplinary team including academics and community partners basing on the “Knowledge, Attitude and Practice (KAP)” model (33) and was organized into two sessions (3 and 4 h per session) delivered on the same day. The academics partners took the lead for theoretical and research design components of the training, and the community partners ensured the acceptability of the training contents and led the component of positive psychology training.

Session I was delivered by a clinical psychologist with experience in teaching positive psychology in the social service sector. This session taught the general concept, applications, and theme-related targeted behaviors of three specific positive psychology themes designated by the EFWB Project, including “Gratitude” (34), “Hope” (35), or “Open-mindedness” (34). The main purpose was to introduce theme-related experiential activities as an example of promoting targeted behaviors in the interventions. Each community intervention implemented by the agency staff was required to be based on one of these three themes. Given different groups of clients served and potential participants to be recruited by the agencies, flexibility of choice of theme allowed the autonomy to choose the theme that was most appropriate. Details on theme selection and theme-related targeted behaviors have been described elsewhere (36).

Session II was delivered by academics in public health (Author THL) and nursing (Author SC) who were key investigators in the FAMILY Project. This session focused on the general concepts of program design, implementation, and evaluation. First, we introduced the Logic Model with a pictorial description. The Logic Model is as an operational plan that uses narrative or graphical depictions to illustrate a sequence of cause-and-effect relationships, i.e., how specific inputs and activities are linked and will accomplish intended short-, intermediate-, and long-term outcomes (30, 31). The application of the Logic Model allows precise communication about the purpose of a project, the components of a project, and the sequence of activities and accomplishments. In the current TTT, we taught agency staff key concepts of the Logic Model and provided a practical example of its application in program design. With the application of the Logic Model, we intended that all activities and interventions from the different NGOs were linked with the same objectives, core messages, and outcomes (i.e., increase in attitude, intention and practice of targeted behaviors devised from positive psychology, and improved family well-being).

Second, we explained the rationale and principle of implementing RCT, and the responsibilities of the interventionists in an RCT, in an attempt to enhance agency staff’s acceptance of the RCT design of our EFWB Project and its limitations. An RCT is a type of scientific experiment, which aims to reduce confounding and bias when testing a new intervention, and the result is classified as Level I evidence (37). However, community practitioners’ and clients’ acceptance of RCT has been found to be low because no-treatment control groups are not seen as a valid utilization of resources but appear to be deceptive to clients in community-based interventions (38, 39). In the EFWB Project, as individual randomization was not feasible, a two-arm cluster-RCT was used to assess the effectiveness of the intervention program plus a theory-based booklet (Arm A) compared to the intervention program without the booklet (Arm B).

Last, we introduced the concept of process evaluation and its application, so as to teach agency staff to evaluate the process of their own intervention programs. Process evaluation is an assessment of program process, to understand when, where, how, and in which context the intervention/program was implemented, and to what extent the intervention/program was implemented as planned (32). Process evaluation included six domains: fidelity (the extent to which intervention was implemented as planned), dose delivered (the amount of intended units of each intervention delivered by interventionists), dose received (the extent of engagement of participants in the program and the participants’ satisfaction with program), reach (the proportion of the intended priority audience that participated in the intervention), recruitment (the procedures used to approach and attract participants), and context (the environmental issues that might affect the intervention implementation or study outcomes) (32). Process evaluation is an important supplement to outcome evaluation as it elucidates reasons for success or failure of the intervention and informs replication and improvement of the intervention. The implementation of process evaluation is increasing (32). Some programs have successfully involved community practitioners in the process evaluation (40). Given the large number of interventions that were delivered in our EFWB Project, there was an important economic advantage to train the agency staff to conduct process evaluations on their own programs. This strategy lowered the overall costs of the project by reducing dependence on outside evaluators.

Based on the principles of adult learning (41, 42), diversified learning methods were used, including didactic instruction, games, role plays, and group discussion (Table 1). We prepared a graphic diagram for practicing the application of the Logic Model, written guidelines for the logistic arrangement and program implementation, and checklists for process and outcome evaluation. We also provided post-training support for the agency staff for designing, drafting the proposal for funding, and implementation (for example, program promotion materials, souvenirs, and workbooks for the participants), and a hotline for consultation and seeking technical assistance, if needed.

The surveys for TTT evaluation were designed by the academic partner and discussed with the multidisciplinary team to ensure that the components of the evaluation model be appropriately included. The baseline survey (T1) was pilot tested on five agency staff in the multidisciplinary research team for ease of understanding and redundancy. The community partners in the research team were responsible for data collection. Agency staff who attended the training session were asked to complete the self-administered questionnaire before (T1) and immediately after (T2) the training. Participants were asked to complete the follow-up questionnaire 6 months (T3) and 12 months (T4) after the training. Participants who did not complete T3 or T4 were reminded by at least two emails or phone calls. The main reasons for loss to follow-up at T3 or T4 were participants had changed jobs, they were too busy at work, or they were on vacation at time of assessments.

Semi-structured focus groups with agency staff who participated in the EFWB projects were arranged by the community partner, upon the completion of the community-based intervention (December 2012). Each NGO was asked to recruit one to two agency staff for the qualitative component of the study. Homogenous segmentation was applied to control the group composition. Groups were formed according to their RCT arm (control vs intervention) and positive psychology theme (gratitude vs hope/open-mindedness), with five to seven agency staff in each group. The academic partner designed the interview guide and conducted the interview, with one moderator and two note-taking observers in the interview panel. Each focus group opened with a short session to explain the purpose of the study and assure the participants of the confidentiality of the discussion. Focus groups were conducted in Cantonese, audiotape-recorded, in a community agency center, and lasted about 1 h. Tapes were transcribed verbatim after the interviews. Qualitative data were generated from both transcripts and notes.

Ethical approval for the study was obtained from the Institutional Review Board of the University of Hong Kong/Hospital Authority Hong Kong West Cluster. Participants in the surveys and focus groups were assured that all information would be kept anonymous and confidential and was gathered for research purposes only. Written consent was sought from each participant before the assessments.

The IMTEE combines variables of training evaluation and effectiveness for comprehensive assessment (3) and was designed to evaluate training in the business world. Guided by the IMTEE, our TTT was evaluated on three dimensions, including reactions to the training content and design, changes in the learner, and organization payoffs.

The principle of intention-to-treat (ITT) analysis was adopted. Missing data of participants who withdrew from the study, were lost to follow-up, declined to complete or did not finish the questionnaire, were replaced by their baseline (T1) values. MANOVA analyses were performed initially to detect the time effect on the outcome variables, followed by post hoc paired t-tests comparing T2, T3, and T4 to baseline scores. A p value of less than 0.05 was considered statistically significant. Effect size (Cohen’s d) was computed. A positive effect size indicated an increase in the mean score of the outcome, while a negative effect size indicated a decrease. An effect size of 0.2 to <0.5 was considered as small, 0.5 to <0.8 as medium, and 0.8 or above as large (43). Sensitivity analysis was performed by using per protocol (or complete case) analysis, which included participants who completed all assessments (T1–T4) and excluded those with missing data. Results from ITT had the same level of statistical significance and similar effect size as the corresponding per protocol analysis results (Tables S1 and S2 in Supplementary Material). Therefore, to avoid exaggerating the positive effects of the TTT, results from the more conservative ITT method are mainly presented in this paper. All analyses were performed using SPSS 20.0. The qualitative data were analyzed by thematic content analyses, with themes corresponding to three dimensions of the IMTEE. Coding was conducted by two research team members and consensus achieved in the case of discrepancy, to maximize reliability. The quantitative and qualitative results were finally merged to compare, interrelate, and validate results. We used the mixed methods triangulation design (44), which places equal emphasis on quantitative and qualitative data in interpretation.

Ninety-three agency staff attended the TTT and 90 completed the baseline survey. Three agency staff declined to participate because of their busy time schedule. Eighty-eight, 63, and 57 agency staff completed the T2, T3, and T4 surveys (follow-up rate: 98%, 70%, and 63%), respectively. The majority were women (80%), aged 25–44 years (71%), had a tertiary degree or above (66%), were registered social service staff (78%), and targeted families in their service (63%). On average, the participants had experience in social service for 10.5 years and been employed in their current organization for 8 years. Their training in positive psychology and research methods prior to the current TTT was perceived as insufficient (4.63 and 3.79 out of 10, respectively). They were involved in the community-based intervention programs of the EFWB Project as program content designers (69%), interventionists (63%), and intervention supporting staff (53%). Some of them had multiple roles in the project (Table 2).

Twenty-three agency staff participated in the focus groups. The majority were women (73%), aged 35 years and above (45%), registered social service staff (83%), and had tertiary education (61%). They were mainly responsible for program design (70%) and implementation (83%) in the EFWB Project.

On a scale with a maximum of 5, satisfaction on the contents and design was high, with all scores at T2 being over 3.8. The contents were considered as sufficient (4.18), inspiring (4.03), and applicable (4.15). Participants were satisfied with the design (3.91), brief format (3.80), and time management (3.84), and considered the training had achieved the objectives well (4.10) and increased their knowledge and skills (4.15). Overall scores on the quality, practicability, and satisfaction of the training were about 4.0 (Table 3).

The qualitative results corroborated the quantitative finding regarding satisfaction with the TTT. Focus group participants gave positive responses (e.g., “motivating” and “provided a clear direction to the program activities”) to the overall impression of the TTT. Suggestions for improvement were also received from focus groups. Agency staff commented that the training on the research components was not intensive enough and had specific requests for supplementation in specific areas such as proposal writing.

I think the training was very fruitful, as it was impressive … But I needed to digest what I had learned before I could use it with my clients (project support staff, woman, aged 44 years).

The training was good … I liked the training in the Logic Model. It helped to link the behavior indicators with the activity goals. It was useful in our center (project designer and implementer, woman, aged 26 years).

Training from the clinical psychologist was the most useful part … Training from the academic in using the chart (the Logic Model) was essential, because it (the application of the Logic Model in program planning) was different from the simple planning we usually did. I think it would be better to have more training about this model (project implementer, woman, aged 35 years).

I agree that we should learn positive psychology in more depth. Our (past) training in positive psychology has been a bit superficial … The agency staff were the ones implementing the project; it was thus better to provide them with more relevant training and different options (of the themes) to choose from (project implementer, woman, aged 35 years).

We had difficulties writing the project proposal initially. Therefore, a potential way to improve the workshop is by providing more training on proposal writing (project designer and implementer, man, aged 23 years).

MANOVA showed a significant effect of time on knowledge, self-efficacy, and attitudes in relation to the training contents. Significant increases in knowledge of self-efficacy and attitudes toward the training contents (positive psychology, the Logic Model, process evaluation, and RCT) were found immediately after the training (all p < 0.05), with effect sizes for the changes from 0.24 (small) to 1.31 (large) (Table 4). At T3 and T4, the positive effects (increases over baseline) for knowledge of all training contents were sustained (with small to medium effect). Positive changes in attitudes did not persist (p > 0.05), except for the item “the Logic Model can provide direction for program design” (T3: Cohen’s d = 0.27, p = 0.01; T4: Cohen’s d = 0.23, p = 0.04). For the increases of self-efficacy, variables related to the application of positive psychology and the Logic Model persisted at T3 and/or T4 (p < 0.05), while that of the application of process evaluation did not persist (p > 0.05) (Table 4). Per protocol analysis showed similar findings (Table S1 in Supplementary Material), including significance level and effect size as those from ITT.

The focus group participants indicated enhanced knowledge of positive psychology after the TTT. Some agency staff learned to motivate their clients by improving the format and style of conducting community programs. Improvement in knowledge and skills in positive psychology appeared to be the major changes in learners as indicated by both quantitative and qualitative methods.

I had a better understanding of positive psychology after the training. Our organization has many programs to promote family relationships and this workshop gave us ideas to use in those activities (project implementer, woman, aged 35 years).

There were not just talks (in the TTT). It (the training) was delivered in various formats, including music and video. When we did our own intervention, we also remembered to use a variety of formats, in order to motivate and retain our clients (project designer and implementer, woman, aged 33 years).

Table 5 shows significant increases in applying positive psychology in program design (Cohen’s d = 0.36), and encouraging clients to use positive psychology strategies to improve family relationships and well-being (family health, happiness, and health) in the EFWB Project (Cohen’s d ranged from 0.29 to 0.37) at T3 (all p < 0.05). These differences did not persist at T4 (all p > 0.05). For the application of these concepts beyond the EFWB Project, there were no increases and even some decreases at T3 and T4, except the increase in “encouraging clients to use positive psychology to improve family relationships” at T3 (Cohen’s d = 0.24, p = 0.03). Per protocol analysis showed similar findings as but slightly greater effect size (Table S2 in Supplementary Material) than the ITT results presented here. For the application of the Logic Model and process evaluation, scores for the EFWB Project were above 3 out of 5, and scores for beyond the EFWB Project were below 3 (Table 5). As described by Zhou et al. (36), after the TTT, the agency staff in the current study had successfully designed and implemented 29 community-based intervention programs to 1,586 clients. The interventions were effective in enhancing participants’ family 3Hs and family relationships.

Medium levels of intention to apply the skills learned to other projects were found at T3 (scores ranged from 4.1 to 4.7, out of 6) and at T4 (score ranged from 4.0 to 4.8, out of 6). Over 70% (89.7% at T3 and 73.7% at T4), about 50% (51.7% at T3 and 48.2% at T4), and over 55% (55.7% at T3 and 59.6% at T4) of the participants intended to teach their colleagues positive psychology, the Logic Model, and process evaluation, respectively (Table 6).

Corroborating the above quantitative results, most of the focus group participants reported that they had used the intervention and evaluation skills taught and would use them in future projects and recommend them to others. In the focus groups, those who did not intend to further use the strategies they had been taught in routine practice considered them too time consuming for routine activities. Integrating the quantitative and qualitative findings, the agency staff and community agencies’ enhanced capacity to use positive psychology would be the most significant payoffs for the TTT, since research skills were not considered suitable for agencies’ routine service.

We learned relevant knowledge and skills (from the TTT), and used them to serve our clients. In fact this was a very good learning process. We shall teach what we learned to our colleagues because I think the information is applicable not only to our project, but also to other activities conducted by my colleagues (project designer and implementer, woman, aged 33 years).

I will be more likely to use positive psychology than other skills taught (in future activities) (project designer and implementer, woman, aged 30 years).

They (the training contents) will probably not be used in our future activities, because the application (of the strategies we learned) is time and money consuming. The Logic Model could be used when we apply for some external funding … But in our usual activities, there is not adequate time for us to use the Logic Model in program planning (project supervisor, designer and implementer, woman, aged 31 years).