This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on http://publichealth.jmir.org, as well as this copyright and license information must be included.
Small-scale community health worker (CHW) programs provide basic health services and strengthen health systems in resource-poor settings. This paper focuses on improving CHW performance by providing individual feedback to CHWs working with an mHealth program to address malnutrition in children younger than 5 years.
The paper aims to evaluate the immediate and retention effects of providing performance feedback and supportive supervision on CHW motivation and performance for CHWs working with an mHealth platform to reduce malnutrition in five districts of Madhya Pradesh, India. We expected a positive impact on CHW performance for the indicator they received feedback on. Performance on indicators the CHW did not receive feedback on was not expected to change.
In a randomized controlled trial, 60 CHWs were randomized into three treatment groups based on overall baseline performance ranks to achieve balanced treatment groups. Data for each treatment indicator were analyzed with the other two treatments acting as the control. In total, 10 CHWs were lost to follow-up. There were three performance indicators: case activity, form submissions, and duration of counseling. Each group received weekly calls to provide performance targets and discuss their performance on the specific indicator they were allocated to as well as any challenges or technical issues faced during the week for a 6-week period. Data were collected for a further 4 weeks to assess intertemporal sustained effects of the intervention.
We found positive and significant impacts on duration of counseling, whereas case activity and number of form submissions did not show significant improvements as a result of the intervention. We found a moderate to large effect (Glass’s delta=0.97,
Calls providing performance feedback are effective in improving CHW motivation and performance. Providing feedback had a positive effect on performance in the case of duration of counseling. The results suggest that difficulty in achieving the performance target can affect results of performance feedback. Regardless of the performance information disclosed, calls can improve performance due to elements of supportive supervision included in the calls encouraging CHW motivation.
The success of small-scale community health worker (CHW) programs in providing basic health services and strengthening health systems in resource-poor settings is well documented in the literature [
Health worker motivation is identified as an important determinant of performance by many studies [
Supportive supervision targeting health workers’ knowledge and skills, motivation, and adherence to correct practices provide incentives that positively impact performance [
In this study, we focus on improving CHW performance by providing individual feedback on specific indicators to CHWs working in an international nongovernmental organization-run mHealth program to address malnutrition in children younger than 5 years. We open two-way communication channels by placing weekly calls to CHWs to provide feedback on their performance and discuss any challenges they are facing in their work. The aim of the intervention is to gain improvement in the performance indicators, through improvement in CHW motivation, knowledge, skills, and adherence to reporting standards. With improved ability to monitor performance and provide targeted support to CHWs, we hope to improve their motivation and performance.
Many theories of motivation distinguish between intrinsic and extrinsic work motivation. Intrinsic motivation involves people deriving satisfaction from doing the activity because they find the activity itself rewarding [
In most work contexts, both extrinsic and intrinsic motivations come into play and affect worker performance. Although early research supported the additivity of intrinsic and extrinsic motivation, more recent work has also highlighted the tradeoffs between the two, with external rewards designed to enhance external motivation detracting from internal motivation [
The complex relationships between intrinsic and extrinsic motivation is best explained by self-determination theory (SDT), which distinguishes between autonomous motivation, analogous to intrinsic motivation, and controlled motivation, analogous to extrinsic motivation. Any behavior falls somewhere on the continuum of controlled to autonomous motivation [
SDT postulates that increased autonomy, competence, and relatedness are the three factors or “needs” that underlie intrinsic motivation and people need to feel autonomous, competent, and need to relate to the work environment to maintain their intrinsic motivation. When all three needs are satisfied, integration is more likely to be achieved. Of these factors, autonomy support, including choice, feedback, and positive interpersonal climate with open communication and empathy, has been identified as more important for internalization [
Self-determination continuum of motivation.
Locke et al’s [
Ensuring that the goals are viewed as important and increase self-efficacy are the two important factors that lead to goal commitment and acceptance. Goal commitment is important and the goal-performance relationship is strongest when individuals are committed to their goals. Feedback is another important factor and it is important to know one’s progress against a target to adjust the level of effort or performance strategies to match the requirements of the target. Many studies have found that a combination of goals and feedback is more effective than goals alone [
Many studies set in different countries and contexts have established that the relationship between goal setting and increased performance is reliable, although some negative effects of goal setting, such as fraudulent reporting, are possible outcomes [
Our analytical framework rests on SDT and goal-setting theory, whereby organizational goals and behavioral standards are internalized by the CHW, enhancing autonomous motivation and performance. We expect that by providing targets, performance feedback, and supportive supervision through weekly phone calls, we will influence the three factors (autonomy, competence, and relatedness) that lead to greater internalization of the external regulation and goals and this will thus increase CHW performance and it will be sustained over time.
Conceptual framework: community health worker motivation, goal setting, and performance.
Regular and reliable supervision is cited as a missing link in increasing CHW performance in India [
Appreciation and encouragement are important elements of supportive supervision. Empirical studies find positive effects of verbal rewards on intrinsic motivation and performance [
Real Medicine Foundation (RMF) has been running a mobile health project using CommCare, with 60 community nutrition experts, targeting infant malnutrition in 600 villages located in five districts of Madhya Pradesh, India: Jhabua, Alirajpur, Barwani, Khandwa, and Khargone. Their aim is to increase communities’ awareness about malnutrition, mobilize communities to increase demands for treatments and services to alleviate malnutrition, and strengthen the capacity of frontline workers such as Anganwadi workers in combatting malnutrition. The community nutrition experts have been working since 2010 and the program has been in effect with CommCare since July 2011.
Since the program began, RMF has successfully treated 2157 children at nutrition rehabilitation centers, improved the nutritional status of 24,822 children suffering from moderate and severe acute malnutrition, and trained more than 329,780 individuals on malnutrition prevention, awareness, and treatment. RMF works closely with the Government of Madhya Pradesh.
The community nutrition experts in the RMF program are monitored and supervised primarily via their district supervisor. Although the community nutrition experts do receive some feedback from their district supervisors, they do not receive any individual-level feedback on performance metrics. The community nutrition experts attend a meeting at the district headquarters every 15 days. The district supervisor reviews the reports the community nutrition experts submit on child health status, discusses issues that the community nutrition experts have faced in the field, and develops a roadmap for the next 15 days. There is general acknowledgment of the community nutrition experts who have performed well for that period. The district supervisor also gives specific acknowledgment to community nutrition experts when they excel in the group environment to motivate other community nutrition experts to improve their performance. The district supervisor also accompanies the community nutrition experts on home visits to monitor performance. She spends the entire day with the community nutrition expert covering the households she visits and monitors the community nutrition expert’s work closely. She provides individual feedback/support at this time, which includes supporting community nutrition experts to motivate households that are resistant to admitting their child to the nutrition rehabilitation centers and supervising reporting (ie, reviewing how the data are entered on CommCare or their paper-based forms).
Focus group discussions and unstructured interviews with the community nutrition experts were conducted to gain insights on their goals, motivation, challenges, and support structures affecting their work. We conducted two focus group discussions with three and 10 community nutrition experts, respectively, and interviewed five community nutrition experts and all five district supervisors to gain better understanding of what motivates these health workers. Field observations shadowing five community nutrition experts also informed understanding of the community nutrition experts work structure and use of their mobile app.
The motivation goals of the community nutrition experts seem to be rooted in experience and the contributions they make to their households as a result of their job. Most community nutrition experts did not have prior work experience and most mentioned that they could not imagine going back to the situation when they did not work. They derive intrinsic motivation from their professional conscience and the visible improvements they see in the nutritional status of children in their catchment area. Working in the communities for 2 to 3 years, they form good relationships with the villagers. As a result, a negative outcome, such as the death of a patient, can be demotivating for them. The financial contributions they make to their households are a major motivational factor because their salary has made a significant impact on their household living standards. Their experiences have made them more confident and eager to learn more, and in many cases enlisted a desire to progress further and deal with more responsibility and gain higher salaries, although most do not have any set goals toward this objective. Very few community nutrition experts have set life or career goals for themselves. Their motivation seems to be based on a combination of intrinsic and extrinsic factors.
In this study, we examined the impact of performance feedback on CHW performance and motivation in a randomized controlled trial with a sample of 60 CHWs who were part of the community nutrition program with RMF in Madhya Pradesh, India.
In their article addressing ways to improve CHW performance in India, Bajpaai and Dholakia [
To communicate detailed performance feedback, we set up a call center, placing weekly calls to the community nutrition experts and relaying feedback on performance metrics. Phone calls provided a way to discuss and receive feedback from the community nutrition experts regarding any work-related issues, personal needs, mistraining, or technical difficulties they might have been having with the mobile app, which could be escalated to district supervisors so that they could be resolved quickly.
The European Foundation of Quality Management identifies the following human resource management principles as effective motivators for health works: supervision schemes, recognition schemes, performance management, training and professional development, leadership, participation mechanisms, and intraorganizational communication processes [
Elements of the intervention and the factors supporting internalization.
Elements of the intervention | Internalization factors | Rationale |
Greetings, introductions, discuss well-being, ask if good time to call | Relatedness, autonomy | Discussing well-being of the community nutrition expert and family displays concern and empathy, and supports autonomy and relatedness because the community nutrition expert can voice any personal difficulties affecting her work. |
Targets and performance feedback | Competence, autonomy | Providing performance feedback against goals targets competence and autonomy by discussing gaps in performance and proving the community nutrition expert has the means to assess her own performance as well. |
Congratulations/encouragement | Autonomy | Encouragement and positive feedback increase autonomy by instilling a sense of self into the community nutrition expert’s work. |
Reminders and retraining on app or work flow | Competence | Retraining and reminders on the community nutrition expert’s desired workflow or on how to use the app increases their competence. |
Discuss work challenges and ways to support their work | Autonomy, relatedness | Participation and being able to discuss any challenges or issues will increase autonomy by empowering the community nutrition expert to provide input. Discussing concerns of others improves relatedness. |
Technical support regarding phone or app | Relatedness, competence | Any technical problems arising on the phone can be fixed and community nutrition expert will be better able to perform the job. Discussing these issues will also improve relatedness |
Reminder of target and next call, goodbyes | Autonomy, relatedness | We provide targets with the aim to improve autonomy by giving the community nutrition expert the information to do her work. It could seem like external pressure, although it is communicated positively. Warm, encouraging, interpersonal communication supports relatedness and autonomy. |
Three performance indicators and respective targets were identified by the RMF program as relevant indicators to measure the performance of the community nutrition experts: (1) case activity, (2) number of infant and child health form submissions, and (3) duration of counseling.
Case activity was defined as the number of clients out of total clients that were visited in the last 2 weeks. The target was to visit all the registered clients every two weeks. By the end of each 2-week period, each community nutrition expert’s case activity was expected to be 100%, regardless of the number of clients registered. The case activity metric monitored the community nutrition expert’s coverage of the villages and clients and provided a measure for access to care available to the communities they served. Goal-setting theory suggests that goal difficulty has an inverse relationship with effort, with moderately difficult goals drawing the most effort out the individual and easy and very difficult goals drawing less effort and motivation to meet the target [
The form submissions indicator was the total number of infant health and nutrition forms submitted in the last week. This form collects information on the child’s nutritional and health status and tracks their nutritional progress at each follow-up visit by the community nutrition expert. The target here was for each community nutrition expert to fill out this form each time she visited a household to register or follow up with a child. Hence, the number of form submissions was measured against the total number of clients visited in the last week (case activity), which provided the target for this particular indicator. The community nutrition expert could exceed the target on this metric because case activity did not register repeat visits within a period as a new visit or account for more than one form submitted during a visit. Hence, the number of form submissions for the infant health and nutrition form could be higher than the case activity if the community nutrition expert visited the same case more than once during the week. The target for form submissions was measured based on case activity; the community nutrition expert was expected to fill out at least as many infant health and nutrition forms as the number of clients visited in the last week.
Family counseling duration was the adjusted mean time spent counseling families on infant nutrition in the last week. The target was for each community nutrition expert to spend at least 15 minutes counseling each family on the topics related to mother and infant nutrition. Tracking time taken by each community nutrition expert to counsel families on the importance of hygiene, nutrition, and admitting malnourished children to the nutrition rehabilitation centers provided a measure for the quality of care available to the communities. Because we wanted to ensure that each family received this minimum amount of counseling, all counseling times longer than 20 minutes were rounded down to 20 minutes during the data collection stage before any analysis. This was done in consultation with RMF, which had a target of 15 minutes for each counseling session. Tracking counseling times was made simple with CommCare, which records the time taken to complete each form and the community nutrition experts use a family counseling form to counsel the families.
The 60 community nutrition experts were randomized into three treatment groups, one for each of the three performance indicators (case activity, form submissions, duration of counseling). The study period was 10 weeks in total and was divided into two stages: the intervention period and the postintervention period. In the intervention period, which lasted for 6 weeks, each community nutrition expert received a weekly phone call discussing their performance on a certain indicator against its target. During this period, the other two groups served as the control group. We had imbalanced randomization for the analysis (2:1) and each treatment had 20 community nutrition experts; therefore, each control had 40 community nutrition experts.
In the postintervention period, which lasted for 4 weeks, the community nutrition expert did not receive any feedback to examine whether the effects of the feedback in the intervention period were sustained or wore off over time. Because we were interested in studying dynamic, intertemporal effects of our intervention, this was a crucial time period for our study in which we could examine whether any effects on performance and reporting standards were lasting. For instance, if we found that results were positive and sustained over the two study periods, the effects of the intervention could be deemed more effective than if these effects were attributed to a novelty factor and did not carry over time without constant monitoring and feedback. We did not have a control group that did not receive any calls because the RMF program requested that all community nutrition experts receive some feedback on their performance.
Randomly allotting the community nutrition experts into the three treatment groups could result in lopsided treatment groups based on baseline performance. To create balanced treatment groups, we first ranked the community nutrition experts baseline performance on each of the three indicators. We gave a percentage score for each indicator by comparing the community nutrition expert’s performance on each indicator against the target for that indicator and converted these into
Data for the study on the three indicators were obtained directly from CommCareHQ, CommCare’s cloud-based server where data recorded by the community nutrition expert were stored in real time. Additional data on the community nutrition expert’s demographic characteristics, personal traits, and motivation were collected in a survey conducted at the end of the postintervention period. We collected additional qualitative data from interviews with five community nutrition experts and the district coordinators as well as a focus group discussion with 10 community nutrition experts exploring motivations and challenges faced by community nutrition experts in their work and communities at the start of the study. We also collected some qualitative information at the end of the study to follow up on what was driving the results. This was done by randomly selecting three community nutrition experts from each treatment group for follow-up interviews to assess the main drivers of any changes in performance from their perspective.
We had panel data spanning 10 weeks, in which we collected performance data on each indicator for the three treatment groups. Our dataset had some important limitations, the more pressing of which was measurement error due to technical errors arising in the app. The community nutrition experts were working in areas with low network connectivity and were using an older version of the app, which had some technical errors that were not resolved before the start of our intervention. Working in low network connectivity means that the telecommunications network required to send the collected data to the central server was often inadequate or missing to send the information; therefore, there was measurement error in the data we could access from the server. Other technical errors, such as app error or missing multimedia and other bugs, meant that sometimes community nutrition experts were unable to access the app to record and send the data to the server. Many community nutrition experts also had repeat registrations for the same client with up to five repeat entries. This also affected our data quality for the first indicator—case activity—because this was measured against the total number of registered clients. During the intervention, community nutrition experts were advised not to fill out repeat registrations as they were doing before the intervention, meaning that the number of form submissions could decrease following the intervention. Similarly, we did not have data for total clients visited for the postintervention period, meaning that we did not have the target against which to measure form submissions data for that period. Therefore, form submissions is divided into two indicators: (1) form submissions number (ie, number of forms submitted in the last week), which was analyzed in the intervention and postintervention period; and (2) form submissions proportion, the number of forms as a proportion of total clients visited in the last week, which was analyzed only in the intervention period.
We administered an end-line survey to all community nutrition experts who were not lost to follow-up to capture their perception of the calls. We also randomly selected three community nutrition experts from each of the three treatment groups and conducted unstructured interviews on the phone to understand what was driving the results. The interviews were conducted by the program coordinator for RMF’s nutrition program.
We also conducted two focus group discussions during the intervention with three and 10 community nutrition experts, respectively, and interviewed five community nutrition experts and all five district supervisors to gain a better understanding of what motivated these health workers. Field observations shadowing five community nutrition experts also informed understanding of the community nutrition experts work structure and use of the mobile app. No feedback calls were placed during this time to any of the community nutrition experts.
Our aim was to study the effects of providing performance feedback on CHW motivation and performance. Because motivation is difficult to measure directly, we focused on assessing the effects of the feedback calls on CHW performance on specific indicators. The effects of other motivators could not be analyzed directly because of the study design, which did not include a control group that did not receive any calls. Therefore, the main focus of the analysis and empirical strategy is on the effects of the performance feedback.
The indicators used in this study were identified as direct measures of quality and access to health care because they were important for the RMF program and for other programs using mHealth. Adherence to correct practices in reporting and utilization of the mHealth platform is important and cannot be separately measured from the performance indicators discussed subsequently because all data were collected using CommCareHQ, which does not distinguish between reporting and performance. Accurate reporting is a vital element of the community nutrition expert’s performance and we viewed any change in performance indicators due to adherence to reporting standards as an improvement in CHWs overall performance as well.
The call center agent for our intervention was an anonymous call center operator, who was introduced to the community nutrition experts as a member of CommCare by RMF before the start of the intervention. The community nutrition experts were told to expect calls from CommCare and respond to the call by the RMF program. Due to unforeseen personal reasons, the call center operator changed after 3 weeks of the intervention and the researcher (SK), also introduced as a member of CommCare, placed the remainder of the calls. The community nutrition experts had no prior relationship to the call center operator or the researcher before the study. All the community nutrition experts received face-to-face interaction at the end of the study period for a survey and 18 of 60 community nutrition experts received additional visits, 13 in the form of a focus group discussion and five in the form of shadow visits to their clients during the intervention.
We began each call by greeting the community nutrition expert and asking her about her family and well-being; we then asked if she had some time to discuss her performance for that week. The community nutrition experts then received their performance feedback. The feedback was nuanced depending on whether she (1) met/exceeded her target, (2) improved a lot or little, or (3) showed no improvement in her work. All feedback included a reminder of the target, appreciation for their efforts, and a discussion of problems and solutions regarding their work. Those with little or no improvement were encouraged to meet their target and retrained on using the app and the desired workflow regarding the relevant indicator. Next, technical problems in the app were discussed and recorded to be escalated to the project coordinator. Finally, community nutrition experts received another reminder about their targets, the day and time of the next phone call, and a warm goodbye.
The mean duration of the calls was approximately 5 minutes, although the initial calls were longer. Each community nutrition expert received a call on a particular day of the week at a particular time determined by the community nutrition expert. Because the community nutrition experts lived in areas with poor network connectivity and could be busy with other work, we attempted to call every community nutrition expert five times in case she could not respond or the call did not go through. Three of these attempts were made on the day the community nutrition expert was scheduled to receive the call, one was made the following day, whereas those who were not reached during the week received an additional attempt at the end of the week when they were not at work.
We estimated main treatment effects and heterogeneous effects of call intensity (number of calls received) for each of the three performance indicators using a random effects model. We first estimated double difference estimates for main effects and heterogeneous effects in the intervention period, and then expanded the analysis to the postintervention period to estimate sustained effects of the intervention. The equations used to look for the main effects of the treatment for each indicator (double difference estimator, cross-partial effects, and comparing all three indicators head-to-head) as well as the equation used to estimate heterogeneous effects are presented in
We used a range of statistical tests fitting the nature of the variables. Community nutrition expert characteristics were compared using one-way ANOVA because we were comparing the means of three different treatment groups, whereas the Fisher exact test was used for the categorical variables because of the small number of observations in some of the categories. Binomial tests were used for the binary variable “other work.” Because the performance indicators did not follow a normal distribution, following Gneezy and Rustichini [
We took all measures possible to ensure that our study followed research governance and ethical protocols necessary for such research. The study benefited the participating community nutrition experts and the larger community where they worked by exploring ways to improve the quality and experience of the health services they provided to their communities. The study posed minimal risk to the participants and was part of the normal operational experimentation done with CommCare users. The community nutrition experts often received feedback through their district supervisors, which is in-line with the feedback provided in the intervention. There was no consequence to the community nutrition expert’s job as a result of the experiment because we did not systematically share performance data with RMF program managers making hiring and firing decisions.
In total, 60 community nutrition experts were randomized into three treatment groups based on baseline performance.
Randomization and treatment allocation of participants.
Descriptive statistics, community nutrition expert characteristics, and performance indicators.
Characteristics and indicators | Total (N=60) | Treatment group | ||||
Form submission (n=20) | Case activity (n=20) | Duration of counseling (n=20) | ||||
Baseline rank | 30.4 (17.22) | 29.95 (17.61) | 30.6 (17.29) | 30.65 (18.03) | ||
Age (years) | 32.15 (7.49) | 31.21 (5.84) | 32.15 (6.81) | 33.05 (9.54) | ||
Number of children in household | 1.73 (1.14) | 2.05 (1.02) | 1.85 (1.18) | 1.3 (1.13) | ||
Number of adults in household | 3.63 (1.98) | 3.42 (1.92) | 3.35 (2.03) | 4.1 (1.99) | ||
Education (years) | 11.55 (2.59) | 10.57 (2.24) | 11.8 (2.59) | 12.25 (2.73) | ||
Form submissions (n) | 43.68 (48.39) | 41.75 (47.69) | 42.4 (50.63) | 46.9 (49.15) | ||
Form submissions (proportion) | 0.42 (0.43) | 0.41 (0.45) | 0.37 (0.37) | 0.49 (0.48) | ||
Case activity | 17.65 (24.55) | 18.33 (27.00) | 16.4 (22.99) | 18.20 (24.71) | ||
Duration of counseling (mins) | 2.01 (3.24) | 2.44 (4.31) | 1.92 (2.69) | 1.67 (2.57) |
a Form submissions number was measured as the number of form submissions in a week. Form submissions proportion was measured as form submissions as a proportion of total clients. Case activity was measured as percentage of total clients visited in a 2-week period. Duration of counseling was measured as duration of family counseling in minutes.
The mean age of the community nutrition experts was 32 (SD 7.49) years with mean 11.5 (SD 2.59) years of education. Approximately 24% (14/60) of the community nutrition experts were engaged in other work, including agricultural work, in addition to their job with RMF. Baseline performance indicator means for the community nutrition experts were (1) form submissions (mean 43.68, SD 48.39), (2) case activity (mean 17.65, SD 24.55), and (3) duration of counseling (mean 2.01, SD 3.24). We did not find any significant differences between the three treatment groups in terms of rank, age, education, number of children, number of adults, marital status, and year started as community nutrition expert. Similarly, baseline performance indicators also did not exhibit a difference in medians for the three treatment groups.
The treatment groups and their relevant control group were also balanced in terms of community nutrition expert characteristics and performance indicators. However, we did see some significant differences between treatment and control means for duration of counseling in terms of number of children (
We tested for differential attrition across the entire sample and within the three treatment groups. The attrition rate for the entire sample was high at 17% (10/60) and 10 of 60 community nutrition experts were lost to follow-up during the course of the study. Most of these were due changes in the community nutrition expert appointments made by the RMF program during the course of the study. We had differential attrition based on rank in our data because (1) the program replaced some lower-performing community nutrition experts during the course of the study (this was done outside of our intervention because the performance feedback was not shared with RMF program managers) and (2) the data in CommCareHQ was not updated at the start of the study, so some community nutrition experts who dropped out or were replaced before the start of the study were included in the baseline data, whereas their replacements were included in the end-line data (ie, we did not have baseline data for the replacements). We found differential attrition for baseline rank (
We found differential attrition in terms of rank for all three treatment groups and in terms of age (
Lower-performing community nutrition experts in the baseline period who dropped out of the study in all three treatment groups could have affected our results.
To ensure that results were not driven as a result of the differential attrition, we re-estimated equations 1 and 4 in
Difference in differences (DID) estimates: impact of calls on performance indicators during the intervention stage.
Dependent variable | Case activity | Form submissions (number) | Form submissions (proportion) | Duration of counseling | |||||||||
DID | DID | DID | DID | t ( |
|||||||||
Treatment | –3.98 | –0.59 |
.55 | –7.09 | –0.97 |
.33 | –0.042 | –0.30 |
.76 | –0.54 | –0.37 |
.71 | |
After | 9.27 | 2.65 |
.008 | –36.94 | –8.93 |
.001 | 0.22 | 3.31 |
.001 | –0.09 | –0.11 |
.91 | |
Treatment×after | 4.66 | 0.80 |
.42 | 2.89 | 0.40 |
.68 | –0.002 | –0.02 |
.98 | 3.86 | 2.91 |
.004 | |
Constant | 22.20 | 5.46 |
<.001 | 52.15 | 12.57 |
.001 | 0.49 | 6.27 |
.001 | 2.43 | 2.97 |
.003 | |
Treatment | –6.57 | –1.48 |
.13 | –1.26 | –0.31 |
.76 | –0.12 | –1.35 |
.17 | –0.64 | –0.49 |
.62 | |
After | 12.43 | 4.41 |
<.001 | –29.50 | –8.80 |
.001 | 0.26 | 4.44 |
.001 | 0.07 | 0.1 |
.92 | |
Treatment×after | 4.78 | 1.23 |
.21 | –1.57 | –0.40 |
.68 | 0.04 | 0.49 |
.62 | 3.48 | 2.88 |
.004 | |
Constant | 21.78 | 5.28 |
.001 | 44.58 | 10.97 |
.001 | 0.51 | 5.91 |
.001 | 2.31 | 3.06 |
.002 |
The results showed a significant increase in performance for duration of counseling in the intervention period, whereas there were no significant treatment effects for case activity and form submissions. The interaction of counseling×after, which captured the treatment effect for receiving feedback on the duration of counseling, was significant at the 90% confidence interval, and receiving performance feedback on the duration of counseling increased mean counseling times by 3.860 minutes (
In
Difference in differences (DID) estimates: impact of calls on performance indicators in the intervention and postintervention stages.
Dependent variable | Case activity | Form submissions (number) | Duration of counseling | |||||||
DID | DID | DID | ||||||||
Treatment | –3.98 | –0.57 |
.57 | –7.09 | –0.95 |
.34 | –0.54 | –0.32 |
.75 | |
After | 9.07 | 2.65 |
.008 | –32.32 | 8.20 |
.001 | 0.57 | 0.77 |
.44 | |
Treatment×after | 5.58 | 0.98 |
.33 | 2.032 | 0.29 |
.77 | 4.78 | 3.66 |
.001 | |
Constant | 22.20 | 5.34 |
.001 | 52.15 | 12.31 |
.001 | 2.43 | 2.54 |
.01 | |
Treatment | –6.06 | –1.40 |
.16 | –1.142 | –0.29 |
.77 | –0.64 | –0.42 |
.67 | |
After | 12.71 | 4.52 |
.001 | –25.36 | –7.77 |
.001 | 0.746 | 1.09 |
.27 | |
Treatment×after | 4.45 | 1.2 |
.023 | –1.39 | –0.37 |
.71 | 4.34 | 3.65 |
.001 | |
Constant | 21.43 | 5.18 |
.001 | 44.51 | 10.7 |
.001 | 2.31 | 2.62 |
.009 |
We tested for statistical significance of the marginal difference in duration of counseling in the intervention and postintervention periods by restricting the data to only include observations in the postintervention period (weeks 7-10). We ran two specifications, first to look for changes in duration of counseling from the baseline period of week 1 and second where we set the baseline at the end of the intervention period in week 6. At week 1, duration of counseling was mean 2.45 (SD 3.80) minutes for the treatment group (n=34) and mean 1.89 (SD 2.83) minutes for the control group (n=16). At week 6, duration of counseling was mean 3.15 (SD 3.63) minutes for the control group (n=16) and mean 10.70 (SD 6.50) minutes for the treatment group (n=34). At weeks 7 to 10, duration of counseling was mean 3.82 (SD 5.53) minutes for the control group (n=64) and mean 9.22 (SD 7.71) minutes for the treatment group (n=136).
The results of the two specifications are presented in
Impact of performance feedback on duration of counseling in the postintervention period looking for sustained effects in performance.
Dependent variable | Duration of counseling | |||||
Baseline=week 1 (n=250) | Baseline=week 6 (n=250) | |||||
Coefficient | Coefficient | |||||
Duration of counseling | –0.536 | –0.30 | .76 | 7.552 | 4.09 | .001 |
After (baseline=week 1) | 1.390 | 2.07 | .39 | |||
Counseling×after (baseline=week 1) | 5.939 | 5.00 | .001 | |||
After (baseline=week 6) | 0.672 | 1.34 | .18 | |||
Counseling×after (baseline=week 6) | –2.149 | –2.43 | .02 | |||
Constant | 2.433 | 2.4 | .02 | 3.151 | 3.02 | .002 |
Treatment and control means in intervention and postintervention stages. Duration of counseling means are the mean duration of counseling per form submitted. BL: baseline.
There were no significant effects of receiving feedback on form submissions or case activity, in the intervention and postintervention periods. In the postintervention period, we did not have data for the total number of clients visited in the 7-day period; hence, we could not include form submissions as proportion of clients visited as a variable. Before the intervention, both form submissions and case activity were lower in the treatment group; however, this difference was not significant. There seemed to be an overall decrease in form submissions for all community nutrition experts (after) and the pre-post changes in the treatment and control groups were essentially the same. This could be explained by the fact that most community nutrition experts had multiple registrations of the same case in their apps, with some community nutrition experts having up to five duplicate registrations of the same case. Before the intervention, the community nutrition experts submitted up to five different infant and nutrition forms for each of the duplicate registrations. This meant that baseline form submissions were much higher than appropriate because the community nutrition experts submitted many duplicate forms for the same case. The performance feedback and retraining given to all community nutrition experts during the study period by the program reinforced that there was no need to submit duplicate forms despite the duplicate registrations, and only one infant health and nutrition form should be submitted per visit rather than submitting multiple forms for each of the duplicate registrations. This explains the overall decrease in form submissions in the intervention and postintervention period. Our interpretation was supported by the increase in form submissions proportional to number of clients visited in specification 3 for the control group and a slightly lower effect for the treatment group in the postintervention period.
Although case activity also showed an increase in performance in both the intervention and postintervention periods, the results were not significant. However, there was an overall increase in case activity for community nutrition experts in the control group in the intervention and postintervention period. The interpretation here could be that regardless of the information received during the calls, receiving calls boosted the community nutrition experts’ case activity. However, because we did not have a control group that did not receive calls, we could not estimate the effect of calling itself. The results are confounded by other factors, which could have caused a secular trend of increasing performance on case activity, including retraining during the intervention period by RMF, which alone could explain the increased performance after the intervention, independent of the intervention itself.
Although there were some program-wide changes to increase community nutrition expert performance during our intervention confounding the effects, including distribution of an additional paper-based job aid for counseling and retraining on CommCare for all the community nutrition experts, our intervention played an important part in increasing community nutrition expert performance for case activity. One interpretation of these results could be that regardless of the information disclosed in the calls, receiving calls boosted community nutrition expert performance by increasing internal motivation and increasing interest in their work. Although providing performance feedback against a target was an important element of our intervention, the feedback calls went beyond providing simple feedback and fluid discussions about the community nutrition experts’ technical and program-related challenges, and some personal/family matters were an important part of the calls. Our intervention strengthened supervisory structures already present in the program, regularly communicating the technical and program-related challenges to the project coordinator and the community nutrition experts’ immediate supervisors, who were able to resolve these more quickly, contributing to the increased case activity for all community nutrition experts in the postintervention stages. Additionally, the calls also resolved problems the community nutrition experts were having in using their app, which could also have contributed to the increasing their case activity because all data was collected through their CommCare apps.
The challenge in drawing out the impact of the calls is that we did not have a control group that did not receive any calls in our design. However, our interpretation is supported by the qualitative work supporting this study where community nutrition experts mentioned that the calls were an important factor in motivating and engaging them, increasing their interest in their work. The results presented in
The end-line survey administered to all community nutrition experts suggested that community nutrition experts perceived the calls as a motivator having a positive effect on their performance. In all, 86% (38/44) reported that they felt the calls improved their performance, 79% (34/43) felt the calls motivated them a lot to do their work, and 45% (20/44) found performance feedback to be the most effective component of the calls for motivation. In addition, 75% (33/44) reported that their technical problems were resolved faster as a result of the calls and 100% (44/44) of community nutrition experts felt that their knowledge about using CommCare had improved as a result of the calls and 94% (43/46) would sign up for this service. These results support our interpretation that the calls improved community nutrition experts’ intrinsic motivation by supporting competence, relatedness, and autonomy, leading to improvements in case activity for all community nutrition experts.
To determine whether receiving feedback on any indicator affected case activity performance or performance on the remaining two indicators, we estimated cross-partial effects of providing performance feedback on one indicator on the performance on the other two indicators. For each of the treatments, the remaining two groups acted as a pooled control and we were interested in estimating cross-partial effects of receiving feedback on case activity, on performance in duration of counseling, or form submissions. We present the analysis for the postintervention period because this generated higher power and was a more policy-relevant result. However, our analysis showed the results in the intervention period were similar to those in the postintervention period. There was one cross-partial effect where feedback on one indicator affected performance on another indicator. It negatively affected duration of counseling by approximately 3.39 minutes (
We compared each of the three treatment groups head-to-head and estimated how feedback on each indicator affected each treatment group separately. The reference group in each specification was the treatment group. The results were in-line with our analysis from equation 1 (
Head-to-head comparison of treatment groups (n=500).
Variable | Case activity | Form submissions (number) | Duration of counseling | |||||||
Coefficient | Coefficient | Coefficient | ||||||||
Case activity | — | — | — | 1.99 | 0.23 | .81 | 0.19 | 0.09 | .92 | |
Form submissions | 3.66 | 0.46 | .64 | — | — | — | 0.94 | 0.47 | .63 | |
Duration of counseling | 4.29 | 0.54 | .58 | 12.81 | 1.46 | .14 | — | — | — | |
After | 4.52 | 0.93 | .35 | –3.29 | –5.28 | .001 | 5.35 | 4.97 | .001 | |
Case activity | 10.13 | 1.53 | .12 | 2.36 | 0.3 | .76 | –4.09 | –2.77 | .005 | |
Duration of counseling | 9.10 | 1.33 | .18 | –6.97 | –0.86 | .39 | ||||
Form submission | –5.56 | –3.65 | .001 | |||||||
Constant | 18.23 | 3.31 | 45.06 | 7.28 | 1.89 | 1.35 | .17 |
The distribution of calls received among the 50 community nutrition experts who were included in the analysis was as follows: 10% (5/50) of community nutrition experts received zero calls, 6% (3/50) received one call, 6% (3/50) received two calls, 16% (8/50) received three calls, 14% (7/50) received four calls, 10% (5/50) received five calls, and 38% (19/50) received all six calls.
We wanted to examine whether the effect of our intervention varied across the intensity of treatment (estimate equation 4 in
Heterogeneous effects: the effect of receiving more calls on performance.
Variable | Case activity | Form submissions, number | Duration of counseling | |||||||
Coefficient | Coefficient | Coefficient | ||||||||
Treatment | –4.70 | 0.28 |
.77 | 8.05 | 0.52 |
.60 | –1.45 | –0.40 |
.69 | |
After | 22.07 | 3.15 |
.001 | 0.21 | 0.02 |
.98 | –0.72 | –0.45 |
.65 | |
Calls | 4.52 | 2.26 |
.02 | 7.98 | 3.70 |
.001 | –0.39 | –0.89 |
.37 | |
Treatment×after | 13.98 | 1.03 |
.30 | –9.97 | –0.70 |
.48 | –0.45 | –0.16 |
.87 | |
Treatment×calls | –0.54 | –0.15 |
.88 | –2.96 | –0.84 |
.40 | 0.22 | 0.27 |
.79 | |
After×calls | –3.47 | –2.12 |
.03 | –7.79 | –3.93 |
.001 | 0.32 | 0.90 |
.37 | |
Treatment×after×calls | –1.35 | –0.47 |
.63 | 2.12 | 0.65 |
.51 | 1.37 | 2.11 |
.04 | |
Constant | 5.24 | 0.61 |
.54 | 18.81 | 1.89 |
.06 | 4.05 | 1.99 |
.05 | |
Treatment | –6.28 | –1.35 |
.17 | –0.39 | –0.10 |
.92 | –1.18 | –0.38 |
.70 | |
After | 26.17 | 4.72 |
.001 | 4.31 | 0.64 |
.52 | –0.15 | –0.11 |
.91 | |
Calls | 4.28 | 2.75 |
.006 | 8.04 | 5.08 |
.001 | –0.26 | –0.65 |
.52 | |
Treatment×after | 10.86 | 1.40 |
.16 | –0.77 | –0.11 |
.91 | –0.90 | –0.36 |
.72 | |
Treatment×calls | –0.18 | –0.12 |
.90 | –0.64 | –0.39 |
.69 | 0.14 | 0.19 |
.84 | |
After×calls | –3.67 | –2.67 |
.007 | –7.93 | –4.97 |
.001 | 0.24 | 0.75 |
.45 | |
Treatment×after×calls | –1.26 | –0.64 |
.52 | –0.02 | –0.01 |
.99 | 1.42 | 2.43 |
.02 | |
Constant | 5.71 | 0.83 |
.40 | 14.44 | 2.06 |
.39 | 3.28 | 1.91 |
.06 |
The number of calls received per community nutrition expert suggested that the number of calls received, regardless of the indicator for feedback, had a positive impact in pre-post differences in the control group. Because all community nutrition experts received calls, this finding suggested that regardless of the information disclosed in the calls, receiving calls improved performance by providing supportive supervision and increasing intrinsic motivation by including appreciation, encouragement, avenues to discuss problems and solutions, and solving technical difficulties.
Our findings show some interesting differences in the effect our intervention has on the different performance indicators. We found positive impacts on duration of counseling sustained in the postintervention period, whereas case activity and form submissions showed no significant effects. We also found heterogeneous impacts of call intensity for duration of counseling, with each additional call improving the community nutrition expert’s counseling times further. There was an overall increase in case activity for all community nutrition experts, whereas the number of form submissions decreased due to multiple registrations and duplicate form submissions declining indicating improvement in overall reporting standards. We discuss possible reasons for these differing results subsequently, further supplemented with qualitative interviews with community nutrition experts to better understand the results.
First, goal-setting theory suggests that the difficulty of the target matters for motivation and performance, and moderately difficult targets induce more effort and motivation compared to difficult or easier targets [
Second, the data quality for duration of counseling is likely to include less measurement error compared with data capturing case activity and form submissions. The community nutrition experts had many duplicate registrations in their app for each client due to mistraining issues at the beginning of the CommCare program. As a result, some community nutrition experts had up to 400 or 500 clients registered in their app with most of these being duplicate registrations. Case activity was measured against total clients and community nutrition experts were regularly submitting multiple forms (for each of the duplicate registrations) skewing form submissions and case activity data. The overall decrease in form submissions number in both the treatment and control groups can be explained by the retraining from RMF, which reinforced that the community nutrition experts should not submit multiple forms for the same visit in spite of the duplicate registrations. Our intervention also included training on how to correctly use the app and reinforced the appropriate workflow that the community nutrition experts should adopt, which also contributed to the reduction in the number of form submissions. Form submission as a proportion of clients visited did not show any significant decrease supporting our reasoning.
Third, case activity was affected by the holiday period of Diwali, which fell around week 3 of the intervention period.
Fourth, all community nutrition experts also received a paper-based counseling aid during the study, which could also drive the increase in duration of counseling. However, we expect this aid to affect counseling times for all community nutrition experts and not just those in the treatment group. It could also be the case that the additional counseling aid interacted with receiving duration of counseling feedback by positively affecting duration of counseling and, in the absence of the additional counseling aid, there would be no significant effects on duration of counseling. However, we cannot perform this analysis due to data constraints.
Lastly, the significance of differential attrition by rank is higher in the duration of counseling treatment group than in the other two groups. Thus, it could be the case that the duration of counseling results are being driven by differential attrition, although this is unlikely because all groups show significant differential attrition by rank.
We expect that the changes in performance also include improvements in reporting standards and are being driven by increased effort on the part of the community nutrition expert, where our intervention improved community nutrition expert motivation and encouraged and increased her interest toward her work. External motivation played a part where community nutrition experts are weary of the consequences of bad performance, such as getting fired or reprimanded by the program. Internalization of the programs’ goals to improve intrinsic motivation was facilitated by improving the community nutrition experts’ competence; the community nutrition experts are better at using their CommCare app after the intervention, which included components where community nutrition experts were trained on proper use of the app to support their work flow. Improvements in duration of counseling are also driven by improvements in reporting standards and increased knowledge in correct use of the app. Before the intervention, many community nutrition experts counseled their clients without opening the family counseling form leading to very low counseling times recorded on CommCare. After the intervention, and retraining on how best to use the family counseling form (ie, we encouraged them to open the form and fill in the details as they counsel), we see an increase in counseling time as a direct impact of our intervention. Accurate reporting is an essential component of performance in the CHW context due to the remote nature of their work and lack of immediate supervision, which means that any CHW’s activities, services provided, and changes in the client’s health status are not known without accurate data provided by the CHW. We also see improvements in reporting standards for form submissions.
Qualitative interviews with three community nutrition experts after the postintervention period support our interpretation that the calls improved the community nutrition experts’ competence and autonomy by improving knowledge and skills and helping to quickly solve any technical issues faced by the community nutrition expert. The continuous monitoring provided by the calls was seen to impact duration of counseling more than the other indicators, with the community nutrition expert giving more time to the client in addition to using the family counseling form while counseling to provide accurate reporting on this indicator.
Case activity also shows overall improvements for all community nutrition experts after our intervention, which is sustained in the postintervention period. Although we do not have a control group that did not receive any calls to provide concrete evidence, the results suggest that the calls, which included a strong component of supportive supervision, helped support autonomy and the internalization of program goals (which all community nutrition experts were aware of) leading to more autonomous motivation in the community nutrition experts and higher performance across all treatment groups. Our hypothesis that the calls lead to greater autonomous motivation is supported by the sustained improvements in duration of counseling in the postintervention period as a result of receiving feedback calls on duration of counseling and overall improvements in case activity, and improvements in reporting form submissions sustained after the intervention as well.
Providing CHWs with regular performance feedback through calls placed by a supervisor or manager is an effective way to reinforce goals and targets, and provide supportive supervision to CHWs encouraging their extrinsic and autonomous motivation and performance. Two-way communication opens avenues to discuss and provide solutions addressing gaps in performance, any challenges faced at work, or technical or systematic issues afflicting the CHW preventing her/him from performing effectively. Strategies targeting improvements in CHW programs should include an element of continuous monitoring and feedback system reinforced with supportive supervision to generate improvements in CHW performance and maintain an effective CHW program.
Other than the limitations posed by the dataset, our study has some important limitations that could potentially bias our outcomes:
1. Risks of unethical behavior leading to false reporting.
2. Spillovers where targets for different indicators are known to community nutrition experts in other groups.
3. Low network connectivity or unwillingness/inability of the community nutrition experts to respond to and answer the calls (could potentially be endogenous to performance if the community nutrition experts are aware of their own performance and do not answer the call if they perceive their performance to be low).
4. Holidays and events in week 3, which meant that the community nutrition expert(s) were absent with consent from their managers. Some may have extended their break. Attendance is a prerequisite to performance.
5. The most important limitation in the study arises because we do not have a control group that received no calls and no performance feedback. Hence, we cannot discern the effects of the calls themselves. This is important because the calls included other components other than the performance feedback, which can impact the community nutrition experts’ performance across all indicators, including the indicators on which they received no performance feedback.
In order to limit and test for the bias, we adopted the following strategies:
1. Draw an upper limit on the indicators where false reporting is most plausible. We imposed a limit of 20 minutes on the duration of counseling so that any outliers did not skew the mean counseling times.
2. Although we ensured that no community nutrition expert received feedback on other indicators other than the treatment she was assigned to, RMF reiterated the targets for all three indicators during the course of the study, which could have affected the community nutrition experts’ performance by setting goals. Goal setting can motivate performance, but we were interested in the effects of goal setting in conjunction with providing feedback.
3. We tested for correlation between initial performance and number of calls received to test whether community nutrition experts were deliberately avoiding calls anticipating unfavorable performance feedback. The pairwise correlation for baseline rank and number of calls received was
Equations used to estimate main treatment effects and heterogeneous effects.
Results of matching based on ranks and treatment groups.
community health worker
difference in difference
Real Medicine Foundation
self-determination theory
We would like to thank USAID for funding the research, as well as RMF for their participation in the study.
Dimagi is the owner of CommCare, the mHealth platform used by the participants in the study. Authors Neal Lesh and Sangya Kaphle were employed by Dimagi during the time of research. Michael-Mathke Ficher works with RMF.