Can womens reports in client exit interviews be used to measure and track progress of antenatal care quality? Evidence from a facility assessment census in Malawi

IntroductionExit interviews given their shorter recall period unlike household surveys provides an opportunity to capture routine performance and level of service quality at the facility level. However, very few studies exist validating whether womens reports in exit interviews can reliably be used to measure quality of care. This study contributes to the sparse literature by examining the validity and reliability of womens report of selected antenatal care (ANC) interventions in the exit interviews in Malawi. MethodsUsing the 2013-2014 Malawi service provision facility census, we compared womens reports at exit interviews of ANC received with reports through direct observation by a trained third-party. Six indicators of ANC common in both direct observation and client exit tool were identified. These include; whether provider prescribed or gave fansidar for malaria prevention; whether provider prescribed or gave iron and folic tablets; whether provider explained side effects of iron and folic tablets; whether provider discussed importance of good nutrition during pregnancy; whether provider discussed delivery preparation and whether provider discussed pregnancy related complications. We assessed indicator accuracy by calculating sensitivity, specificity, area under the receiver operating characteristic curve (AUC) and inflation factor (IF). Indicators considered to have both high individual accuracy (an AUC value of 0.70 or greater) and low population-level bias (0.75<IF<1.25) were considered to have acceptable validity. To measure agreement, we calculating Kappa coefficient ({kappa}) and the prevalence adjusted bias adjusted kappa (PABAK). Both {kappa} and PABAK values were interpreted as follows: almost perfect (>0.80), substantial (0.61-0.80), moderate (0.41-0.60), fair (0.21-0.40), slight (0.00-0.20) and poor (<0.00). Using chi-squared tests we also examined factors that are associated with high agreement between exit interview reports and direct observations. Resultsfor both validity and agreement measures, our findings show that womens self-reports in the exit interviews presented overall higher accuracy and agreement for indicators related to concrete, observable interventions. For example, indicators which met accuracy criteria reflected those to do with medical prescription. In contrast, indicators related to counselling or advice given, performed less reliably. The results also show that age, primiparous status, number of antenatal visit and type of health provider were associated with high level of agreement. ConclusionIn the context of calls for enhanced measurement of the components that lead to effective coverage, study findings such as these suggest that careful consideration of the type of information women are asked to recall is needed and also the timing is important. While household survey programmes such as the DHS and MICS are frequently relied on as data sources for measuring intervention coverage, triangulation of such findings with other data sources such as client exit interviews is important.