Evaluation synthesis analysis can be accelerated through text mining, searching, and highlighting: A case-study on data extraction from 631 UNICEF evaluation reports

BackgroundUNICEF works to protect childrens rights and improve their well-being by partnering with governments and communities through various programs. These include protection, education, health, water, sanitation, and emergency support. Publicly available evaluation reports1 provide insights on results, recommendations, and lessons learned. ObjectiveThis article describes semi-automated methods to synthesize UNICEFs extensive evaluation report collection to assess UNICEFs impact, identify achievements, challenges, and inform future strategy. MethodsA semi-automated approach was used to extract data from 631 evaluation reports across 64 outcomes in UNICEFs 2022-2025 Strategic Plan. Text pre-processing involved PDF-to-text extraction, section parsing, and neural network sentence mining. Data extraction was facilitated by SWIFT-Review using adjacency-search queries for outcome-based filtering. ResultsWe reduced text volume by 92%, identified relevant sentences with high recall (0.93), and outcomes within evaluation texts with median precision 0.6 (reading 21 reports per outcome). ConclusionsSemi-automation accelerates synthesis while maintaining scientific rigor and reproducibility. Plain Language summaryUNICEF is a global organization that works to improve the lives of children by protecting their rights and helping them access essential services like education, health care, clean water, and protection from violence. UNICEFs work is guided by a strategic plan and is evaluated through reports that describe their programs in detail. These reports, available to the public, contain valuable information on the results, recommendations, and lessons learned from UNICEFs work around the world. We would like to understand UNICEFs impact and identify areas where they can improve. This can be done by summarizing the main achievements, challenges and opportunities for future strategies based on UNICEFs evaluation reports. Because there are many reports, reviewing all of them manually would be too time-consuming, so a semi-automated process was used to speed up the work. We developed a method that used computer tools to highlight key information from 631 evaluation reports. These reports covered 64 different outcomes based on the goals in UNICEFs 2022-2025 Strategic Plan. We automatically turned text into machine-readable format, used artificial intelligence to identify important sentences, and then used advanced software to filter and find the most relevant information. The semi-automated approach was effective, reducing the amount of text by 92%. It also identified relevant outcomes with high accuracy. This method helped us speed up the process while still maintaining the quality and reliability of our findings. Overall, using automation helped save time and effort, making it possible to analyze large amounts of data efficiently while still ensuring the results were scientifically sound. Strengths and limitations of this studySystematic impact evaluation syntheses are a vital tool to critically evaluate and plan future work of organisations such as UNICEF; but they are often not feasible due to the size, structure, and amount of evaluation report documents. To increase feasibility of analysis we describe semi-automated human-in-the-loop methods which were applied in a synthesis of 631 evaluations across 64 outcomes. The proposed open-source code and methods made an evaluation synthesis feasible by reducing text and streamlining the identification of relevant reports for each outcome. By making code open-source and adaptable we aim to encourage accelerated, yet transparent and reproducible results. While the methods cannot produce 100% complete or correct results for each outcome, they present useful automation methods for researchers facing otherwise non-feasible evaluation syntheses tasks.