Systematic Evaluation of Common Natural Language Processing Techniques to Codify Clinical Notes

Proper codification of medical diagnoses and procedures is essential for optimized health care management, quality improvement, research, and reimbursement tasks within large healthcare systems. Assignment of diagnostic or procedure codes is a tedious manual process, often prone to human error. Natural Language Processing (NLP) have been suggested to facilitate these manual codification process. Yet, little is known on best practices to utilize NLP for such applications. Here we comprehensively assessed the performance of common NLP techniques to predict current procedural terminology (CPT) from operative notes. CPT codes are commonly used to track surgical procedures and interventions and are the primary means for reimbursement. The direct links between operative notes and CPT codes makes them a perfect vehicle to test the feasibility and performance of NLP for clinical codification. Our analysis of 100 most common musculoskeletal CPT codes suggest that traditional approaches (i.e., TF-IDF) can outperform resource intensive approaches like BERT, in addition to providing interpretability which can be very helpful and even crucial in the clinical domain. We also proposed a complexity measure to quantify the complexity of a classification task and how this measure could influence the effect of dataset size on models performance. Finally, we provide preliminary evidence that NLP can help minimize the codification error, including mislabeling due to human error.