Performing record linkage and deduplication in master patient index using machine learning classifiers

Abstract

The prevalence of duplicate patient records and the difficulties in accurately linking them is a big concern in healthcare. Duplicate records hinder quality healthcare, making the need for an effective Master Patient Index (MPI) system compelling. This paper investigated the performance of five machine learning classification algorithms (random forests, extreme gradient boosting, logistic regression, stacking ensemble, and deep multilayer perceptron) for data linkage and deduplication on four datasets. The result demonstrates the applicability of machine learning models for effective data linkage and deduplication of electronic health records. The random forest algorithm achieved the best performance (identifying duplicates correctly) based on accuracy, F1-Score, and AUC-score for three datasets (ePBRN: Acc = 99.83%, F1 = 81.09%, AUC = 99.98%; FEBRL3: Acc = 99.55%, F1 = 96.29%, AUC = 99.77%; Custom-synthetic: Acc = 99.98%, F1 = 99.18%, AUC = 99.99%). In contrast, the multilayer perceptron (Artificial Neural Network) had the best performance on the one dataset (FEBRL4: Acc = 99.93%, F1 = 96.95%, AUC = 99.97%). However, no single model was universally effective across all datasets, emphasising the need for tailored solutions. This study highlights the potential of machine learning to address the problem of duplicate records in healthcare specifically. By enhancing MPI systems, the study contributes to safer patient care and more efficient healthcare operations through improved record management.