A Latent Variable Model for Evaluation of Disparate Ratings of Stem Cell Colonies by Two Experts

The visual inspection of pluripotent stem cell colonies by microscopy is widely used as a primary method to assess the quality of the preparations and degree of pluripotency. The lack of ground truth and the possible inconsistency of evaluations from multiple experts within and between stem cell laboratories are sources of uncertainty about the state of the cells, the reproducibility of preparations, and the efficiency of expansion protocols. To examine how to evaluate the level of confidence one has in disparate rating from experts, we explored a statistical method for assessing the differences in ratings of pluripotent stem cells by two different experts. Two experts rated phase contrast microscope images of human embryonic stem cell (hESC) colonies on a scale of 1 (poor) to 5 (maximum pluripotency character) but agreed with one another only 48% of the time. To assess whether experts used similar criteria to rate colonies, we developed custom image feature algorithms based on the stated visual criteria provided by the experts for selection of colonies. These features, plus others, were then used to develop pluripotency scoring algorithms trained to reflect ratings of both experts. We treated expert ratings as inexact indicators of a continuous pluripotency score and considered the inconsistency between expert ratings in developing our models. The model suggests that the two experts use somewhat different scales for discriminating between colony quality. Covariance analysis indicated that both experts use features that are not included in the model. Two image features, colony perimeter and a feature based on texture, were the most important for both experts for predicting the ratings. Interestingly, colony perimeter was not one of the expert-provided criteria for rating colonies, showing that this modeling approach allowed identification of features that the experts were not aware they were using. A linear model based on both experts identified each experts top-rated colonies as well as, or better than, the ratings of the other expert, as indicated by receiver operator characteristic curve analysis. By providing an understanding of the differences and similarities in disparate sets of expert ratings, this analysis helps to establish confidence in the ratings and the criteria for ratings, even when the experts disagree.