In which cases does it make sense to classify an individual into one population and what are the alternatives?

During biogeographical ancestry (BGA) analysis, an unknown individual for whom ancestry informative markers (AIMs) were analyzed is often assigned to one of a few reference populations. For some individuals, such an assignment is not meaningful: the two parents of the individual may come from different populations, or the individual may come from a population not included in the reference database. In this study, we examined classification methods in combination with 104 AIMs from the VISAGE Enhanced Tool to provide one of the following three possible outcomes: (i) in-reference (IR), with an assignment to one of the reference populations; (ii) admixed (AD), with parents from two in-reference populations; and (iii) out-of-reference (OR), where the individual's ancestry is not sufficiently represented by the reference populations. We therefore compared seven classifiers in combination with class-specific acceptance thresholds for the OR decision. The best-performing method according to our selection criterion was a naive Bayes classifier over pairs of populations, which we call the recent naive Bayes (RNB) classifier. Populations held out of the reference panel were correctly rejected in $84.9\%$ of cases. We further tested the method on $800$ individuals from seven additional groups not included in the reference panel. Only $29.4\%$ of individuals from four non-admixed populations were rejected, resembling joint ancestry with reference populations. In addition, we investigated the prediction accuracy in a newly collected dataset of 28 individuals with self-reported ancestry by analyzing AIMs using massively parallel sequencing.