Notes: 3 Often the purpose of testing is to select the best of a set of similar devices for a particular application. The selection is made by comparing the devices on a specification of the metrics or attributes, reflecting the intended application. Typical examples include system cost, system operational constraints (hardware, software limits), enrollment time, verification time, probabilities of false accept and false reject as a function of threshold, receiver operating characteristics, and user acceptability. User acceptability refers to the users' perceptions about ease of use, the invasiveness of the device, and perceived safety concerns. User acceptability is more important in some applications than others. For example, a system to be used for customers may need a higher degree of user acceptability than a system to be used with employees. The validity of the study depends in large part on the identification, prior to testing, of potential performance influencing factors, and structuring the test to estimate their effects on performance. We look for factors in three classes: population, environment, and procedure. Population factors are those aspects of an individual that may effect the ability of the system to correctly verify him/her. For voice verification systems, our research indicated that gender and age were potential population factors. Procedural factors are those issues of using the system, specified by the vendor, that may contribute to, or detract from, high performance. For example, if recorded instructions are unclear or misleading, the performance characteristics may be suboptimal. Environmental factors are those features of the use environment that can detract from or enhance performance. For example, back-ground noise can detract from voice verification performance. Ideal systems are robust for variations in population, procedural, and environmental factors. To compare different systems, it is critical to know the factors that most detract from performance. Sources of variability include those introduced by the system itself, in the sense of measurement error from different data collection attempts.