Sandia Approach

Statistical and Human Factors Evaluation :

Test Design
Performance Estimation
Uncertainty Computation

Previous slide Next slide Back to the first slide View Graphic Version

Notes:

Sandia's approach to evaluation and comparison of biometric authentication devices is based on the application of sound statistical and human factors methodologies in the test design, performance evaluation, and uncertainty calculation. The individual, procedural, and environmental variables are reviewed from a Human Factors perspective to identify those variables that can affect task peformance and user acceptability. The principles of experimental design are employed when designing a study. The use of statistical methods allows us to consider the potential factors affecting performance when planning the test, and to size the test appropriately for these factors. Statistical tests of hypotheses for the effects of the factors, both singly and jointly, are performed. Useability questionnaires are designed and evaluated to assess the acceptability of the devices in terms of ease of use, time demands, privacy issues, and health issues. Statistical models are developed for the effects of the factors on the score distributions. These models are the bases for the estimation of the performance characteristics, and their uncertainties, in the form of confidence intervals representing sampling variability. Uncertainty estimation is essential to report the variation in performance characteristics due to sample limitations. Uncertainty bounds can indicate whether observed performance differences between two devices are real, or due to chance. The performance results and usability questionnaire may suggest improvement areas, in equioment design, software interface design, operational procedures, and algorithms.

This presentation does not cover vulnerability analyses, susceptibility to intentional damage, or unauthorized access by insiders familiar with the system design. This is usually the first analysis performed on systems; performance characteristics are irrelevant if the system has a feasible mode for by-pass. The results of vulnerability analyses are extremely sensitive, and must be protected with a high security level.