Some Adventures in Reliability Estimation

In many measurement settings, the reliability of a measure can be straightforwardly estimated. One might have a test that is at least supposed to be unidimensional, on which everyone is supposed to respond to all the same items, and the score is a simple sum of correct answers. Or maybe one has a computerized adaptive test, producing score estimates with individual errors of measurement and corresponding individual reliability values, all based on the same posterior distribution. But sometimes, such as in industry applications of gamified assessment, one arrives at measures for which one wants to estimate the reliability-and yet, they don't look much like traditional tests, or use IRT for scoring. Then, one is obliged to be adventurous. This collection of anecdotes from a career-matching platform features applications of a variety of techniques for estimating reliability in unusual situations, such as composite reliability, structured applications of split-half, and modeling and simulation.