A study of the existing problems of estimating the information transfer rate in online brain-computer interfaces

Objective. Today, the brain-computer interface (BCI) community lacks a standard method to evaluate an online BCI's performance. Even the most commonly used metric, the information transfer rate (ITR), is often reported differently, even incorrectly, in many papers, which is not conducive to BCI research. This paper aims to point out many of the existing problems and give some suggestions and methods to overcome these problems. Approach. First, the preconditions inherent in ITR calculation based on Wolpaw's definition are summarized and several incorrect ITR calculations, which go against the preconditions, are indicated. Then, the issues affecting ITR estimation during the test of online BCI systems are discussed in detail. Finally, a task-oriented online BCI test platform was proposed, which may help BCI evaluations in real-world applications. Main results. The guidelines for ITR calculation in online BCIs testing are proposed. The platform executed in the Beijing BCI Competition 2010 shows that it can be used as a common way to compare the online performances (including the ITR) of existing BCI paradigms. Significance: The proposed guidelines and task-oriented test platform may reduce the uncertainty and artifacts of online BCI performance evaluation; they provide a relatively objective way to compare different BCI's performances in real-world BCI applications, which is a forward step toward developing standards for BCI performance evaluation.