Comparison of straight line curve fit approaches for determining parameter variances and covariances

Pressure balances are known to have a linear straight line equation of the form y = ax + b that relates the applied pressure x to the effective area y, and recent work has investigated the use of Ordinary Least Squares (OLS), Weighted Least Squares (WLS), and Generalized Least Squares (GLS) regression schemes in order to quantify the expected values of the zero-pressure area A0 = b and distortion coefficient λ = a/b in pressure balance models of the form y = A0(1 + λx). The limitations with conventional OLS, WLS and GLS approaches is that whilst they may be used to quantify the uncertainties u(a) and u(b) and the covariance cov(a, b), it is technically challenging to analytically quantify the covariance term cov(A0, λ) without additional Monte Carlo simulations. In this paper, we revisit an earlier Weighted Total Least Squares with Correlation (WTLSC) algorithm to determine the variances u2(a) and u2(b) along with the covariance cov(a, b), and develop a simple analytical approach to directly infer the corresponding covariance cov(A0, λ) for pressure metrology uncertainty analysis work. Results are compared to OLS, WLS and GLS approaches and indicate that the WTLSC approach may be preferable as it avoids the need for Monte Carlo simulations and additional numerical post-processing to fit and quantify the covariance term, and is thus simpler and more suitable for industrial metrology pressure calibration laboratories. Novel aspects is that a Gnu Octave/Matlab program for easily implementing the WTLSC algorithm to calculate parameter expected values, variances and covariances is also supplied and reported. © V. Ramnath, published by EDP Sciences, 2020.