High performance lossless compression of scientific floating data

Scientific computing is playing more and more important role in the scientific research and in the industry, and enormous valuable scientific data are produced. Since the read/write speed of secondary storages (e.g. hard disks) is usually slow, besides occupying storage space, mass data also affect system performance. This paper studies the properties of floating-point data thoroughly, and establishes the theoretical relationship between the prediction precision and the compression ratio. By exploiting the interdependency between scientific data and making use of several predictor and high performance entropy encoding method, this paper proposes a high performance lossless compression method for scientific data. The method doesn't require users having priori knowledge about the data, nor is custom-designed predictor required. Comparing results with other methods, show the method has much higher compression ratio and constant high throughput. The method has been used for the compression of scientific data in large scale discrete element simulation of particle dynamics.