Euclid: image compression activities for the VIS instrument

Euclid is a space mission dedicated to the high-precision study of dark energy and dark matter. Its visible instrument (VIS) will acquire wide field images by means of an array of 36 CCD focal plane detectors. Considering that each acquired full frame produces a huge amount of data (similar to 1.2GByte), an overall daily production of similar to 120 GByte is expected, which must be compressed to fit the 520 Gbit VIS daily telemetry. Due to the highly demanding science requirements such compression must be rigorously lossless. This software requirement is very hard to meet because of the following constraints: i) the average Compression Ratio (CR) must be greater than 2.8; ii) the activities of data compression inside the Control Data Processing Unit and transmission towards the satellite shall complete in less than 369s, that fits to the acquisition time of the near-infrared instrument; and iii) the compressors parameters as well as the transmission packet size must be tuned to ensure minimal data loss in case of transmission errors. The results obtained with 1D and 2D compression algorithms based on the CCSDS 121 and CCSDS 122 recommended standards, fed with improved focal plane simulations, have been compared to each other. Moreover, a set of various reordering and pre-processing procedures has been applied to the read-out data stream, considering different sizes of the input data segments. The overall scope of these comparative works has been not only to maximize the compression ratio and to minimize the compression time, but also to provide a trade-off between the input data size and the minimum output compressed data segment in order to minimize the data loss due to transmission errors propagation. From our test we found that performing a full (at CCD level) reordering of the read-out data-stream leads to a better compression ratio with both algorithms. The CCSDS 121, however, gives the best results in terms of CR. Finally we found that, for the considered simulated images, the standard pre-processing activities like bias subtraction, bitshift and windowing do not affect the CR significantly. Analogously an additional analysis of the effect of the expected source crowding showed that it is also not important.