Using OPUS to perform HST on-the-fly re-processing (OTFR)

The Hubble Space Telescope (HST) OPUS implementation of On-The-Fly Calibration (OTFC) processing currently provides the benefit of applying the most current calibration algorithms, reference files, and repairs of errant header keyword values (aperture, shutter, etc.) to Level-1b datasets as they are retrieved from the HST archive. While OTFC has performed well, a number of concerns about maintenance and flexibility have resulted in the evolution towards an On-The-Fly Re-Processing (OTFR) System. Also based on the OPUS pipeline architecture, OTFR carries further the notion of creating products for archive users at the time of their request, by completely regenerating calibrated (Level-2) data products for an exposure from the base telemetry files sent from the HST (Level-la data). By starting processing at this earlier state, and taking advantage of the changes to the data processing software that are made as that software matures, improved, more consistent calibrated (Level-2) data products are produced. Use of OPUS distributed multiprocessing, the relatively small size of HST datasets, and the efficiency of the data processing and calibration software results in a very small impact on the overall time it takes to complete an archive retrieval. There could be an impact to archive research, however, since the archive catalog meta-data will not completely reflect the reprocessed products as they would be delivered to the archive user. This problem will be addressed by performing catalog updates for any major discrepancies. This paper will describe the concerns raised about OTFC, the design of the OTFR pipeline system, and the benefits of using the OPUS architecture in this design.