CineGrid Exchange: A workflow-based peta-scale distributed storage platform on a high-speed network

The Academy of Motion Picture Arts and Sciences (AMPAS) report "The Digital Dilemma" describes the issues caused by the rapid increase of storage requirements for long-term preservation and access of high quality digital media content. As one of the research communities focusing on very high quality digital content, CineGrid addresses these issues by building a global-scale distributed storage platform suitable for handling high quality digital media, which we call CineGrid Exchange (CX). Today, the CX connects seven universities and research laboratories in five countries, managing 400TB of storage, of which 250TB are dedicated to CineGrid. All of these sites are interconnected through a 10 Gbps dedicated optical network. The CX distributed repository holds digital motion pictures at HD, 2K and 4K resolutions, digital still images and digital audio in various formats. The goals of the CX are: (1) providing a 10 Gbps interconnected distributed platform for the CineGrid community to study digital content related issues, e.g., digital archiving, the movie production process, and network transfer/streaming protocols; (2) building a tool with which people can securely store, easily share and transfer very high definition digital content worldwide for exhibition and real-time collaboration; (3) automating digital policies through middleware and metadata management. In this publication, we introduce the architecture of the CX, resources managed by the CX and the implementation of the first series of CX management policies using the iRODS programmable middleware. We evaluate the first phase of CX platform implementation. We show that the CX has the potential to be a reliable and scalable digital management system. Published by Elsevier B.V.