PRISPARK: Differential Privacy Enforcement for Big Data Computing in Apache Spark

Differential privacy has emerged as a gold standard privacy definition due to its persuasive mathematical guarantee. While various data protection mechanisms provide differential privacy for SQL queries of RDBMSs, enforcing differential privacy for big data platforms needs to be further researched. This work presents PRISPARK, which enforces differential privacy for Spark - the advanced distributed engine for large-scale data computing in big data ecosystems where sensitive data is often processed. PRISPARK targets to support various data processing (i.e., relational and unstructured queries) on Spark. In particular, to calculate a tighter sensitivity bound and improve the utility of results, we design the overall statistics estimation algorithm for estimating the upper bound of statistics with the filter condition, and propose a novel fine-grained operation-oriented rules set for calculating sensitivity of various relational and unstructured queries. Moreover, we propose a general differential privacy mechanism, PRISPARK, a suite including PRISPARKSQL and PRISPARKDAG. We enforce PRISPARKSQL at the Catalyst optimization layer for relational queries in Spark SQL and PRISPARKDAG at the RDD execution layer for unstructured queries in Spark core. Finally, we experimentally evaluate PRISPARK on TPC-H, TPC-DS, PigMix benchmarks, and real-world dataset LANL. The experimental results suggest that PRISPARK supports various applications/queries while improving the utility of all query results by orders of magnitude with negligible performance overhead.