Secure multi-tenant stream processing

Details

Supervisors

Riviere, Etienne

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en cybersécurité, à finalité spécialisée: conception et analyse de systèmes

Abstract

enModern stream processing engines, such as Apache Flink, enable the processing of large volumes of data in real time within distributed environments. However, in many collaborative scenarios involving multiple organizations, these streams contain sensitive data, and processing them on untrusted cloud infrastructure raises major privacy concerns. Traditional approaches that rely solely on encryption protect data at rest and in transit but do not preserve its confidentiality whilst processing is underway. This thesis addresses the problem of collaborative processing of sensitive streams belonging to multiple tenants in an untrusted distributed environment. To address this issue, we propose a hybrid architecture combining a distributed stream processing engine with Intel Software Guard Extensions (SGX) secure enclaves. The aim of this approach is to retain the advantages of modern stream processing engines for the distributed orchestration of streams, whilst delegating operations handling sensitive data to secure components running within enclaves. Our architecture is based on a clear separation between the distributed infrastructure and confidential processing. Apache Flink continues to handle the orchestration, routing, and parallelization of data streams, whilst sensitive operations, e.g., joins between streams from distinct tenants, are executed exclusively within SGX enclaves after the data has been decrypted in it. We also offer an architecture that separates the responsibilities for processing orchestration from those for processing sensitive data, as well as a configurable operator that enables the dynamic integration of secure collaborative stream processing. A working prototype based on Apache Flink and Intel SGX has been developed to experimentally evaluate this architecture. The results show that it is possible to integrate confidential collaborative processing into a distributed stream processing pipeline without running the entire engine within an enclave. However, the evaluation highlights the costs of secure execution, particularly latency, sustained throughput, and state management in complex collaborative processing. Despite these limitations, the results demonstrate the practical feasibility of a hybrid architecture enabling the confidential processing of multi-tenant streams whilst maintaining compatibility with existing distributed infrastructures.