Detect SQL injections with Pixie
October 14, 2021
End-to-end encryption has become increasingly popular as users demand that any data they send - a file, email, or text message - is not decipherable by any unauthorized recipients. This consumer trend is evident in the recent surge in popularity of Signal, an encrypted instant messaging service.
In this post, we’ll cover what end-to-end encryption is and walk you through how we implemented it in our system.
Pixie is designed with a hybrid cloud architecture where data is collected and stored on the customer's environment. The cloud component is used for user management, authentication and proxying data.
We use standard security practices to secure data in transit; all network communication between the cluster, proxy and client is TLS encrypted.
But TLS encryption is only point-to-point. When data passes internally through our proxy, the data is temporarily unencrypted. Pixie is an open source project, so users might deploy Pixie Cloud (and the accompanying proxy) in a variety of environments. We wanted to provide privacy guarantees for users given the heterogeneity of deployment scenarios.
By adding end-to-end encryption, we can ensure that the proxy only sees an encrypted form of the telemetry data.
Pixie provides multiple clients for developers to interact with its platform:
Since we needed to support E2E encryption across multiple languages, using a crypto standard with readily available implementations in multiple languages was a must. Given that we already use JSON Web Token (JWT) for user claims, we chose to look at the IETF proposed JSON Object Signing and Encryption (JOSE) standard for our E2E encryption needs. We settled on using JSON Web Key (JWK) for key exchange and JSON Web Encryption (JWE) as our encryption format.
There are multiple libraries that implement the JOSE spec in different languages. We chose the following:
All three libraries have active communities, are well designed, have thoroughly documented APIs, and contain extensive test suites.
JWE supports a variety of key types and algorithms, however RSA-OAEP seems to be the most widely supported one across the many libraries. So we chose to use 4096 bit RSA keys with the RSA-OAEP encryption scheme across all our clients.
The client generates an asymmetric keypair and sends the public key with any requests for data. Telemetry data is encrypted with the given public key on the cluster. It remains encrypted from the moment it leaves the cluster until it reaches the client.
The asymmetric keypairs are intentionally ephemeral and generated at client creation time and rotated across sessions. This lack of reuse of keys allows an additional layer of protection from any accidentally leaked private keys.
We encrypt all telemetry data. Other message fields currently remain unencrypted within the proxy and are used by the proxy to make routing decisions.
Once we identified the various client libraries we wanted to use, implementing E2E encryption was straightforward. Check out the commits below for implementation details:
We hope that the JOSE proposal becomes an IETF standard and this set of libraries and commits acts as a reference for anyone looking to implement E2E encryption in their own project!