Pixie is at Kubecon NA! Learn more ☸️Pixie is at Kubecon NA!
pixie logo
Blog / Pixie Team Blogs

Adding End-to-End Encryption for Proxied DataPermalink

Vihang Mehta

September 21, 2021

End-to-end encryption has become increasingly popular as users demand that any data they send - a file, email, or text message - is not decipherable by any unauthorized recipients. This consumer trend is evident in the recent surge in popularity of Signal, an encrypted instant messaging service.

In this post, we’ll cover what end-to-end encryption is and walk you through how we implemented it in our system.

Why End-to-End Encryption?Permalink

Pixie is designed with a hybrid cloud architecture where data is collected and stored on the customer's environment. The cloud component is used for user management, authentication and proxying data.

This is a simplified architecture diagram of our system before end-to-end encryption.

We use standard security practices to secure data in transit; all network communication between the cluster, proxy and client is TLS encrypted.

But TLS encryption is only point-to-point. When data passes internally through our proxy, the data is temporarily unencrypted. Pixie is an open source project, so users might deploy Pixie Cloud (and the accompanying proxy) in a variety of environments. We wanted to provide privacy guarantees for users given the heterogeneity of deployment scenarios.

By adding end-to-end encryption, we can ensure that the proxy only sees an encrypted form of the telemetry data.

ImplementationPermalink

Pixie provides multiple clients for developers to interact with its platform:

  • a web UI (JavaScript)
  • a CLI (Golang)
  • APIs (client libraries: Golang, Python)

Since we needed to support E2E encryption across multiple languages, using a crypto standard with readily available implementations in multiple languages was a must. Given that we already use JSON Web Token (JWT) for user claims, we chose to look at the IETF proposed JSON Object Signing and Encryption (JOSE) standard for our E2E encryption needs. We settled on using JSON Web Key (JWK) for key exchange and JSON Web Encryption (JWE) as our encryption format.

There are multiple libraries that implement the JOSE spec in different languages. We chose the following:

All three libraries have active communities, are well designed, have thoroughly documented APIs, and contain extensive test suites.

End-to-End Encryption in PixiePermalink

JWE supports a variety of key types and algorithms, however RSA-OAEP seems to be the most widely supported one across the many libraries. So we chose to use 4096 bit RSA keys with the RSA-OAEP encryption scheme across all our clients.

This is how a client interacts with Pixie after enabling end-to-end encryption.

The client generates an asymmetric keypair and sends the public key with any requests for data. Telemetry data is encrypted with the given public key on the cluster. It remains encrypted from the moment it leaves the cluster until it reaches the client.

The asymmetric keypairs are intentionally ephemeral and generated at client creation time and rotated across sessions. This lack of reuse of keys allows an additional layer of protection from any accidentally leaked private keys.

We encrypt all telemetry data. Other message fields currently remain unencrypted within the proxy and are used by the proxy to make routing decisions.

SummaryPermalink

Once we identified the various client libraries we wanted to use, implementing E2E encryption was straightforward. Check out the commits below for implementation details:

  • commit #1: Add encryption support to the Golang Server
  • commit #2: Add key creation & decryption support to the JavaScript UI
  • commit #3: Add key creation & decryption support to the Golang API
  • commit #4: Add key creation & decryption support to the Python API

We hope that the JOSE proposal becomes an IETF standard and this set of libraries and commits acts as a reference for anyone looking to implement E2E encryption in their own project!

Questions? Find us on Slack or Twitter at @pixie_run.

Vihang Mehta

Principal Engineer @ New Relic, Founding Engineer @ Pixie Labs
This site uses cookies to provide you with a better user experience. By using Pixie, you consent to our use of cookies.