Export Pixie data in the OpenTelemetry format.Learn more🚀
Pixie has an OpenTelemetry plugin!
​
Ctrl/Cmd + K
Adding End-to-End Encryption for Proxied Data
Vihang Mehta
September 21, 2021 • 2 minutes readPrincipal Engineer @ New Relic, Founding Engineer @ Pixie Labs
End-to-end encryption has become increasingly popular as users demand that any data they send - a file, email, or text message - is not decipherable by any unauthorized recipients. This consumer trend is evident in the recent surge in popularity of Signal, an encrypted instant messaging service.
In this post, we’ll cover what end-to-end encryption is and walk you through how we implemented it in our system.
Why End-to-End Encryption?
Pixie is designed with a hybrid cloud architecture where data is collected and stored on the customer's environment. The cloud component is used for user management, authentication and proxying data.
This is a simplified architecture diagram of our system before end-to-end encryption.
We use standard security practices to secure data in transit; all network communication between the cluster, proxy and client is TLS encrypted.
But TLS encryption is only point-to-point. When data passes internally through our proxy, the data is temporarily unencrypted. Pixie is an open source project, so users might deploy Pixie Cloud (and the accompanying proxy) in a variety of environments. We wanted to provide privacy guarantees for users given the heterogeneity of deployment scenarios.
By adding end-to-end encryption, we can ensure that the proxy only sees an encrypted form of the telemetry data.
Implementation
Pixie provides multiple clients for developers to interact with its platform:
- a web UI (JavaScript)
- a CLI (Golang)
- APIs (client libraries: Golang, Python)
Since we needed to support E2E encryption across multiple languages, using a crypto standard with readily available implementations in multiple languages was a must. Given that we already use JSON Web Token (JWT) for user claims, we chose to look at the IETF proposed JSON Object Signing and Encryption (JOSE) standard for our E2E encryption needs. We settled on using JSON Web Key (JWK) for key exchange and JSON Web Encryption (JWE) as our encryption format.
There are multiple libraries that implement the JOSE spec in different languages. We chose the following:
- jose for JavaScript (imported as @inrupt/jose-legacy-modules for compatibility with our tooling)
- lestrrat-go/jwx for Golang
- Authlib for Python (notably, this library successfully handles messages that include null bytes)
All three libraries have active communities, are well designed, have thoroughly documented APIs, and contain extensive test suites.
End-to-End Encryption in Pixie
JWE supports a variety of key types and algorithms, however RSA-OAEP seems to be the most widely supported one across the many libraries. So we chose to use 4096 bit RSA keys with the RSA-OAEP encryption scheme across all our clients.
This is how a client interacts with Pixie after enabling end-to-end encryption.
The client generates an asymmetric keypair and sends the public key with any requests for data. Telemetry data is encrypted with the given public key on the cluster. It remains encrypted from the moment it leaves the cluster until it reaches the client.
The asymmetric keypairs are intentionally ephemeral and generated at client creation time and rotated across sessions. This lack of reuse of keys allows an additional layer of protection from any accidentally leaked private keys.
We encrypt all telemetry data. Other message fields currently remain unencrypted within the proxy and are used by the proxy to make routing decisions.
Summary
Once we identified the various client libraries we wanted to use, implementing E2E encryption was straightforward. Check out the commits below for implementation details:
- commit #1: Add encryption support to the Golang Server
- commit #2: Add key creation & decryption support to the JavaScript UI
- commit #3: Add key creation & decryption support to the Golang API
- commit #4: Add key creation & decryption support to the Python API
We hope that the JOSE proposal becomes an IETF standard and this set of libraries and commits acts as a reference for anyone looking to implement E2E encryption in their own project!
Questions? Find us on Slack or Twitter at @pixie_run.
Related posts
Terms of Service|Privacy Policy
We are a Cloud Native Computing Foundation sandbox project.
Pixie was originally created and contributed by New Relic, Inc.
Copyright © 2018 - The Pixie Authors. All Rights Reserved. | Content distributed under CC BY 4.0.
The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage Page.
Pixie was originally created and contributed by New Relic, Inc.
This site uses cookies to provide you with a better user experience. By using Pixie, you consent to our use of cookies.