Adding End-to-End Encryption for Proxied Data
September 21, 2021
In this blog post we will discuss:
Pixie is an observability platform recently acquired by New Relic. To make the project more accessible to the developer community, New Relic is open sourcing Pixie. As part of our commitment to OSS, we want Pixie to be deployable in a completely open source environment. However, Pixie had a strong dependency on Auth0 that would complicate this effort.
In this post we detail how we redesigned Pixie's auth to be open source compatible. We discuss trade offs of several approaches and dive into our final open source implementation. Let's start with our Auth0 design.
Adding Auth0 to Pixie was very simple. Integrating Auth0's API took less than a day and our implementation continues to work two years later. Auth0 has been incredibly easy to use throughout the entire experience.
In our original authentication scheme (figure below), Pixie's UI redirects to Auth0 and receives an access token back. The UI forwards the token to the Pixie Cloud backend and the backend validates the token by making a call to the external Auth0 server.
Auth0 does all the heavy lifting and provides an easy way to setup different login providers - we went with Google-based Signup and Login flows. During signup, users could give us access to their name and profile picture. We incorporated this into our small profile dropdown, rounding out the user experience.
The Auth0 + Google flow was simple and worked extremely well for us. Auth0 saved development time that we later spent on our core observability product. We would recommend Auth0 to anyone who is comfortable with a hosted, third party solution.
Unfortunately, Auth0 does not work for Pixie's open source offering. We don't want to require open source users to depend on remotely hosted, closed source APIs. This is especially important for our users with air-gapped clusters that cannot make external network requests. We need a standalone version of Pixie that exclusively relies on open source components.
As part of this change, we also needed to enable username and password login. Our hosted implementation requires an identity from a third-party provider (ie Google) - another dependency that breaks our open source commitment. Username / password, on the other hand, can be implemented with open source libraries and works regardless of a cluster's network connectivity.
We considered three designs to support open sourced authentication in Pixie.
This option is compelling, but would limit our users to only those willing and able to open an Auth0 account. It also adds friction to get Pixie Cloud up and running, while also preventing network-isolated Pixie Cloud deployments. We want to open up as much access to Pixie Cloud as possible, so option 1 alone won't satisfy our needs.
While open source projects commonly implement authentication from scratch, we're wary of spending time rolling our own solution. Pixie, like all development teams, has an engineering budget that we allocate across our products. Rolling our own auth would require us to spend a large chunk of that budget on this feature, while taking on the risk of opening security vulnerabilities if done incorrectly. We would rather rely on a project that has been built by experts and has been vetted by the open source community.
Outsourcing logic to third-party libraries is often easier said than done. Using an off-the-shelf solution sounds appealing, but integration into an existing project requires time to learn the new system and adapt it to your specific needs. The upside is that you inherit the working knowledge of the library's developers and save yourself many pains of edge-case discovery.
In this case, we felt the benefits outweighed the disadvantages. Security is extremely important to us because we deal with sensitive user data. It was worth the effort to redesign our system to use an existing solution written by security/authentication experts.
We found that a combination of Ory's Kratos and Hydra projects best fit our needs. Many other alternatives don't provide username/password authentication, and instead rely on OAuth/OIDC. Others restrict users to pre-built UIs. Kratos and Hydra together gave us a flexible and full solution that fit our needs while also remaining accessible to anyone in the open source community.
Before we jump into our implementation, let's quickly discuss Kratos and Hydra's capabilities.
At a high level, Ory's Kratos is an open source system for identity and user management. Kratos supports registration, login, authentication, and other user-related tasks, all through a convenient REST API. At the moment, Pixie only uses Kratos for username/password-based authentication, but Kratos also supports login with third-party identity providers through OIDC (ie Google Identity Platform).
Unlike our Auth0 use-case, Kratos uses stateful session cookies rather than stateless access tokens to store identity information. The most important difference is that session cookies require a service roundtrip to check authorization, while access tokens like JWTs can be verified at the endpoint, reducing the latency necessary to access an authenticated endpoint. (You can learn more about the difference here).
Our auth design incorporates JWTs for their performance advantage, but Kratos only provides the session-cookie option. We fill this gap with Hydra.
Hydra, another project by Ory, is an open source OAuth 2.0 server and OpenID Connect provider. Hydra does not come packaged with identity or user management - instead it completely focuses on enabling API access to the OAuth flows and expects users to come with their own identity solutions (such as Kratos). An added benefit is that the project is very mature and also in production at several notable companies. We use Hydra in Pixie Cloud to create the stateless access tokens we use for auth in the rest of our backend. However, we had to do some work to convert the Kratos session cookies into the access tokens.
We document our design here for reference in case future readers also want an access token based, open source, auth solution.
The Hydra/Kratos integration is complex, so it's easiest to demonstrate with an example. Let's break down a successful login flow. The steps match the diagram above.
/loginendpoint in Pixie's API service.
With the Hydra/Kratos implementation for authentication, the following users will be able to use Pixie:
However, we still appreciate the benefits that Auth0 provides:
Given the benefits, we will still support Auth0 in Pixie. Pixie users will have the choice of configuring their deployment with Auth0 or relying on Kratos and Hydra. We reduced the auth solution to a simple interface, as shown in the following diagram.
We then implemented the interface with the Kratos/Hydra logic from above.
We are big fans of both Auth0 and Ory's Hydra/Kratos. Each comes with their own sets of trade-offs and we offer both options to enable our users to choose the use case that works best for them. Most importantly, both solutions are vetted by security experts and hardened by their wide user-base. Pixie's development team can spend less time worrying about security and instead focus on making our no-instrumentation observability platform better. We hope this blog post will help you decide which auth solution works best for you.