Part 2: A Simple Continuous Cross-Language (Go, Rust, C/C++) Profiler written in eBPF
June 01, 2021
We are thrilled to announce that Pixie has officially been open sourced by New Relic. Pixie is an in-cluster observability platform for Kubernetes. It's designed to be a low friction tool for developers to debug and monitor their applications.
As a quick background for those new to the project, here are Pixie's three most important capabilities:
Manually adding instrumentation to existing codebases can be a burden for teams. Pixie provides immediate, significant baseline visibility into the target system. Once deployed, it automatically collects full-body application requests from a variety of protocols, system metrics, and network-level data. Pixie's auto-instrumentation is powered by eBPF, a Linux kernel technology popularized by Brendan Gregg.
As developers, we wanted Pixie to be a fully programmatic interface so that it can better fit into our own workloads. Pixie uses a Pythonic query language called PxL, based on Pandas syntax. All of Pixie's clients (CLI, API, and web UI) use PxL scripts to analyze data. Pixie ships with a rich set of PxL scripts out of the box, but users can also write their own PxL scripts to perform custom analysis. PxL also serves as the interface for importing and exporting Pixie data to other systems.
Shipping large amounts of telemetry data to remote data stores often introduces a significant burden on the network as well as privacy concerns when the data is sensitive. Pixie performs all data storage and computation entirely on the user’s Kubernetes cluster. This architecture allows the user to isolate data storage and computation within their environment for finer-grained context, faster performance, and a greater level of data security.
With today's release, it is now possible to run an entirely self-hosted version of Pixie without third-party dependencies or vendor lock-in.
Here is a summary of the major components we have made available:
Users can choose to self-host Pixie entirely, or to run Vizier in conjunction with Pixie Cloud hosted by New Relic to reduce management burden. New Relic-hosted Pixie will remain entirely free, and users can choose to send data to New Relic One.
The blog and website assets look similar today, but we expect them to diverge over time as the OSS project develops.
Our vision for Pixie is to build a ubiquitous data platform for application infrastructure. We hope that developers will build new applications that use Pixie data in ways we haven't thought of yet. In terms of building a community around Pixie, it was important to make Pixie accessible to any developer using Kubernetes. In order to support these goals, we decided to open source the project. New Relic boldly supported this decision as part of acquiring Pixie Labs in December 2020. After the acquisition, New Relic also committed to ensuring that the entire Pixie Labs team remain 100% focused on the Pixie project.
Here are three decisions we made in order to preserve the integrity of Pixie as an open source project:
We believe that by contributing Pixie as a truly open source project to the community, we can maximize the impact it has. We hope to see it power entirely new applications build on top of the data we collect.
Here are some materials to get started with our OSS version of Pixie:
As mentioned earlier, docs for our hosted solution can be found at pixielabs.ai.
We would like to thank all of our users for their feedback and help in building Pixie. Big thanks to our advisors, Kelsey Hightower and Jaana Dogan, as well as the entire Pixie team. Thanks to New Relic for supporting our open source vision for Pixie. Finally, thank you to Brendan Gregg for his trailblazing work with eBPF.