Virtual Event
May 4 - May 7
Learn More and Register to Attend

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for KubeCon + CloudNativeCon Europe 2021 Virtual to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.

Please note: This schedule is automatically displayed in Central European Summer Time (UTC +2). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date." The schedule is subject to change.

Back To Schedule
Friday, May 7 • 11:50 - 12:25
High Throughput with Low Resource Usage: A Logging Journey - Eduardo Silva, Calyptia

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
In Logging, there is a common fact: more applications means more data to handle. Running services at scale in a distributed environment brings exciting challenges for data management, but with the volume of data increasing there is a necessity to ship this data faster, but a few ones realize the side effect: high resource consumption. On implementing a logging pipeline, pre-processing of the data is mandatory, a simple example of this is Kubernetes metadata enrichment for every log record, but more data means more computing time, the same cost applies when delivering to the final storage or cloud service. In this session, we will do a deep dive into our journey of performance challenges that we faced in the Fluent Bit project around Network I/O + TLS, filesystem buffers, routing, and multiplexing for high throughput. We will share how did we go from 5k/sec to more than 30k/sec using a single-core CPU using purely design improvements and taking the most of Linux OS interfaces.

avatar for Eduardo Silva

Eduardo Silva

CEO & Founder, Calyptia
Eduardo is an entrepreneur and Software Engineer. He is currently one of Fluentd project maintainers and creator of Fluent Bit, a lightweight Logs and Metrics processor. He also is the founder of Calyptia (the Fluent company).

Friday May 7, 2021 11:50 - 12:25 CEST
Performance Theater