Continuous Feedback

Fixing performance problems can be tricky. I joined a new team last spring, and my first assignment was to investigate and fix some performance problems they were having.

Most of the reported performance problems were related to a complicated function with many moving parts.

I started with creating some test data that I could use to measure the current performance to have something as a baseline to compare my changes to.

I then used a combination of manually inspecting the code and running a profiler to try and pinpoint the performance bottlenecks in the code.

After a few days of work, we managed to locate and fix several issues that negatively affected the performance, most related to database access through Hibernate. We found that the same data was read multiple times from the database and had to be cached. It turned out that Hibernate produced N+1 queries due to misconfiguration of the entities. We solved one issue by adding a missing table index.

Manually finding these issues is doable but very work-intensive. There must be a better way. Our applications can give us lots of information we can utilize by enabling OpenTelementry, something I wrote about in my previous article. Observability for Agile Development: Why It’s a Game Changer

Digma is a tool that analyzes OpenTelemetry data to give us continuous feedback from our applications. I got involved as a beta-tester around the same time as I was working with the performance issues I described above, so I could directly see the benefit of having a tool to help me identify problem areas.

With Digma, we get continuous feedback on the performance of the application. Digma will instantly show us where the bottlenecks are, so we don’t have to search for them ourselves. We can now get direct feedback on how our latest fix performs compared with the previous version. It also found bottlenecks that we did not even know about. It takes the guesswork out of the picture. We save lots of time finding and fixing performance issues.

The tool is free to use locally for developers. All they ask is that we register so that they can get information about how many users they have. It consists of a plugin for your IDE and a backend running in containers. No data is exposed externally, so there should be no issue with security or privacy. The best thing is that the installation is super easy. The plugin will enable OpenTelemetry automatically for many common frameworks and application servers.

There is also an option to do a central installation of Digma to get even more benefits, but there is a licensing fee. In this article, I will share my experience working with a local installation and my thoughts on why you should consider doing a central installation. Let’s get into it!

Local environment

Digma comes bundled with Jaeger, so we can get a visualization of collected tracing data directly without leaving the IDE. We can navigate between code and traces. We can follow the flow of a request and see things like parameters and queries.

Tracing data is automatically collected when you run or debug your code in the IDE or running tests. It gives us continuous feedback on how our code is performing and if the latest change has introduced any new issues.

A new feature shows you which tests have touched a particular part of your code. This feature makes it possible to run the tests affected by the changes we are making, saving us time and providing faster feedback on the issue we are working on.

It will automatically detect many of the issues I mentioned in my example above. This feature alone saves lots of time when we no longer have to locate where the problem lies. But can focus on fixing it instead. I estimate that we could have solved the performance problems in the example above in 1/10 of the time by using this tool. It is a huge deal and the main reason I’m writing this article. We should work smarter, not harder.

Make sure you try with your application. Digma is free, easy to set up, and will boost our productivity.

Doing a central installation will allow us to take full advantage of Digma.

CI/Testing environment

It will be possible to monitor our releases over time by connecting our CI environment to Digma. OpenTelemetry has to be enabled on our product to make this possible. We will then have the option to detect changes over time. It can tell us if the performance has increased or decreased over time. We can run performance suites to detect scaling issues.

We now have the option to compare the results from the code running on the CI environment with the one we currently are working on. We get feedback on whether our latest change fixes a performance problem even before submitting the code to the repository. There is no need to wait for the build and deploy process to complete.

It can do a usage analysis to find unreached code. It can also find code reached by many flows so that we know that changes to that will affect many flows.

Production environment

By connecting our production environment to Digma, we can get real-world data about your code and issues currently in the wild. It allows us to be proactive instead of reactive.

This step requires that your customer is allowing OpenTelemetry to be collected and that there is a way for developers to connect their IDE to the central Digma instance. We have to be sensitive to their concerns regarding privacy and security.

We will now have a way to compare code running in production with code currently being developed. We can also get information about problems as they happen and even where they occur.

How impressive would it look to identify and fix a problem and have it deployed into production before our customer notices it? With Digma, this is possible!

Please visit https://digma.ai to learn more about this tool.