Using dynamic kernel instrumentation for kernel and application tuning

The authors have designed a new technology-fine-grained dynamic instrumentation of commodity operating system kernels-that can insert runtime-generated code at almost any machine code instruction of an unmodified operating system kernel. This technology is ideally suited for kernel performance profiling, debugging, code coverage, runtime optimization, and extensibility. They have written a tool called Kernlnst that implements dynamic instrumentation on a stock production Solaris 2.5.1 kernel running on an UltraSparc CPU. They have written a kernel performance profiler on top of Kernlnst. Measuring kernel performance has a two-way benefit: it can suggest optimizations to both the kernel and applications that spend much of their time in kernel code. In this paper, the authors present their experiences using Kernlnst to identify kernel bottlenecks when running a Web proxy server. By profiling kernel routines, they were able to understand performance bottlenecks inherent in the proxy's disk cache organization. The authors used this understanding to make two changes-one to the kernel and one to the application-that cumulatively reduce the percentage of elapsed time that the proxy spends opening disk cache files for writing from 40% to 7%.