None of these tools are [sic] new — this stuff’s been around for a while.

While some of the ideas have been kicking around for a while, plenty of DTrace is new, as outlined in length in the Related Work section of our paper at USENIX ’04. Yes, we’re familiar with the offerings on z/OS like GTF and OmegaMon and yes, DTrace advances the state of the art beyond them — significantly beyond them in some dimensions. So contrary to popular belief, everything wasn’t actually done by z/OS or MULTICS in the 1960s…

Strongly disagree. Debate about the relative differences of DTrace and SystemTap should take place not on one list or the other, but a separate (newly created if need be) higher-level list somewhere — e.g. observability tools.

Eric

I have seen lots of people using DTrace fixing problems. Systemtap has come a long way in the last year; I’m wondering has anybody solved a problem or tuned an application, found bottlenecks with Systemtap. So I’m putting out a request for people using Systemtap to show how Systemtap made there life easier, blog it or post it somewhere and a link to the systemtap mailing list. These could be very motivational to the community, perhaps they could even be made part of the official webpage.

In a short time span SystemTap is able to provide significant functionality not by accident but due to it’s flexible and extensible architecture. We are doing extensive testing to disallow probes only in a very few places where it is not safe, as per our design goal. The bugs Bryan mentioned in his comment are not architectural flaws but our effort to isolate areas where probes are not allowed to the smallest possible set.

I couldn’t agree more with Stephen that the value to the customer is in providing tools on top of the infrastructure. I think flexible architecture of SystemTap that has ability to develop higher level tools by including the lower level tapsets has an advantage. I also agree with Stephen that adoption of these tools and technologies in the customer base can be accelerated with automated script generation using a GUI front end.

We recently load-tested a Sun T2000 server in our environment. (This is the Niagara-based server, 8 cores with 4 threads per core.) Our first test was on an in-house multi-threaded SMTP server. Where we expected to see 6-8 times performance over our current hardware, we barely saw 3x, even though the server was still mostly idle. The obvious thought was that it was a lock contention problem, and the most likely candidate was the mutex around the shared lock file. The standard procedure would have been to increase the debugging level, log a lot of information, and analyze it afterwards. Given that the log file was a candidate for lock contention, the results of that analysis might not have been conclusive, because the extra logging itself could have masked the problem.

I performed a very simple analysis with two DTrace scripts. The first told me what function the application was spending most of its time in (__lwp_mutex_timedlock(), for the curious), and the second gave me the most common stack traces leading to that particular function. The lock contention ended up being in libc’s implementation of malloc() and free(). It hadn’t occurred to anyone to even think about this as a possibility, and the standard method of adding extra logging probably wouldn’t have caught this. (For the curious, Solaris 10′s libumem provided the silver-bullet solution to the performance problem, letting the Niagara perform as expected but also getting us a 70% improvement on the existing hardware.)

There’s a second case with a similar analysis that pointed to lock contention in readdir_r(), but the details would essentially be redundant. What is important to point out, though, is that we were performing this analysis on live production systems under real conditions. We were comfortable doing so given the safety features of DTrace, and doing so allowed us to spend a few hours analyzing the problem instead of spending much longer setting up a lab environment in which we could generate an equivalent amount of traffic.

As for SystemTap, there’s a major sticking point with respect to using it in our environment (which is a mixture of Solaris and Linux.) As far as I understand, SystemTap involves generating C code and compiling that C code into a kernel module. Given the requirement for a C compiler, we’ll never be able to use SystemTap on production systems, as we do not install a C compiler on our production systems. It’s feasible that we would change that policy if SystemTap could provide some truly outstanding benefit, but it’s far more likely that we would run Solaris x86 in order to use DTrace. While there are architectural arguments to be made with respect to this issue, this is an operations argument that I would expect to be an issue in many environments.

I will not start a technical debate between DTrace and Systemtap. Not at all. I would add two, three points about what we have right now in DTrace and how our community works.

We have a nice collection of D scripts, the DTraceToolkit , what Brendan mentioned previously. He has created a nice toolkit which contains over 100 scripts based on DTrace which are tested and documented. We are working to enhance the toolkit and better document the scripts for our user base. The scripts can be very easily used then in any companies around: telcos or banks to simplify the use of DTrace.

The DTrace community is a big team I would say: we have engineers working on DTrace internals, we have folks which are writing tools to easy the administration based on D scripts and we have people which document and test all these things.

I think SystemTap is not here…maybe later. And when they will get to this point should I think is this a progress ?

To start with, there has been much marketing hype about DTrace, and (suprisingly) it is quite well justified. DTrace has changed the way I think about OSes, and raised the bar for what I expect to be able to measure and achieve. If a DTrace-like framework is written for other OSes, it will also be immensely valuable, so long as saftey is carefully respected.

In previous lives as a system administrator, there were times when I really needed to measure something, and was suprised that the system couldn’t do so – or do so easily. DTrace solved all those long-term limitations in one blow. Not only that, but it has provided the means for me to measure just about any behaviour imaginable – so long as I really want to (and will sit down and figure out the code). Other OSes now feel absurdly restricted to their own suite of status tools.

I agree that documentation is an important question – and that people need to be able to use this framework for the value to be realized. For some people that involves writing DTrace scripts – especially people with a heavy programming background.

I wrote the DTraceToolkit for those who don’t have the time or the background to sit down and hammer out DTrace scripts themselves. It’s giving people instant value from DTrace, providing useful information that was previously not available. I’m also working on the documentation to go with the toolkit – which I’m a believer is an integral part for success (the toolkit is really 3 parts: 1. the scripts; 2. the man pages; 3. the example documentation).

The DTraceToolkit can be found at: http://www.opensolaris.org/os/community/dtrace/dtracetoolkit.

Other documentation sources for DTrace include the DTrace Guide – which is online and an excellent reference; and the upcoming release of “Solaris Internals” 2nd edition, which contains many useful DTrace scripts, demonstrated and explained. (I also helped write the book: for info see Richard’s blog).

It’s difficult to comprehend the effect DTrace will have on operating system analysis; although I’ve written hundreds of scripts, I feel I’ve only scratched the surface. I’m going to be quite busy in the coming years.