High-performance computing blogs like "horse-race" journalism, especially when covering competing architectures like CPUs and GPUs. One commonly hears of 20x or even larger speedups when porting a CPU code to a GPU. Recently someone pointed out to me, though, that one rarely hears of the "reverse port": taking an optimized GPU code, and porting it back to the CPU. Many GPU optimizations relate to memory access, and many scientific codes spend a lot of time reading and writing memory. For example, coalescing loads and stores (for consecutive threads on a GPU warp) corresponds roughly to aligned, contiguous memory accesses on a CPU. These are more friendly to cache lines, and are also amenable to optimizations like prefetching or using wide aligned load and store instructions. CPUs are getting wider vector pipes too, which will increase the branch penalty. From what I've heard, taking that GPU-optimized code and porting it back to the CPU might result in only a 2x slowdown over the GPU.
I don't see this as bad, though. First, GPUs are useful because the hardware forces programmers to think about performance. After investing in learning a new programming language (such as CUDA or OpenCL) or at least a new library (such as Thrust), and after investing in new hardware and in supporting GPU runtimes in their applications, coders are obligated to get return on that investment. Thus, they take the time to learn how to optimize their codes. GPU vendors help a lot with this, and expose performance-related aspects of their architecture, helping programmers find exploitation points. Second, optimizing for the GPU covers the future possibility that CPUs and GPUs will converge. Finally, programmers' dissatisfaction with certain aspects of GPU hardware may cause divergence rather than convergence of GPUs and CPUs, or even the popularization of entirely new architectures. (How about a Cray XMT on an accelerator board in your workstation, for highly irregular data processing?)
04 September 2010
25 August 2010
"Archetype by prototype": a suggestion for making compile-time polymorphism more usable
A recent post on interface matching in ParaSail (a new parallel programming language under development) inspired me to think about compile-time polymorphism in languages like C++. The author of the article observed that "C++ templates define essentially no 'contract' so there is no easy way to find out what sort of type will be acceptable without trying the instantiation."
This stirred me to think about what C++ programmers can (and do) do to get around this. A standard technique is to use "concept checks." These test the required syntax by, well, trying the instantiation ;-) in a way that makes it easy to catch syntax errors. They can be tedious, because doing them right requires two things:
1. Concept checking code, that verifies the interface statically and as simply as possible (so that compiler error messages are easier to read)
2. An "archetype class": like a Java interface, except with (skeletal) implementations of the required interface
In the above, i'm using the terminology of the Boost Concept Check Library. #1 alone adds to the code maintenance burden, since as the "contract" evolves, you have to synchronize the checks with how you are actually using the object. #2 alone doesn't ensure synchronization of your idea of the contract, with the classes you think are implementing it. Nevertheless, there are still four separate bodies of code to synchronize: the archetype class, the concept checking code, the application class(es), and the application code that invokes these class(es). The most straightforward (but not easiest) way to avoid this burden would be to
1. Write an archetype class, and make the compiler check the concept, using application code
I call this approach "declared archetypes." It calls for a syntax for declaring that a particular concrete data type implements an archetype's interface. This approach has implications for avoiding error-prone tedium. A common case for templates is generic code for different kinds of numbers. I've written a lot of code templated on "Scalar", which could be anything representing a real or complex number, and Ordinal, which represents an array index type. It would be tedious to declare an archetype supporting all the different kinds of things one might like to do with a Scalar, for example. It would at least have to include all of arithmetic, and all kinds of transcendental functions (such as trigonometric functions and logarithms). That would be an awful lot of effort, especially since most people are only going to instantiate Scalar with "float" or "double". (A few unlucky folks will make Scalar a complex number and discover all the places where the supposedly generic code was using less-than on Scalar!)
To avoid this tedium, the "concepts" proposal considered for addition to the C++ standard (but rejected) offered a shorthand for "aggregate archetypes." You could say that Scalar is "FloatingPointLike" or that Ordinal is "SignedIntegralLike", and that would hopefully bring in the syntax that you want. You could also "inherit" from archetypes to create your own.
The trouble with this approach is that somebody has to write a huge library of arbitrarily named predefined archetypes. Huge, because pretty much anything you might want to do with a "plain old data" type has to have its own concept. Arbitrarily named, because somebody has to decide what "ArithmeticLike" means. (Does it mean integers or real numbers?) This path seems to call for abstract algebra, where you have semigroups and rings and fields of a certain characteristic, and all kinds of things that nonmathematicians don't want to understand. (I'm convinced the main reason why people don't use Haskell more is because it's so hard to explain what a monad is, and why you want to know.)
This is overkill because in most cases, programmers can accomplish their work without so much formality. The ParaSail blog post alludes to the reason why: "... when ad hoc matching is used, there is a presumption that the module formal (or target type) is very stable and new operations are not expected to be added to it." The typical use case of C++ templates (for example) is for things that "look like double" or "look like int." That suggests a second approach:
2. "Archetype by prototype" or "looks like type T"
If T is something simple like "double" or "int", then you save a lot of syntax and / or library writing. If T is complicated, this forces you to write an archetype class, which is probably a good thing if T is complicated! For the numerical linear algebra algorithms i write and maintain, this would help remind me whether an algorithm (that claims to be generic on a Scalar type) can handle complex numbers. (Complex arithmetic changes linear algebra in subtle ways that don't only have to do with syntax.)
This approach does not require special syntax for defining archetypes. However, it might still be nice to have such a syntax, and also to have a small library of archetype classes. I could see this being useful for iterators, for example.
"Archetype by prototype" would impose requirements "lazily," much like users of C++ template metaprogramming expect. The "actual archetype" would consist only of those operations with the concrete datatype that are actually written in code. For example, if you never ask for the cosine of a Scalar, you don't need to have an implementation of cosine that takes a Scalar argument. It would be nice to have some development environment support for "extracting" the "actual archetype." C++ compilers do some of this already in their error messages, if you have the patience to read them (after about a page of compiler backtrace, you get something like "no implementation of cos(Scalar) exists for Scalar = YourType").
In summary, i propose "archetype by prototype" as an alternative to "concepts" for making compile-time polymorphism more usable.
This stirred me to think about what C++ programmers can (and do) do to get around this. A standard technique is to use "concept checks." These test the required syntax by, well, trying the instantiation ;-) in a way that makes it easy to catch syntax errors. They can be tedious, because doing them right requires two things:
1. Concept checking code, that verifies the interface statically and as simply as possible (so that compiler error messages are easier to read)
2. An "archetype class": like a Java interface, except with (skeletal) implementations of the required interface
In the above, i'm using the terminology of the Boost Concept Check Library. #1 alone adds to the code maintenance burden, since as the "contract" evolves, you have to synchronize the checks with how you are actually using the object. #2 alone doesn't ensure synchronization of your idea of the contract, with the classes you think are implementing it. Nevertheless, there are still four separate bodies of code to synchronize: the archetype class, the concept checking code, the application class(es), and the application code that invokes these class(es). The most straightforward (but not easiest) way to avoid this burden would be to
1. Write an archetype class, and make the compiler check the concept, using application code
I call this approach "declared archetypes." It calls for a syntax for declaring that a particular concrete data type implements an archetype's interface. This approach has implications for avoiding error-prone tedium. A common case for templates is generic code for different kinds of numbers. I've written a lot of code templated on "Scalar", which could be anything representing a real or complex number, and Ordinal, which represents an array index type. It would be tedious to declare an archetype supporting all the different kinds of things one might like to do with a Scalar, for example. It would at least have to include all of arithmetic, and all kinds of transcendental functions (such as trigonometric functions and logarithms). That would be an awful lot of effort, especially since most people are only going to instantiate Scalar with "float" or "double". (A few unlucky folks will make Scalar a complex number and discover all the places where the supposedly generic code was using less-than on Scalar!)
To avoid this tedium, the "concepts" proposal considered for addition to the C++ standard (but rejected) offered a shorthand for "aggregate archetypes." You could say that Scalar is "FloatingPointLike" or that Ordinal is "SignedIntegralLike", and that would hopefully bring in the syntax that you want. You could also "inherit" from archetypes to create your own.
The trouble with this approach is that somebody has to write a huge library of arbitrarily named predefined archetypes. Huge, because pretty much anything you might want to do with a "plain old data" type has to have its own concept. Arbitrarily named, because somebody has to decide what "ArithmeticLike" means. (Does it mean integers or real numbers?) This path seems to call for abstract algebra, where you have semigroups and rings and fields of a certain characteristic, and all kinds of things that nonmathematicians don't want to understand. (I'm convinced the main reason why people don't use Haskell more is because it's so hard to explain what a monad is, and why you want to know.)
This is overkill because in most cases, programmers can accomplish their work without so much formality. The ParaSail blog post alludes to the reason why: "... when ad hoc matching is used, there is a presumption that the module formal (or target type) is very stable and new operations are not expected to be added to it." The typical use case of C++ templates (for example) is for things that "look like double" or "look like int." That suggests a second approach:
2. "Archetype by prototype" or "looks like type T"
If T is something simple like "double" or "int", then you save a lot of syntax and / or library writing. If T is complicated, this forces you to write an archetype class, which is probably a good thing if T is complicated! For the numerical linear algebra algorithms i write and maintain, this would help remind me whether an algorithm (that claims to be generic on a Scalar type) can handle complex numbers. (Complex arithmetic changes linear algebra in subtle ways that don't only have to do with syntax.)
This approach does not require special syntax for defining archetypes. However, it might still be nice to have such a syntax, and also to have a small library of archetype classes. I could see this being useful for iterators, for example.
"Archetype by prototype" would impose requirements "lazily," much like users of C++ template metaprogramming expect. The "actual archetype" would consist only of those operations with the concrete datatype that are actually written in code. For example, if you never ask for the cosine of a Scalar, you don't need to have an implementation of cosine that takes a Scalar argument. It would be nice to have some development environment support for "extracting" the "actual archetype." C++ compilers do some of this already in their error messages, if you have the patience to read them (after about a page of compiler backtrace, you get something like "no implementation of cos(Scalar) exists for Scalar = YourType").
In summary, i propose "archetype by prototype" as an alternative to "concepts" for making compile-time polymorphism more usable.
24 June 2010
"The limits of my language define the limits of my world"
John Rose, speaking of programming languages, quotes the Logisch-philosophische Abhandlung of
Ludwig Wittgenstein: "The limits of my language define the limits of my world" (Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt). John nicely links to the German version of Wittgenstein's work, which gives me an lets me practice my rusty German!
The phrase recalled a question a coworker asked me today: "If I could choose the programming language I use for my work, what would it be?" I answered that it depends on the work I want to do: if I need to have a dialogue with the operating system and with low-level hardware details, C(++) is the tool of choice (mainly for ill); for numerical computations, some midway point between (modern) Fortran and Matlab would be nice. C is more expressive than Fortran, I went on to say. "Doesn't that mean C is better than Fortran?" he asked? No, it doesn't mean that. Fortran's restrictiveness (esp. its rules about pointer declarations and aliasing) make it easier for compilers to generate efficient code. C's expressiveness makes it easier to control the computer at a low level.
Wittgenstein's observation applies here: in C, everything looks like a pointer. In Fortran, everything looks like an array (C doesn't have real multidimensional arrays, remember!). In Smalltalk, everything looks like an object. Lisp programmers fear writing domain-specific languages much less than Java programmers. SISAL programmers fear returning an (apparently) freshly created array much less than C programmers. SQL programmers fear complicated searches over tables of data much less than programmers of other languages, etc. By choosing the language, I choose the way in which I attack programming problems.
If the language suggests a programming model (or a small set of them), then if a new programming model suggests itself, the rational response is to create a new language or modify an existing one in a way that naturally leads the programmer to work within the model. Programming models ultimately reduce to our assumptions about computer hardware. Where are these assumptions going? It's not quite a Turing machine anymore. We have to extend our hardware assumptions to include unreliability: segfaults, kernel panics, nodes of a cluster going down and up, the occasional bad bit in memory or on disk.
"Cloud" programming models ("everything is a reduction pass from disk to CPU and back to disk again") already handle nodes crashing to some extent, but it doesn't look like programming models for handling unexpected bit flips have gotten past the discussion phase. In part this is because the hardware hasn't presented us with a programming model other than "compute and pray": even fixed addresses in read-only code pages aren't invulnerable, as the above link explains. It's fair to assume, though, that the hardware will let us distinguish between "accurate" and "inaccurate" data or computations, just like it lets us distinguish between IEEE 754 float and IEEE 754 double.
This distinction between accurate and possibly inaccurate computations and data will enter the programming model. Languages have type systems for that sort of thing; it doesn't seem much different than a "const" annotation in C++, or a "synchronized" annotation in Java. This attribute needs to be orthogonal to the actual datatype; the typical "oh well, floating-point numbers are inexact anyway" viewpoint will lead to disaster in an LAPACK LWORK query, for example. The programming model will have to include promotions and demotions; these will have costs, of which the programmer must be made aware (just like with Java's "synchronized").
What are the implications of such a model? First, programmers will have yet another type annotation to learn. Just like with C's "volatile," it will lead to endless confusion, especially as hardware evolves. Compilers can help with type deduction, but programmers will be tempted to use the "inaccurate" annotation as a magic "go faster" switch. If computer science students can't wrap their heads around tail recursion in Scheme, how can we expect them to understand yet another low-level annotation?
Second, programmers' view of the hardware memory space will grow even more fragmented. We already have "GPU memory" and "CPU memory" (for OpenCL and CUDA programmers), "local" and "remote" memory (for those working in a Partitioned Global Address Space (PGAS) model), caches and local stores and "shared memory" (which is really a cache) and vector registers and... In the future, we may have this times two: for each, an "accurate but slow" and an "inaccurate but fast" version. Going from one to the other will require copying -- which may be implicit (as in a PGAS-like model), but still costs something. Relaxing bit accuracy is a hardware performance optimization and so if you don't want to declare everything "accurate," then you likely are worried about the costs of copying things around. This means you will have to reason about all of those fragmented memory spaces.
Given this likely state of hardware, I can rephrase my colleague's question: "What programming model would I like to use if the hardware behaves in this way?" First, the PGAS model seems most natural: it lets me reason about different memory spaces if I need to, or program as if there is only one memory space (at potentially higher cost) if it helps me get my work done. Second, the model should help programmers distinguish between "data" and "metadata." Metadata (indices, permutations, pointers) should only be bit accurate. (That doesn't mean integers always have to be! Integer types get used a lot for signal processing and other bit-oriented domains.) Third, programmers should have to handle metadata as little as possible. The less metadata you handle explicitly, the less likely you are to "pollute" it with inexactness, and the more freedom the compiler has to perform optimizations on the metadata. That means the language should support all kinds of collections. Operations on collections should be fast, otherwise programmers won't want to use them. Fourth, by extension of the third point, the language should prefer sequence operations ("map f over the array v") that let programmers avoid naming variables that don't really matter, such as index variables and intermediate quantities in loops. (If you name it, the compiler has to keep it around unless it is clever enough to reason it away. Storage necessitates annotation of its accuracy, which necessitates conservativism in optimizations, in the same way that C aliasing rules force the C compiler to be conservative.)
This post rambles, but I'm thinking out loud, and I'm curious to hear what others have to say about programming models for this new world of unexpected flipping bits. The one thing that encourages me is the intelligence of those thinking about the problem. As Shakespeare has Miranda say in The Tempest, "How beauteous mankind is! O brave new world, That has such people in't!"
Ludwig Wittgenstein: "The limits of my language define the limits of my world" (Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt). John nicely links to the German version of Wittgenstein's work, which gives me an lets me practice my rusty German!
The phrase recalled a question a coworker asked me today: "If I could choose the programming language I use for my work, what would it be?" I answered that it depends on the work I want to do: if I need to have a dialogue with the operating system and with low-level hardware details, C(++) is the tool of choice (mainly for ill); for numerical computations, some midway point between (modern) Fortran and Matlab would be nice. C is more expressive than Fortran, I went on to say. "Doesn't that mean C is better than Fortran?" he asked? No, it doesn't mean that. Fortran's restrictiveness (esp. its rules about pointer declarations and aliasing) make it easier for compilers to generate efficient code. C's expressiveness makes it easier to control the computer at a low level.
Wittgenstein's observation applies here: in C, everything looks like a pointer. In Fortran, everything looks like an array (C doesn't have real multidimensional arrays, remember!). In Smalltalk, everything looks like an object. Lisp programmers fear writing domain-specific languages much less than Java programmers. SISAL programmers fear returning an (apparently) freshly created array much less than C programmers. SQL programmers fear complicated searches over tables of data much less than programmers of other languages, etc. By choosing the language, I choose the way in which I attack programming problems.
If the language suggests a programming model (or a small set of them), then if a new programming model suggests itself, the rational response is to create a new language or modify an existing one in a way that naturally leads the programmer to work within the model. Programming models ultimately reduce to our assumptions about computer hardware. Where are these assumptions going? It's not quite a Turing machine anymore. We have to extend our hardware assumptions to include unreliability: segfaults, kernel panics, nodes of a cluster going down and up, the occasional bad bit in memory or on disk.
"Cloud" programming models ("everything is a reduction pass from disk to CPU and back to disk again") already handle nodes crashing to some extent, but it doesn't look like programming models for handling unexpected bit flips have gotten past the discussion phase. In part this is because the hardware hasn't presented us with a programming model other than "compute and pray": even fixed addresses in read-only code pages aren't invulnerable, as the above link explains. It's fair to assume, though, that the hardware will let us distinguish between "accurate" and "inaccurate" data or computations, just like it lets us distinguish between IEEE 754 float and IEEE 754 double.
This distinction between accurate and possibly inaccurate computations and data will enter the programming model. Languages have type systems for that sort of thing; it doesn't seem much different than a "const" annotation in C++, or a "synchronized" annotation in Java. This attribute needs to be orthogonal to the actual datatype; the typical "oh well, floating-point numbers are inexact anyway" viewpoint will lead to disaster in an LAPACK LWORK query, for example. The programming model will have to include promotions and demotions; these will have costs, of which the programmer must be made aware (just like with Java's "synchronized").
What are the implications of such a model? First, programmers will have yet another type annotation to learn. Just like with C's "volatile," it will lead to endless confusion, especially as hardware evolves. Compilers can help with type deduction, but programmers will be tempted to use the "inaccurate" annotation as a magic "go faster" switch. If computer science students can't wrap their heads around tail recursion in Scheme, how can we expect them to understand yet another low-level annotation?
Second, programmers' view of the hardware memory space will grow even more fragmented. We already have "GPU memory" and "CPU memory" (for OpenCL and CUDA programmers), "local" and "remote" memory (for those working in a Partitioned Global Address Space (PGAS) model), caches and local stores and "shared memory" (which is really a cache) and vector registers and... In the future, we may have this times two: for each, an "accurate but slow" and an "inaccurate but fast" version. Going from one to the other will require copying -- which may be implicit (as in a PGAS-like model), but still costs something. Relaxing bit accuracy is a hardware performance optimization and so if you don't want to declare everything "accurate," then you likely are worried about the costs of copying things around. This means you will have to reason about all of those fragmented memory spaces.
Given this likely state of hardware, I can rephrase my colleague's question: "What programming model would I like to use if the hardware behaves in this way?" First, the PGAS model seems most natural: it lets me reason about different memory spaces if I need to, or program as if there is only one memory space (at potentially higher cost) if it helps me get my work done. Second, the model should help programmers distinguish between "data" and "metadata." Metadata (indices, permutations, pointers) should only be bit accurate. (That doesn't mean integers always have to be! Integer types get used a lot for signal processing and other bit-oriented domains.) Third, programmers should have to handle metadata as little as possible. The less metadata you handle explicitly, the less likely you are to "pollute" it with inexactness, and the more freedom the compiler has to perform optimizations on the metadata. That means the language should support all kinds of collections. Operations on collections should be fast, otherwise programmers won't want to use them. Fourth, by extension of the third point, the language should prefer sequence operations ("map f over the array v") that let programmers avoid naming variables that don't really matter, such as index variables and intermediate quantities in loops. (If you name it, the compiler has to keep it around unless it is clever enough to reason it away. Storage necessitates annotation of its accuracy, which necessitates conservativism in optimizations, in the same way that C aliasing rules force the C compiler to be conservative.)
This post rambles, but I'm thinking out loud, and I'm curious to hear what others have to say about programming models for this new world of unexpected flipping bits. The one thing that encourages me is the intelligence of those thinking about the problem. As Shakespeare has Miranda say in The Tempest, "How beauteous mankind is! O brave new world, That has such people in't!"
23 May 2010
Post to deter spammers
I need to post something here to deter spammers, since they seem to be targeting idle blogs. I'll think of something Real Soon Now (tm) to post about generic APIs for parallelism, or something like that.
28 November 2009
James Hardy Wilkinson, FRS
I love this photo of James Hardy Wilkinson,
the great numerical analyst and computer scientist. Something not so
commonly known is that Wilkinson's ability to get along with Alan Turing helped bring the Automatic Computing Engine, which was designed by Turing, to fruition.
21 July 2009
C is too high level
I propose that C is too high level of a language for the purposes for which it's used. While it purports to give programmers power over memory layout -- in particular, over heap (via malloc) and stack (via alloca) allocation -- it gives them no control, and does not allow them to describe, how function arguments or struct fields are laid out in memory. Knowing how struct fields are laid out means you can use structs from languages other than C, in a portable way. (Some foreign function interfaces, such as ANSI Common Lisp's CFFI, refuse to allow passing structs by value for this reason.) You can know exactly how much memory to allocate, and bound more tightly from above how much stack space a particular C function call requires.
Given the state of C as it is, what I would like is a domain-specific language for describing binary interfaces, such as struct layout or function call signatures. I would like C compilers to support standard annotations that guarantee particular layouts. Currently this is done in an ad hoc, compiler-specific way -- sometimes by command-line flags ("pack the structs") and sometimes by pragmas or annotations.
The main reason I want standard compiler support for such a minilanguage is for interoperability between C and other languages. I have the misfortune of needing to call into a lot of C libraries from my code, but I don't want to be stuck writing C or C++ (The Language Which Shall Not Be Parsed). Nevertheless, I don't want to tie any other users of my code to a particular C compiler (if the code were just for me, it wouldn't matter so much).
Given the state of C as it is, what I would like is a domain-specific language for describing binary interfaces, such as struct layout or function call signatures. I would like C compilers to support standard annotations that guarantee particular layouts. Currently this is done in an ad hoc, compiler-specific way -- sometimes by command-line flags ("pack the structs") and sometimes by pragmas or annotations.
The main reason I want standard compiler support for such a minilanguage is for interoperability between C and other languages. I have the misfortune of needing to call into a lot of C libraries from my code, but I don't want to be stuck writing C or C++ (The Language Which Shall Not Be Parsed). Nevertheless, I don't want to tie any other users of my code to a particular C compiler (if the code were just for me, it wouldn't matter so much).
01 July 2009
Python win: csv, sqlite3, subprocess, signal
I've been working these past few months (ugh, months...) on a benchmark-quality implementation of some new parallel shared-memory algorithms. It's a messy, tempermental code that on occasion randomly hangs, but when it works, it often outperforms the competition. Successful output consists of rows of space-delimited data in a plain text format, written to standard output.
I spent a week or so on writing a script driver and output data processor for the benchmark. The effort was both minimal, and paid off handsomely, thanks to some handy Python packages: csv, sqlite3, subprocess, and signal.
The csv ("comma-separated values") package, despite its name, can handle all kinds of text data delimited by some kind of separating character; it reads in the benchmark output with no troubles. sqlite3 is a Python binding to the SQLite library, which is a lightweight database that supports a subset of SQL queries. SQL's SELECT statement can replace pages and pages of potentially bug-ridden loops with a single line of code. I use a cute trick: I read in benchmark output using CSV, and create an SQLite database in memory (so I don't have to worry about keeping files around). Then, I can issue pretty much arbitrary SQL queries in my script. Since I only use one script, which takes command-line arguments to decide whether to run benchmarks or process the results, I don't have to maintain two different scripts if I decide to change the benchmark output format.
The subprocess and signal packages work together to help me deal with the benchmark's occasional flakiness. The latter is a wrapper around the POSIX signalling facility, which lets me set a kind of alarm clock in the Python process. If I don't stop the alarm clock early, it "goes off" by sending Python a signal, interrupting whatever it might be doing at the time. The subprocess package lets me start an instance of the benchmark process and block until it returns. "Block until it returns" means that Python doesn't steal my benchmark's cycles in a busy loop, and the alarm means I can time out the benchmark if it hangs (which it does sometimes). This means I don't burn through valuable processing time if I'm benchmarking on a machine with a batch queue.
I wish the benchmark itself were as easy to write as the driver script was! I've certainly found it immensely productive to use a non-barbaric language, with all kinds of useful libraries, to implement benchmarking driver logic and data processing.
I spent a week or so on writing a script driver and output data processor for the benchmark. The effort was both minimal, and paid off handsomely, thanks to some handy Python packages: csv, sqlite3, subprocess, and signal.
The csv ("comma-separated values") package, despite its name, can handle all kinds of text data delimited by some kind of separating character; it reads in the benchmark output with no troubles. sqlite3 is a Python binding to the SQLite library, which is a lightweight database that supports a subset of SQL queries. SQL's SELECT statement can replace pages and pages of potentially bug-ridden loops with a single line of code. I use a cute trick: I read in benchmark output using CSV, and create an SQLite database in memory (so I don't have to worry about keeping files around). Then, I can issue pretty much arbitrary SQL queries in my script. Since I only use one script, which takes command-line arguments to decide whether to run benchmarks or process the results, I don't have to maintain two different scripts if I decide to change the benchmark output format.
The subprocess and signal packages work together to help me deal with the benchmark's occasional flakiness. The latter is a wrapper around the POSIX signalling facility, which lets me set a kind of alarm clock in the Python process. If I don't stop the alarm clock early, it "goes off" by sending Python a signal, interrupting whatever it might be doing at the time. The subprocess package lets me start an instance of the benchmark process and block until it returns. "Block until it returns" means that Python doesn't steal my benchmark's cycles in a busy loop, and the alarm means I can time out the benchmark if it hangs (which it does sometimes). This means I don't burn through valuable processing time if I'm benchmarking on a machine with a batch queue.
I wish the benchmark itself were as easy to write as the driver script was! I've certainly found it immensely productive to use a non-barbaric language, with all kinds of useful libraries, to implement benchmarking driver logic and data processing.
Subscribe to:
Posts (Atom)
