ColdFusion Performance and Language Abstractions - Good News For Railo Users

Not too long ago I posted a few entries describing some of the features, changes, and improvements I'd like to see made to ColdFusion as well as some of my concerns/complaints (see http://www.horwith.com/index.cfm/2009/7/16/another-open-letter-to-the-coldfusion-server-team, http://www.horwith.com/index.cfm/2010/1/6/can-coldfusion-handle-enterprise-applications, http://www.horwith.com/index.cfm/2010/5/11/a-few-thoughts-about-various-programming-languages, and http://www.horwith.com/index.cfm/2010/2/25/standardizing-coldfusion). A recurring theme is desire for more control over how the server compiler works and the fact that sometimes the server doesn't do what you want or expect and because it's a closed-box, developers have little or no recourse. A developer friend of mine recently asked me what my point was, exactly, so I put together a few code samples and thought I'd write about them.

My objective in these code samples was to test the efficiency of ColdFusion when using abstractions. The following code illustrates behavior that I believe reveals an inefficiency in the the way that ColdFusion compiles, or uses compiled, code. I have three test pages - all three have a cfscript block containing a 'for' loop. All three were run on ColdFusion 9 on my laptop. The first page loops and does nothing at all, then outputs how long it took to loop 'x' times. On my laptop the average time it takes to loop 1,000,000 times and do nothing is about 150 milliseconds. That code looks like this:

My second test page is exactly the same as the first, but it has a function defined on the page. The function does nothing, and inside the loop I call that function. On my laptop the average time it takes to loop 1,000,000 times and call a function that does nothing is about 700 milliseconds. That code looks like this:

The page that calls a local function takes about 5 times longer on average. Though the immediate reaction might be "Of course - after all, it's calling a function" that's just not right. Better put, it's just not efficient. In simple terms, the server is just a compiler - it turns CF pages into java bytecode and it executes that code to respond to requests. There's no reason why the compiler, after building a parse tree, etc., can't do a better job optimizing the byte code. In other words, the byte code generated for calling the function that does nothing should be identical to the bytecode for doing nothing. Obviously, when code is parameterized and/or has behavior that is dynamic in some way, it's not so straight-forward.
My third test page is exactly the same as the second, but I moved the function to a CFC, create one instance of the CFC before the loop, and inside the loop I call that function on the CFC instance (it's the same function, which does nothing). On my laptop the average time it takes to loop 10,000 times and call a function that does nothing on a CFC instance already in memory, is about 2,000 milliseconds. That code looks like this:

and the CFC code looks like:

So - looping 10,000 times and calling the CFC method averages almost three times as long as looping 1,000,000 times and calling the same function locally. There is a valid argument that it's more difficult for the compiler to optimize the same way for the CFC method because now it's in a shared place (a CFC) so the compiler would have to find every call to this method everywhere on the server and recompile those files in order to replace the method calls with the equivalent byte code - which could be done but it is ugly, yes. That said, I'm still at a loss as to why the performance degrades so much just because the method is being called on a CFC instance. That implies to me that either CFCs are very inefficiently compiled or that the process the server uses for accessing and/or locating methods and variables is not efficient (in LISP this would be a/the Read Table but I'm not quite sure exactly how CF/Java handle it). My CFC method code, when looping 100,000 times, would time out... I didn't bother trying to loop 1,000,000 times and call a cfc method.
I'll state for the record that I'm not an expert compiler-writer but I've read a lot about how compilers work and how to write them. I am an expert software developer by some definitions (I hate to call myself an 'expert' anything, but after doing this for more than 15 years it's fair to say I'm not a novice), which makes me an expert compiler-user. Besides, a compiler is still software, so I think it's fair to say that I'm qualified enough to make theses observations and offer some speculation.

One thing that a skeptic might immediately point-out in opposition to my examples is that they're not practical. After all, isn't it fair to say that you're unlikely to write code that is so static let alone that does nothing at all? I'll admit that encapsulating code that does absolutely nothing IS a stretch - but I'd argue that well-architected code SHOULD contain a good deal of functions that are essentially static. Static in the sense that they either always return the same value or that given the same input they will consistently return the same result with no side-effects. This is one of the characteristics of code that's been written in an expressive style because of the use of abstractions... and it is the staple trademark of applications written in a functional style. Abstractions are one of the most powerful tools in software development (functional or otherwise), and often times the most under-utilized by developers (especially web developers). So, here is a different version of the first and second tests I described earlier - this time the code from both tests does SOMETHING. The first test loops 1,000,000 times and sets a variable equal to the sixth character in a string contained in another variable. On my laptop the average time this takes is 175 ms - barely longer than the time it took to loop 1,000,000 times and do nothing. The code looks like this:

The code in my second test uses a function to abstract the task of retrieving the sixth character from the string. Note that as before, there's no reason why the server shouldn't generate the same byte code as my last test. On my laptop the average time it took to loop 1,000,000 times and set a variable to the return value of a function call that outputs the sixth character from a string is 1,000 ms, which is a comparable degradation to what I saw in the first two tests that did nothing. The code for this second test looks like:

The conclusion I draw from these tests is that using abstractions to build your ColdFusion applications comes at an hefty price. This is unfortunate in that for the sake of performance, a developer is discouraged from expressing the intent of a program with readable reusable code in an optimal way.

Though Railo is not a black box in the same sense as the Adobe ColdFusion Server (since Railo is Open Source), I haven't had time to read the source so for the purposes of this writing let's pretend it is. So what happens when I run the exact same code on Railo on my laptop?
The first test file that loops 1,000,000 times doing nothing averaged 150 milliseconds on CF. On Railo, it averaged 450 ms... more than three times longer. My second test that loops 1,000,000 times and calls a function that does nothing averaged 700 milliseconds with CF. On Railo the average time to run this code was 850 milliseconds... not much longer than CF and certainly no where near the performance degradation I saw between test 1 and test 2 with CF. So I'm beginning to guess Railo is a slightly slower server than Adobe CF but doesn't degrade so poorly. Let's try my last test. On Adobe CF it took 2,000 milliseconds to loop 10,000 times and call a CFC method that does nothing... and the server choked if I tried looping 1,000,000 or 100,000 times. On Railo, looping 10,000 times and calling the CFC method took 24 milliseconds. Yes, that's right - 24 milliseconds! I'm not sure but my assumption is that the Railo server is either taking advantage of multi-threading or isn't building a call stack when accessing the CFC method. Whatever the case, the Railo server appears to reward developers for using abstractions... in a very big way. This is something I intend to research in much more depth, as any compiler/runtime environment that is optimized for code that makes use of abstractions is a much more desirable platform. I should note that looping 100,000 times on Railo and calling the CFC method averaged 430 milliseconds, but Railo also choked on 1,000,000. I also ran my two tests that actually do something and, as I expected, Railo was consistent with its earlier behavior. The inline version, which took CF 175 milliseconds took Railo an average 800 milliseconds. The version which calls a function that does something took ColdFusion an average of 1,000 milliseconds and took Railo an average of 2,000 milliseconds.

I should mention that I kept both application servers running locally during all of the tests and that I used the plain vanilla default installation for both CF and Railo. No CF or JVM settings were modified in any way. I can't explain why Railo appears to be slower than CF when all your code in in one file any more than I can explain why ColdFusion is so inefficient when you use CFCs. Perhaps the application server chosen for a project should be determined by the style and architecture of the code it will run? Or perhaps code should be refactored and/or optimized to take advantage of the Application Server on which they reside? If anyone from the Adobe CF or Railo server development teams cares to comment on my findings, I'm all ears.