Category Archives: Software

General stuff regarding software development.

“It’s more of an art than a science”

I’ll be honest, this phrase bothers me. Perhaps it’s because I’m a scientist by training. Perhaps it’s because this seems to be a misuse of the work ‘art’ or a misinterpretation on my part. But whenever I hear it used with reference to software development, I hear: “we use heuristics and guesswork because we don’t have time to do research and there is no body of research from which to draw”. Does that really make the solution to an underlying question or problem an ‘art’ rather than a science?

I of course tried googling the phrase to determine what it’s supposed to mean, but didn’t get very far. The top result from my search was:

It means it is not something which is governed by clearly-defined rules, as a science would be. In science things are either right or wrong; in psychology (or any art) it’s not possible to say what is ‘right’ or ‘wrong’.

This particular answer seems to misunderstand what science is. In essence, it’s a way to understand the way the world works through experimentation and verification. It’s also typically methodical because reproducibility of research results is important.

Perhaps the reference to rules is based on experience with things like mechanics in Physics and Newton’s laws of motion. We can predict the trajectory of projectiles in the air for example. But this is a very limited view of science. I have been watching the programme Horizon on the BBC recently and learned about the science of taste and even creativity. Yes, we’re learning through science about how creativity works!

At the end of the 19th Century and beginning of the 20th Century, we thought that we would soon learn everything there was to learn about the world. Over a century later, there still seems to be no end to the growth of our scientific knowledge. And things that used to be firmly considered “arts” are much less so now.

Consider cooking: more and more chefs are learning the basic science of both taste and cooking. From that base of understanding they can be even more creative in what they do. It allows people like Heston Blumenthal to create bacon and egg ice cream or snail porridge. If you’re interested, McGee on food and cooking is an essential read on the underlying science.

This also highlights an important point: creativity and science are in no way mutually exclusive. In fact, each enables the other. As I mentioned, a scientific base allows for more creativity because of the deeper understanding of how things work, but creativity is also essential in providing insights into how things work.

Coming back to the original point of this post, my ire was recently raised by a discussion on Hacker News where someone wrote

I’m not sure that there is a sure-fire way to quantify what tests are or are not necessary. In my opinion, this is something that comes with experience and is more of an art than a science. But I’m okay with that.

This seems innocuous enough and I wouldn’t be surprised if many people agree with it. But do we really think that it’s not possible to learn through research what a good level of tests is? Software is typically structured and full of patterns, so the pool of possible structures to investigate is limited. In addition, we already have tools to detect cyclomatic complexity and other metrics of software, so would it be so hard to determine which parts of the software are involved with the “critical” features?

I think what bothers me the most is that despite the huge revenue of the industry as a whole, and how much money depends on the successful completion of IT projects, so little research seems to be done to help improve the software development process. Perhaps the research is being done but it’s not widely disseminated. But I would at least have expected to come across research to back up the claims of agile practitioners (as one example). Not that I necessarily disagree with what they say, but it seems that going agile requires more faith than should be necessary.

Does the software development industry and community require a more scientific mindset? What do you think?

DRY JSON and XML with Grails

Have you ever tried to support both JSON and XML in your REST API with Grails? There is the very straightforward:

class MyController {
    def index = {
        def objs = ...
        withFormat {
            json {
                render objs as JSON
            xml {
                render objs as XML

It works well and is trivial to implement, but it does suffer from a significant problem: you have no control over the JSON or XML generated. So if you’re serialising domain classes, any change to your internal domain model will be reflected in the public REST API. That is Not A Good Thing.

You might then look into using the JSON and XML builders instead, but you will soon discover that they have different syntax and the structure of JSON and XML is different anyway. So you find yourself writing separate code to render JSON and XML for each action that’s part of your REST API. That’s more work than you really want.

Can you somehow marry the two to get the best of both worlds: minimal coding but control over what goes into the public API? If you don’t mind a few constraints on the output, I think you can. My proposal boils down to:

  1. Transform the data in nested maps and lists of maps, filtering out anything you don’t want in the public API
  2. Use ‘... as JSON‘ for JSON responses
  3. Use a custom method for rendering the same data as XML

Let’s take an example from the application. I want to generate a list of plugins in both JSON and XML forms so that consumers have a choice of format. The first step is easy: get a list of the plugins I want to render from the database. The next step involves transforming those plugin domain instances into a hierarchical structure based on maps. Here’s the code I came up with:

class PluginController {
    protected transformPlugins(plugins) {
        return [ pluginList: plugins ?
                plugins.collect { p -> transformPlugin(p) } :
                [] ]

    protected transformPlugin(plugin) {
        def pluginMap = [
                version: plugin.currentRelease,
                title: plugin.title,
                authorEmail: plugin.authorEmail,
                description: plugin.summary,
                grailsVersion: plugin.grailsVersion,
                documentation: plugin.documentationUrl,
                file: plugin.downloadUrl,
                rating: plugin.avgRating ]
        if (plugin.issuesUrl) pluginMap.issues = plugin.issuesUrl
        if (plugin.scmUrl) pluginMap.scm = plugin.scmUrl

        return pluginMap

The dynamic nature of Groovy and its expressiveness make the transformation pretty simple to effect. This could probably be simplified even further if we had something similar to the bindData() method that would map all properties of a domain instance to Map entries except those specified in an exclusion list.

Rendering the plugins as JSON then becomes a simple matter of:

def plugins = Plugin.list(...)
render transformPlugins(plugins) as JSON

since JSON maps perfectly to the map structure I’ve created. XML is a trickier proposition because it doesn’t have a straightforward concept of objects and lists. We could use as XML, but then we would end up with a whole bunch of <map> and <list> elements. No, we have to use a different approach.

I plumped for using the render() method in its XML builder form:

class PluginController {
    protected renderMapAsXml(map, root = "root") {
        render contentType: "application/xml", {
            "${root}" {
                mapAsXml delegate, map

    protected mapAsXml(builder, map) {
        for (entry in map) {
            if (entry.value instanceof Collection) {
                builder."${entry.key}" {
                    for (m in entry.value) {
                        "${entry.key - 'List'}" {
                            mapAsXml builder, m
            else {
                builder."${entry.key}"(entry.value, test: "test")

So now rendering the plugins as XML becomes:

def plugins = Plugin.list(...)
renderMapAsXml transformPlugins(plugins), "plugins"

where the second argument is the name of the root element in the generated XML. The great thing is, this method can be applied to _any_ transformed data as it’s not specific to plugins.

One thing to note is that the mapAsXml() method assumes that the name of any map key that has a list as its value consists of ‘<name>List’. The corresponding XML becomes a parent <nameList> element with nested <name> elements for each of the list elements. This convention simplifies the whole process without being excessively onerous.

So there you are: Don’t Repeat Yourself rendering with JSON and XML. It may not be the most efficient approach computationally, but it will save a fair bit of development and maintenance time. And don’t forget that you always have the option of caching the responses.

Java’s new keyword

Creating objects in Java is easy with the new keyword. In fact, it’s one of those things that you don’t think about. Need to access a file? Just create a new File instance: new File(""). For most Java developers, that’s all they need to know. Life becomes more interesting, though, when you start working with multiple class loaders.

Class loaders? Argh! Run away, run away!

That was pretty much my reaction for many a year. I just didn’t want to know about them. They were some kind of black magic and always Somebody Else’s Problem. It’s strange, because class loaders are actually pretty straightforward. Most Java developers know that you compile Java files to these *.class files and that those compiled classes have to be loaded by the JVM somehow. That’s basically what the class loader does. But like threads, the problem is not understanding what they do, but getting them to work together.

How many times have you heard the phrase “it’s a class loader issue?” I’ve certainly heard (and said) it more times than I’d care to admit. As soon as you have more than one class loader in an application, you have to start worrying about which classes can “see” which others. It can easily become a nightmare. But class loader behaviour is perhaps a post for another time. Let’s get back to new.

So, the first time that you create a new object, the JVM has to first load the class. This happens transparently when you use new. The question is, what class loader is used? And why does it matter?

Consider a scenario from Grails. We have a build system based on Gant that loads build scripts and executes them. In one of them, we instantiate a Jetty server and start it. The sequence of object creation goes like this:
Starting Jetty from the Grails build system
In fact, the above is a simplification of what actually happens, but it suits the purpose of this post.

The JARs for the first three classes are all on the classpath of what we will call the build class loader. This loads all the classes used directly by the build. So what about Jetty’s Server class? The most important thing to understand is that the Server class must be loaded by the same class loader that loads the Grails web application. Although you can pass your own class loader to the embedded server, if it’s different to the one that loads Server you’ll run into those dreaded class loader issues.

Bearing that in mind, let’s look at what happens if the RunApp script uses new to create the server instance:

def server = new org.mortbay.jetty.Server()

Right about now, you should be asking yourself “what class loader was used to load the Server class?” It’s a critical question because it determines what class loader is used to load the entire web application and hence what classpath the application’s runtime dependencies should be on. In this case, the class loader used is whichever one loaded the RunApp script. The new operator effectively delegates to this.getClass().getClassLoader().

What does that mean for our example? It means that the build class loader is used to load the Server class and therefore must also be used to load the web application classes. In other words, all the application’s runtime dependencies must be included in the build class loader! What’s the problem with that, you may ask. There is one potential problem and one actual.

The potential problem is class conflicts. What if the web application depends on a different version of a library that’s already on the build system? It’s a particular problem if any of the Apache XML API libraries are on the classpath. These cause absolute havoc.

The other problem is that the more JARs you have on the classpath, the longer it takes for the JVM to find the class it’s after. That means longer start up times. It’s one of the problems OSGi was designed to solve (he was told by a man in the pub). Why put JARs on the build classpath that the build itself doesn’t need?

The solution is to work out where you want a class loader boundary and use reflection to instantiate your object:

def runtimeClassLoader = new URLClassLoader(...)
def server = runtimeClassLoader.loadClass("org.mortbay.jetty.Server").newInstance()

This is pretty easy in Groovy because the start() method is evaluated at runtime, but Java needs to know the type at compile-time. You can’t do this:

ClassLoader runtimeClassLoader = new URLClassLoader(...)
Server server = (Server) runtimeClassLoader.loadClass("org.mortbay.jetty.Server").newInstance()

because you’ll get a ClassCastException on line 2. The declared type of server is loaded by this.getClass().getClassLoader(), whereas the new instance is loaded in a different class loader. Different class loader means different classes. So you have to use reflection to invoke the methods and access the fields you need. Fortunately, you only have to jump through these hoops at class loader boundaries.

As you’ve seen, the new operator is normally something you don’t have to think about, but as soon as you start dealing with multiple class loaders, you have to be aware of and understand its behaviour. The trick is to work out suitable class loader boundaries and then use reflection to load and instantiate classes at those boundaries. It may sound like unnecessary extra work, but you can gain real improvements in application/framework reliability. If you’re lucky, things may even run a bit faster 🙂

A Tomcat gotcha

Servlet filters. Straightforward, right? A request comes in, goes through each filter in the chain, hits the servlet and then the response goes the other way. Sort of. What could possibly go wrong?

Not much…unless you throw error handling into the mix. The trouble is, the servlet specification doesn’t lay down many rules on how error pages are invoked. My expectation was for the servlet container to transparently offload the request to the error page and then execute the filters in reverse. In other words, the filters shouldn’t be or need to be aware that the request has gone via the error page. The request should behave like any other request.

Tomcat seriously disabused me. It takes a different approach in which the main request almost completes (all the filters execute in reverse) before the request goes to the error page. The following diagram should make it clear what I mean:

Tomcat & Jetty error handling

Tomcat & Jetty error handling

Having read the servlet specification, it does make a bit of sense. One of the few requirements is that the servlet container should pass the original, unwrapped request and response to the error page, unless a filter is configured to execute for the ERROR dispatcher. Unfortunately, if you add something to the request in your filter, or attach a variable to the thread, it won’t be available to the error page if your filter also removes that data.

That may all be a bit too abstract, so how about a concrete example? Some of the filters in Apache Shiro bind an object called the security manager to the request’s thread. Any security code can then easily grab it on demand. When the request has finished, the filter removes (unbinds) the security manager from the thread. That’s just good practice. But what happens if the error page wants to access the security manager?

Hopefully it’s obvious from the discussion so far that you need to add Shiro’s filter to the ERROR dispatcher as well as the REQUEST one. This should work. When the servlet container triggers the error page, it first executes the filter. But that’s not what happened when I tried it. Instead, the error page threw an exception saying that no security manager was bound to the current thread.

What had gone wrong? After some debugging, I discovered that a super class of the filter, OncePerRequestFilter, was skipping execution of the filter if it detected that it had already been processed. How did it do that? By checking whether a particular request attribute was set. In this case, the attribute was still there from the main request, but the security manager had been unbound from the request thread.

This problem didn’t exhibit in Jetty at all, which is kind of annoying because you would hope that code that runs on one container would also run on another. Fortunately, there is a simple moral to this story: filters should always clean up after themselves when they finish and never leave stuff in the request, session, or thread.

Are polyglot systems a good idea?

A recent tweet pointed me to a post on polyglotism by Bill Burke of JBoss. It comes in the midst of a lot of interest in alternative languages on the JVM, such as JRuby, Groovy, Scala, and many others. Its basic point is that companies indulge in multiple languages at their peril and a Java shop should really stick to Java. His view struck me as over the top, but a little reflection made me realise that software teams should take on board his message or risk some serious long term problems.

Now, I’m not a dyed-in-the-wool Java developer who believes Java is the solution to all problems. In fact, I like a profusion of languages on the JVM. Variety is the spice of life as they say. A competitive field ensures that only the strongest survive, and without new languages we would probably still be stuck with C or even assembly language. I want to see new languages that improve my productivity as a developer, enable me to write more reliable code, or simplify the code I write. Even better are languages that give me all three.

So why am I echoing Bill’s warning? Because there is a difference between having a vibrant ecosystem of competing and complementary languages and actually using many of them in a single software project, team, or even company. Here are some of the things you should consider:

Language libraries

Almost every language comes with its own class or function library these days. That’s often the case even when a language runs on a virtual machine such as the JVM or .Net’s CLR. Look at JRuby and Scala: they both run on the JVM but have their own class libraries. So if you use more than one language in a project, you’ll have to know all the various libraries pretty well too.

Context switching

It’s not uncommon to have both Groovy and Java files in a Grails project and developers often have to work in both languages to implement features or fix bugs. That means developers have to switch contexts when they move from one type of file to the other. Even with languages as close in syntax as Groovy and Java, this can cause errors to creep in, for example when you try to iterate over a list in Java using the each() method. Imagine mixing Java and JRuby, where you have different class libraries as well as wildly different syntax.

Certainly a good developer can manage this, and if he or she switches between languages frequently, the cost in time and mistakes will dwindle. But can you guarantee that a developer will always be working in all the required languages at any given time?

Other developers

Just because you can pick up a new language and be proficient with it in a few weeks, that doesn’t mean everyone will be able to do that. If you have a team, you have to take into account the other team members. Realise that some will have trouble learning new concepts, such as functional programming or dynamic languages. Don’t forget as well that using non-mainstream languages shrinks the pool of developers you can recruit from.

It takes time

So, you got the Hello World example up and running followed by the Fibonacci sequence. Great! It only took you a few minutes! Then what? Learning the basics of a language can often be fairly simple, but to become proficient takes time and you have to use it on a regular basis. Can you really afford that time for all the team members on a critical project?

When you leave

As Bill pointed out, development teams don’t always stick together. What happens when you leave? The company is potentially left with a hole they can’t fill because there aren’t any free developers that know all the languages you put into the system.


Last but not least, not all languages play nicely together in a single build. How much effort do you really want to put into making different languages fit into your build system? If it’s not much trouble, then great, but otherwise you could end up with an ongoing maintenance headache.

I don’t mean to dismiss the idea of polyglot projects or systems – a new language may provide real benefits that outweigh the problems. Just make sure that a decision is based on a decent risk assessment or back of an envelope cost-benefit analysis rather than a “hey, that language looks cool, let’s try it out” impulse. Of course, feel free to play around and try new things when you’re working on personal projects!