The future of Groovy

With the recent revelation that Pivotal will stop sponsoring the Groovy and Grails projects after March of this year, I want to give some of my own thoughts on where this leaves the two projects and what their future might hold. I’ll do this across two blog posts, starting with Groovy.

It may come as a surprise, but I feel pretty sanguine about Groovy. That’s not to say that I don’t sympathise with the teams for the situation they find themselves in, but Groovy itself is in a strong position. First and foremost, it’s stable with a solid set of features already there. Recent additions include:

  • Traits
  • Static type checking and compilation
  • Dedicated Android support

As far as I’m concerned, Groovy doesn’t need any major changes in the near term and so it’s not a big deal if its development slows down. It supplements Java well and provides enough differentiation that it’s worth you using Groovy over Java. It’s great for scripting, Spock is fantastic for unit testing, and Groovy works really well with Spring Boot, Ratpack, Grails, Griffon and other application frameworks. And those are just a fraction of its potential use cases.

The language also benefits from an existing open development process, probably because it originated as a collaborative project between people from different companies. So even if the Groovy team is broken up – something I hope can be avoided – I can’t see there being significant disruption. You’re seeing one of the key benefits of open source software: it survives beyond the withdrawal of a key sponsor. You can’t really say that about commercial software. And while Groovy satisfies a need among developers, you can be sure it will thrive and prosper.

To sum up, Groovy has reached a stable place that puts it in good stead whatever the future might hold. It should still be an essential part of every Java developer’s tool chest. Development may or may not slow down, but it will continue. And I have my fingers crossed that Guillaume, Cédric, and Jochen will find new homes swiftly and I hope they manage to stay together as a team.

One final thought: on the last Groovy Podcast (audio-only version) I half-jokingly said that Google would sponsor Groovy. Of course, I don’t know whether that will happen, but in light of Swift for iOS development, I think it’s something that Google should definitely consider. I’m sure a more productive and expressive alternative to Java would be more than welcome within the Android development community and it fits nicely with their use of Gradle for the build system. Let’s see what happens!

The Grails post may take a bit longer to appear as I think there’s more to say for that.

Why Gradle?

Back in my youth, programs used to come as listings in magazines that you copied into the ZX81’s BASIC editor and then ran. How times have changed. Building software has become more complex as the underlying runtimes and platforms have evolved. As a result of that, we have seen an evolution in build tools. I started my working life using Make (for both C++ and Java), before progressing onto Apache Ant and Apache Maven 2. For a brief interlude I even worked with CMake, which I found to be one of the more challenging tools to learn and understand.

Gradle is a recent entrant to the field, and my current build tool of choice. What lifts it above the more established tools, Ant and Maven? This is a question that anyone involved in building Java software in particular should be asking themselves.
Continue reading

Code reuse in micro services

It seems to me that one of the big questions around micro services at the moment is how to share code between the services that make up the overall system. This is important because you should be able to deploy individual services independently of others. So what can you share and how should you do it?

I don’t have a straight answer, but I do have a suggestion for how to think about the problem. Imagine that each service is implemented by teams in different companies, each with their own private source repositories. How would you share code then? For me, the answer is through shared dependencies (JARs of compiled classes in the case of Java). And those shared dependencies should probably only contain utility code such as that provided by commons-lang and Guava (Java examples again).

I guess shared client API JARs could work too if you don’t want to code against JSON/XML directly. The key code that _shouldn’t_ be shared is that related to the internal model of the service. The public API though is not internal.

So, is this a reasonable or even helpful way to look at this issue?

Adding Spring Security users in bulk (in Grails)

Earlier this week I was at a client trying to help them diagnose some issues with a CSV import process. I was of course aware of Ted Naleid’s seminal blog post on bulk updates in Grails, and the issues mentioned there seemed the most likely culprits. Unfortunately, it didn’t turn out nearly as straightforward as I’d hoped.

We started with VisualVM, looking for any obvious problems with the memory usage. Nothing showed up, and in fact the import wasn’t creating a lot of records anyway. We progressed on to JProfiler and P6Spy, hoping to see some hotspots in the code or particularly slow queries. We did identify a couple of places that seemed to be taking the majority of the time, but it still wasn’t clear to me whether the issue was in code, Grails, Hibernate, or the database.

That day we implemented a workaround that shifted some work into a background thread using the Platform Core event bus. This was a reasonable thing to do anyway, considering the requirements of the business logic. Yet I was left still wondering why certain parts of the import process were fundamentally slow.

It bugged me enough that I decided to investigate one of the major culprits of the slow import: the Spring Security Core plugin’s UserRole.create() method. Perhaps I could reproduce the problem in a small project without the complexity of the client project. It seemed simple enough to be worth a try. And so I created a new Grails 2.3.7 project with Spring Security Core installed and the following controller action:

@Transactional
def createUsers()
    def startTime = System.currentTimeMillis()
    
    def roles = (0..<60).collect {
        new Role(authority: RandomStringUtils.randomAlphabetic(12)).save()
    }
    
    def users = (0..<100).collect {
        new User(
                username: RandomStringUtils.randomAlphabetic(12),
                password: RandomStringUtils.randomAscii(20)).save()
    }
    
    for (r in roles) {
        for (u in users) {
            UserRole.create u, r, false
        }
    }

    println "Total: ${(System.currentTimeMillis() - startTime) / 1000} s"

    redirect uri: "/"
}

To my relief, this took more than 30 seconds to complete on the first run. That seemed a lot slower than it should considering it’s only creating a total of 760 records. There was obviously some underlying issue here that I wasn’t seeing. I tried to clear and flush the session every 20 iterations, but that didn’t have a significant impact.

My next step was to simply create 760 Role records and then, independently, 760 User records. Both of these only took a few seconds. So what was special about UserRole? Why did its creation seem to be so expensive? I wanted to eliminate the database as a problem, so I tried using Groovy SQL (basically native JDBC) for the UserRole persistence. The total time dropped to a few seconds. So not the database then.

A Google search brought up another blog post on inserting data via Grails, by Marc Silverboard. In addition to using native JDBC, he suggests using a Hibernate stateless session. This sounded like an interesting possibility, so I shoe-horned it into my test action:

def createUsers()
    ...
    def session = sessionFactory.openStatelessSession()
    def tx = session.beginTransaction()
    def counter = 0
    for (r in roles) {
        for (u in users) {
            session.insert(new UserRole(user: u, role: r))
            counter++
            if (counter % 20 == 0) {
                session.flush()
            }
        }
    }
    tx.commit()
    session.close()
    ...
}

It’s certainly uglier code, partly as I decided to do batch flushing every 20 rows (I also configured Hibernate’s JDBC batch size to 20). The results were worth it: the total import time came down to just 1 second! Obviously the issue was Hibernate’s caching in the session. Conundrum solved. I was still left wondering why the caching was such an issue only for UserRole, but that was a question for another time.

It would have been easy to stop at this point and bask in the glow of a job well done. Unfortunately, that’s not really me. With my engineering background, I did wonder whether the new code was bypassing more than just Hibernate’s caching. And then I remembered validation. Could validation be the real issue? In order to isolate that particular feature, I reverted all the code back to its original state and then modified the UserRole.create() method to use the validate: false option. I restarted the server and then clicked on the link that triggered the user creation. 10 seconds! I did it again. 6 seconds. After a few more times it settled down at just under 4 seconds. Wow.

Why is validation such an issue on UserRole? I have no idea. I did give deepValidate: false a go, but it didn’t show nearly as big an improvement as switching off validation completely. Maybe one of my readers understand what’s going on and can provide us with the answer. Or perhaps not knowing will bug me enough to get me looking deeper. But for now, I just want to summarise my findings:

  • Grails and Hibernate have a lot of moving parts – it can be hard to diagnose issues
  • You really do need to invest some quality time and rigour for any diagnosis phase
  • Hibernate stateless sessions are work investigating for any bulk inserts
  • Validation could be a significant hidden problem – try disabling it

I think in this case, the drop from 4s to 1s may make it worthwhile using a stateless session. But in either case, be sure you can do without the validation! And I hope this helps you with your own GORM bulk insertions, either with the diagnosis or the solution.