Where next for Grails?

A time comes for every open source project when it has to take a step back, reflect on the past and decide where it needs to go next. The world rarely stays the same as when the project was born and in the tech world things change year on year. Back when Grails first came out, Java web frameworks were still a pain to configure and you had to pull together a whole host of libraries to do what you wanted. The use of conventions was rare and ease of use didn’t seem to be a high priority.

Now in 2013 there are probably hundreds of web frameworks for the JVM, most of which have learned lessons from Rails and its ilk. In addition, we have Node.js spawning lookalikes and derivatives (such as Meteor) aiming to provide lightweight servers that can handle hundreds of thousands of concurrent connections through asynchronous IO. In this world, what is Grails’ value proposition and where should it be heading?

This is a pretty long post, so if you want to skip straight to my suggestions, head on down to the conclusion at the end of the article. But I’m hopeful you’ll find value in the rest of the article!

Let’s look at where Grails stands right now. It’s a rapid web application development framework based on the MVC pattern, servlets, and data stores. Once installed, you can have an application running in under a minute. You don’t have to know SQL in order to use a database (although it does help a lot) and you can, with limitations, use the same API (GORM) with NoSQL databases. In fact, most of the components of Grails are now plugins even if you can’t uninstall or replace all of them.

Here’s a quick run-down of what you get:

  • Data access – GORM (Hibernate, MongoDB, DynamoDB, Neo4J, etc.)
  • Transactions – services, @Transactional, withTransaction {}
  • Business logic – services, domain classes, command objects
  • HTTP – data binding, URL mappings, content negotiation, caching
  • JSON/XML – JSON/XML serialisation/deserialisation, automatic request parsing into params
  • HTML templates – GSPs, layouts, tag libraries
  • Static resource management – Resources plugin
  • Security – Codecs, form tokens, access control plugins (Spring Security, Shiro, etc.)
  • Integration – Spring, REST client plugins, messaging plugins
  • Build – custom, integrated build system
  • Testing – unit test harness, integration testing mode, functional testing plugins
  • Data migrations – database migration plugin
  • RAD – automatic reloading, Groovy, DSLs

That’s a pretty extensive list. Much is based on existing Java libraries, but even when that’s the case, there is a significant amount of Grails-specific code involved. And all the parts are designed to work together. For traditional web applications, you have pretty much everything you need at your fingertips, and when you need something that’s not provided out of the box, there is usually a plugin to help.

So, who’s the target audience for this? Beginners? Experts? Somewhere in between? Perhaps both? Is it a RAD MVC framework? Or a platform for building all sorts of web applications? Is it designed to be performant? Or scalable? Is TDD required or encouraged? Is it supposed to work well within larger projects and teams? Or it is mainly for 1-5 person startups? That’s a lot of questions and I could probably come up with even more.

I think that Grails is currently defined more by what it provides than by what it’s for. What any project needs, just like any business, is a vision. Normally there is a vision for a project at it’s inception, because that provides the drive for it’s creation. Years down the line though, that vision needs to be refreshed. Look at how Apple changed it’s vision to focus on easy-to-use, desirable consumer devices. Google is trying to refocus and develop it’s own vision for what it wants to do. Yahoo! seems to have lacked one for some time now and is suffering the consequences.

A vision helps focus development and motivate the participants. It also let’s users know where the project is going so that they can make informed decisions about whether to use it or not. Or where to contribute, if they’re eager to give something back. Such a vision was recently articulated by Adrian Colyer of SpringSource in a blog post. To quote:

Grails 3.0 will separate Grails from the traditional application server and extend Grails’ reach to allow for the development of lightweight, asynchronous applications. Grails’ persistence technology GORM has also been evolving beyond the traditional relational database, with implementations for NoSQL databases now available. GORM will continue to be an important technology for us as enterprise data fabrics evolve.

Is this enough of a vision? Is it the right vision?

The truth is, there are many different ways to write web applications these days. More and more of them have rich JavaScript user interfaces that simply communicate with the server over REST APIs using JSON. “Realtime” web applications are becoming more popular too with WebSockets and an event-based back-end. Messaging architectures, CQRS, rich domains, services over REST or SOAP – the list goes on. Should it be possible to code apps like these with Grails? What added value would Grails provide for each of these approaches?

So many questions. I don’t profess to have the answers but I do want to offer some suggestions in the hope that it kicks off a discussion about what we want from the framework.

To begin with, developers should have the flexibility to choose different types of architecture while still getting some benefit from Grails. I was struck recently by a talk on using command objects as the primary artifact of the application architecture. Command objects are an underrated part of Grails that can be used in all sorts of patterns and architectures. I have also dabbled with the Platform Core Plugin’s event bus as an alternative, message-oriented backend that can also push messages to Javascript in the browser. Both of these examples demonstrate that Grails already provides features for non-conventional approaches to application architecture.

Building direct support for lots of different architectures into Grails would be a bad idea. You would end up with a monolithic monster that no one would use. It would also likely be too much for the core team to manage alone. A better approach is to decouple the different parts of Grails into plugins and allow for alternative architectures through community-contributed plugins. This has already happened to some degree, but there is still some more decoupling to do. In other words, the plugin system should become the central part of Grails. Applications then become a collection of easily composed pieces.

Composable systems like this provide flexibility at the cost of a steeper learning curve and an extra barrier to getting started. One of the great things about Grails is how quickly you can get going with the standard application architecture. It would be a shame to lose that in the process of decoupling the different parts of the framework. The answer is probably a combination of project templates and strong conventions and defaults, where project templates are simply default project configurations with different sets of plugins depending on what application architecture you want. I think it’s possible to make most architectures accessible using this approach.

I have to admit that this idea of focusing on the plugin system and project templates (or archetypes) is not new, but it does make sense to me. It also raises challenges: who will maintain project templates and the associated plugins? There is no way that the core Grails team can do it. That means the community needs to step in and that’s why there has to be work to make it as easy as possible to contribute, from guides on how to do things to consistent and well documented APIs or other integration points. Fortunately, the act of decoupling the internals of Grails and putting in place APIs will make contributions easier. And the existing GORM decoupling work has shown the way here.

A vision is only one part of the story. It gives a destination, but what guides the day-to-day decision making? That’s where values come in. Things like “every feature will be documented” or “upgrading should be as automatic as possible” or “the user should never be left waiting.” These can then be codified, for example by having commit guidelines that state new or changed features should never be committed without documentation and integration or functional tests. Such rules are particularly important when you’ve spent lots of time developing and debugging a feature and just want to be done with it. It’s very easy to skimp on the testing and documentation at that point.

Conclusion (and “too long, didn’t read” version)

To sum up, I think any project needs to have a clear vision and a core set of values that all committers and contributors must cleave to. I would like to see Grails focusing on being a rapid web application development framework that is accessible but allows for flexibility. Things like performance are important, but secondary. In other words, Grails should aim to reach a certain number of requests per second for a given setup, but it shouldn’t aim to be the smallest and fastest framework in the field (for example).

The idea of rapid development should come from several features:

  • Near immediate feedback on changes
  • Very few (preferably zero) long wait times (say more than 10s sitting on your hands)
  • Error reporting that makes it easy to identify the source of a problem
  • Conventions, defaults, little to no boilerplate
  • Easy integration with common external systems

Grails already ticks these boxes to a greater or lesser extent, and to be honest I don’t have major feature requests. Instead, I would like to see a focus on the reloading side of things, even better error reporting for all parts of Grails (the simple stuff is reported nicely, but things like Spring initialisation errors are hard to interpret) and finally a well-documented, straightforward plugin API that plugin developers can have confidence in. Because ultimately I think Grails’ success depends heavily on the quality of the plugin ecosystem.

61 thoughts on “Where next for Grails?

  1. Roberto Guerra

    @Peter, yes ActiveRecord is persistence. And I’m fine with GORM being a persistence API, but my problem is that it bleeds right through the domain model. I would have preferred an adoption of the Datamapper pattern over the Active Record Pattern for GORM.

  2. ken

    Wow..the comments are pretty depressing 😛

    I’d say priorities would be:
    – stability
    – small core and plugins for everything (20MB for a minimal size is wayyy too big)-:
    – better unit tests
    – gradle for builds
    – async for scaleability (I’d have to disagree that this isn’t important…most people are going to look at stuff like the techempoower benchmarks as a checkbox item and grails is currently slow w/o async…so even if Node.js is single-process, it still goes fast w/ async and is near the top for performance on modern/RESTful/JSON web sites)

  3. Eric

    I’m late to this discussion, but it’s still interesting. I find myself looking for alternatives to Grails even though it has been my go-to framework for the past few years. I want something lighter weight since most everything I do now has a Javascript-heavy UI that uses a REST backend.

    For Grails to stay relevant it needs:
    – To be more modular: I don’t need templating, GORM, etc. I don’t really need the resources feature since I’m using something like RequireJS. All those extra unused pieces bloat the WAR. I’m lucky to keep my WAR under 30MB.
    – Easily distributable: Frameworks like vert.x with its event bus and actor-like model provide very flexible and powerful ways to distribute your app.
    – Better stability and performance. I still get memory leaks with Tomcat. Tickets like http://jira.grails.org/browse/GRAILS-9525 were opened last year haven’t been touched.
    – Better support for async request handling: Something better than the Servlet async API, please.
    – Polyglot support: Groovy is great, but we have developers who work better in other langauges (e.g. Scala, Javascript, etc).

  4. Sérgio Michels

    I really liked David’s point of view in his post reply.

    To complement his idea, last week there was a conf about JavaScript (BrazilJS) and I saw a presentation about Yeoman – http://yeoman.io/

    I just looked about some concepts of this in the site, but it seems really interesting.

    It have tools for scaffold, build and control dependencies. Nothing much new here, but if you look at the scaffold part it becomes interesting.

    The concept of generating your application based on a model I think that totally applies to Grails and the plugin ecosystem.

    Instead of giving us a lot of binary plugins, let the community build the stack of an application.

    Maybe for fresh Grails dev teams, choosing witch plugins to attach for his application can be hard. Generators can guide them in this task.

    Perhaps we can have a better approach to notify plugins that aren’t up-to-date in the build too. In my list there are plugins that depends on the Grails version, for example:

    Plugins with available updates are listed below:

    tomcat 2.2.3 2.2.5.BUILD-SNAPSHOT
    rest-client-builder 1.0.3 1.0.2
    resources 1.2 1.2.RC2
    release 2.2.1 3.0.0
    hibernate 2.2.3 3.6.10.M6

  5. Grails Fan

    Where next for Grails? that implies that it has “arrived”. Well it really hasn’t arrived from a productivity and enterprise application development robustness perspective. See the posts above from Marc Palmer and Rodrigo Rosenfeld Rosas for some context. Before moving to the next level, the following issues need to be addressed:

    1. Grails platform stability. There still remain plenty of bugs. See the issues log.
    2. While there are plugins galore, several of them are stale, unsupported and of poor quality. So what’s a developer to do, evaluate each one and see what works? Majority of plugins are written by good intentioned individuals but with limited time at their disposal.

    Instead of adding new features, perhaps the next 6 months (and yes even a year) should be spent on:

    a. Bug fixes in grails
    b. Documentation
    c. Addressing real world use cases (e.g. being able to update Nested domain objects with complex object graphs)
    d. Writing more realistic sample applications
    e. Building a unit-test case battery. Just curious as to what’s the code coverage achieved by Grails current test suite?
    f. Adoption and support of key plugins directly by Pivotal (and not by individuals or by Pivotal employees acting in individual capacity)

    In my 1 year experience with Grails, I always keep wanting to ditch it for the stability of Java + Spring + Hibernate/JPA. However, there’s still something in the wonder and productivity promise of Grails and Groovy that keeps me suckered-in for my after-hours hobby work, despite the loooong hours tracking down plug-in bugs, grails platform issues, lack of documentation and examples. With all this headache, in all honesty I cannot recommend Grails usage in an enterprise setting – I would be fired once any of the many “problems that are obtuse and difficult” (quoting Marc Palmer) would come up. The productivity benefits very quickly dry up and developer angst sets in.

    Moving on to other feature sets, while building on a less than solid foundation, will only lead to bad press and even less adoption. Let’s the get fundamentals solid and then move to where Grails hasn’t gone before 🙂

    Hope this helps some.

    Grails Fan.

  6. Sergey Orshanskiy

    Well, let me give some feedback on Grails’ adoption. I have 15 years of programming experience, but only 1 year of web programming, so I am somewhat of a beginner in this space. So far, as far as web frameworks are concerned, I only really have experience with Django and now Groovy/Grails.

    Here are some things that make one want to adopt Grails:
    1. The website grails.org does a good job “selling” the framework 🙂
    2. The Groovy language is fun to learn and to use.
    3. The scaffolding feature does a good job of “selling” the framework, even if it is not particularly useful. However, it gets very confusing once you are trying to create anything slightly nontrivial, and then you realize that you still have to do the work 🙂 Although it did find the auto-generated GSP’s helpful as examples to be studied.
    4. The documentation is decent, though not even close to Django’s.

    Here are the things that make me want to not use Grails:
    1. Both Groovy and especially Grails are terribly slow. That makes learning very slow. Usually I can learn very quickly by making little changes in some sample code and seeing the result. Here the feedback loop is so slow, probably 10 times slower than with python/django, so the learning curve is slowed down considerably.
    2. Lots of mysterious bugs with application reloading. “grails clean”, “grails compile –refresh-dependencies”… At one point I had to manually delete *.class files for DataSource.groovy . I’ve literally spent hours looking for bugs when all I had to do was to rebuild the whole thing.
    3. The need to understand Hibernate and its issues. In general, completely incomprehensible exception messages. Object’s id is null? How is it my problem? I didn’t even specify that the object has to have an id, it’s an implementation detail. If I wanted to deal with this kind of thing, I would be writing in C++.Similarly, I have recently spent several hours trying to connect Grails to a MySQL database, and I ended up writing a Groovy script to test this connection first. It produces those exceptions with literally dozens of pages of stack traces. And yes, the key information is there, if you know where to look.
    4. Lack of decoupling. I want to be able to just write a groovy script accessing GORM. I know, I can kind of do it with “grails run-script”, but it is so terribly slow. I tried to change java path, etc., to at least debug certain things in a separate groovy shell, but every once in a while I still get some weird exceptions — which I don’t get when using a grails console.

    Overall, I think, I’d look into Ruby on Rails 😉

  7. Owen Rubel

    For me Grails has been that breath of fresh air and the Java community has seemed to be the blocker; I have seen nothing but a lack of vision and clinging to old thinking from old-school Java devs and that Grails/groovy appeals to a new gen of developers looking for a Java for the next wave.

    I actually was a LAMP deve longing for that new wave of Java for the longest time and Groovy / Grails fit it so well.

    My needs really are:

    1. Unit/integration test documentation. This has been a pain in the butt.

    2. Gradle integration documentation for Grails. Want to start using this more and everywhere. I mostly do alot of copying of code and figuring things out as I go but I would love to have a deeper understanding.

    3. It is LONG overdue for a ‘ADVANCED GRAILS’ book. Everyone has the first books and they were awesome but alot of people stopped there because going beyond that took too much. There really needs to be a LARGE advanced volume or couple of volumes to get people beyond the basics. I spent the last few years pouring over code and I still feel like there is so much more to learn. I’m actually happy for that but there are people who aren’t as thrilled and give up alot easier.

    For those PO’d about the API’s, I’ve actually been spending the last 3-4 mopnths doing a massive rewrite of a group of API plugins I have released into one cohesive api toolkit that will be being released in February (https://github.com/orubel/grails-api-toolkit)

    This simplifies API development to the extreme so ANYONE can get started with API development.

    I will also be extending it eventually for API developer interface(hopefully) and integrating authentication methods so you can authenticate however you want.

  8. Peter Post author

    @Owen Burt Beckwith’s book Programming Grails should be advanced enough as I understand it.

    The API toolkit sounds interesting. Is that orthogonal to the Grails Plugin Platform or does it overlap somewhat?

  9. Owen Rubel

    @Peter No overlap except if the fact it uses grails same functionality for request/response handling.

    There are some things that I improved on from last few API plugins I hav built like

    – adding roles to apis thus requiring them to be secure by default
    – allowing multiple roles to support HEAD, OPTION and one of GET/PUT/POST/DELETE for example
    – roles for hooks on apis
    – roles on apidocs so one dataset can be defined even though the method returns for two different roles; this allows the apidoc to show only the appropriate data for the role.
    – conversion of domain objects on the fly; though this is still bad practice for an api to use domain objects.
    – improved filter
    – improved encoding
    – passing of content type through headers via HATEOAS spec

    … and more. Its a full API toolkit per the name and not just a one-to-one mapping of api calls to domains as per in Grails. This is meant as a full API support toolkit.

  10. Owen Rubel

    @Peter Oh actually I think I misunderstood your question. Yes overlap in the fact that it EXTENDS the functionality. It uses Grails existing functionality and extends.

    Grails new API functionality is good but it is still mapping domains with CRUD and has direct one-to-one relationship with domains.

    In building api’s, you will often have to do joins in your methods (sometimes across databases) and serve that as your api so you will build a commndobject to validate. What I attempt to do is work not just with one domain but multiple and allow apidoc mapping as well as realtime notifications (webhooks) and integrate all of this with Spring security so your api’s are secured, your hook/callbacks are secured, your apidocs show only to who they are supposed to show, etc.

    And they work with a one lined annotation through your method rather than through the domain so you are mapping the output which is effectively which is what we need to do to take into account all possible solution since a by-the-book RESTful solution doesn’t take into account all possible solutions and is CRUD based; we need to also take into account ‘naming conventions’ that dont comply with the REST naming style but allow them to use REST calls as well.

    Stuff like that. Am really trying to work with all three approaches of RPC, REST and HATEOAS to hit a common api approach which will hit a 90% mark and be accomodating for the rest.

  11. Owen Rubel

    The one thing I would promote (and promote hard) is an abstraction of the API from the model and moving it more towards the ‘front controller’ so you can apply rules definitions and roles at an atomic level to the incoming request and outgoing response separate from the processing/validation of the model data.

    This allows us to have an api object that can be reused for rules and definitions but also to reject at the front controller where rules are not met (separate from validation of the model) and even process separate data for roles at the response given the same model output.

    This also enables api chaining, the api objects to reference each other at the front controller, obfuscation of keys and several other functionality.

    We need to stop think of the URI being associated with the model and the URI being associated with an api object which can associate with a variety of different types of models (ORM, Command Object, etc) thus privilege checking, definitions, rules can be placed prior to continuing on toward processing in the model.

Leave a Reply

Your email address will not be published. Required fields are marked *