Once more ... feedback please!

You've probably heard about "Not Invented Here" syndrome: the drive among developers to create something of their own, rather than just use an off-the-shelf library or component. It's almost universally painted as a bad thing, a sign of immaturity, or even arrogance.

But there's a flip side to this: every bit of code ever written contains within it tradeoffs: speed versus maintainability is a common tradeoff that everyone has seen. Perhaps the code is insufficiently flexible in the face of real-world requirements, but is really well tested for what it does cover. These choices reflect the developer's principles applied to the code. In fact, it is rare for it to be an easy give-and-take between two simple goals; more likely, there's lots of conflicting goals in the code, in the requirements, and in the developer's head. "Not Invented Here" can also mean "Not Reflecting My Principles".

Tapestry has it own set of guiding principals: Simplicity, Consistency, Efficiency, and Feedback ... and as a reusable framework, Feedback is very important. Feedback may be the most important principle when things go wrong. A framework that obscures problems, through bad feedback, is a framework that shouldn't be used.

Which brings us back to "Not Invented Here". Only in an impossibly perfect world would there be some ideal blob of code out there, ready to be reused, with zero impedance mismatch issues. In fact, when you bring in other people's code, you are forced to mesh your goals and principals with theirs. In my case, I'm adding support to Tapestry for converting Less files to CSS, using WRO4J (Web Resource Optimizer for Java). They've been working on WRO4J for several years, it makes sense to reuse their code, and it would be arrogant to think I could whip something better together under any kind of time constraints.

In fact, it's actually been pretty smooth sailing ... until we tripped across a violation of Tapestry's Feedback principle. As soon as I tested an error case, where the Less source file was not valid I hit bad feedback. Can you spot what's wrong with this exception report?

Oops! Looks like someone didn't get the memo about the importance of toString(). See, that Feedback principal is important to me, but for the majority of developers, useful feedback is too often an afterthought. I don't want to single out WRO4J here ... I'm pretty disdainful about feedback in nearly all software: open source or proprietary.

So what are our options here?

• Write our own wrappers around the Less Processor, and throw out WRO4J
• Beg the WRO4J guys to implement a real toString(), and wait for the next release
• Fork WRO4J in the short term, and hope they'll take a patch in the long term
• Patch around this reporting problem

Obviously, we should find a way to patch the reporting problem; we don't want to throw out the baby with the bath water. Fortunately, Tapestry provides the necessary hooks to override how it presents objects inside the exception report; it's all about providing a mapping from a Java type to a matching implementation of ObjectRenderer. Because of Tapestry's IoC container, this is actually quite straight forward:

And with those changes, the exception is presented quite differently:

Well, those are actually the raw ANTLR parser errors, but at least that's enough to help you find location of the problem ... whereas, with the bad feedback, you would only know that there was an issue somewhere in your Less source file.

Crafting Code in Clojure

The other day, I was working on a little bit of code in Clojure, just touching up some exception reporting, when I was suddenly struck by one of the fundamental reasons that Clojure is so enjoyable to code in. Clojure is craftable: that is, in Clojure you have the option to craft at your code to make it more concise, easier to read, and easier to maintain. That is not the case for all, or perhaps even most, programming languages.

In my case, I was constructing an error message where I needed to convert the keys of two maps into a comma-seperated string (I don't like to say "you guessed wrong" without saying "here's what you could have said").

What I want my code to do is easily expressed as an informal recipe:

• Extract all the keys from both maps
• Remove any duplicates
• Convert the keys to strings
• Sort the strings into ascending order
• Build and return one big string, by concatinating all the key strings, using ", " as a seperator
• Return "<none>" if both maps are empty

If I was writing this in Java, it would look something like this:

There's enough looping and conditionals in this code (along with tip-toeing around Java Generics) that its easier to look at its test specifiction (written in Spock) to see what it is supposed to do:

The first pass at a Clojure version is already simpler than the Java version ...

I couldn't resist using the clojure.string/join function, rather than building the string directly (which would be slightly tedious in Clojure). In many ways, this is a lot like the Java version; we're using let to create local symbols for each step in the process in just the same way that the Java version defines local variables for each step.

However, there's room for improvement here. Let's start to craft.

For example, let's assume that both maps being empty is rare, or at least, that the cost of sorting an empty list is low (it is!). Our code becomes much more readable if we merge it into one big let:

Now we're getting somewhere. I think this version makes it much more clear what is going on that the prior Clojure version, or the Java version.

However, if you've written enough code, you know one of the basic rules of all programming: names are hard. Anything that frees you from having to come up with names is generally a Good Thing. In Java, we have endless names: not just for methods and variables, but for classes and interfaces ... even packages. Long years of coding Java has made me dread naming things, because names never quite encompass what a thing does, and often become outdated as code evolves.

So, what names can we get rid of, and how? Well, if we look at the structure of our code, we can see that each step creates a value that is passed to the next expression as the final parameter. So all-keys is passed as the last parameter of the (map) expression, resulting in key-names, and then key-names is passed as the last parameter of the (sort) expression. In fact, ignoring the empty check for a moment, the sorted-names value is passed to the (s/join) expression as the last parameter as well.

This is a very important concept in Clojure; you may have heard people trying to express that you code in Clojure in terms of a "flow" of data through a series of expressions. We'll, you've just seen a very small example of this.

In fact, it is no simple coincidence that the last parameter is so important; this represents a careful and reasoned alignment of the parameters of many different functions in clojure.core and elsewhere, to ensure that flow can be passed as that final parameter, because it becomes central to the ability to combine functions and expressions together with minimal fuss.

We can use the ->> macro (pronounced "thread last") to rebuild our flow without having to come up with names for each step:

The ->> macro juggles our expressions into an appropriate order; without it we'd have to deeply nest our expressions in an unreadable way: (sort (map str (set (concat (keys map1) (keys map2))))). Even with a short flow of expressions, that's hard to parse and interpret, so ->> is an invaluable and frequently used tool in the Clojure toolbox.

We can continue to craft; the first expression (that builds the set from the keys), can itself be broken apart into a few smaller steps. This is really to get us ready to do something a bit more dramatic:

This is getting ever closer to our original recipe; you can more clearly see the extraction of keys from the maps before building the set (which is only used to ensure key uniqueness), before continuing on to convert keys from objects to strings, sort them, and combine the final result.

In fact, we're going to go beyond our original brief, and support any number of input maps, not just two:

The mapcat function is like map, but expects that each invocation will create a collection; mapcat concatinates all those collections together ... just what we want to assemble a collection of all the keys of all the input maps.

At this point, we don't have much more to go ... but can we get rid of the sorted-names symbol? In fact, we can: what if part of our flow replaced the empty list with a list containing just the string "<none>"? It would look like this:

... and that's about as far as I care to take it; a clean flow starting with the maps, and going through a series of expressions to transform those input maps into a final result. But what's really important here is just how fast and easy it is to start with an idea in Clojure and refine it from something clumsy (such as the initial too-much-like-Java version) into something elegant and surgically precise, such as the final version.

That's simply not something you can do in less expressive languages such as Java. For example, Tapestry certainly does quite a number of wonderful things, and supports some very concise and elegant code (especially in green code) ... but that is the result of organizing large amounts of code in service of specific goals. We're talking tons of interfaces, a complete Inversion-Of-Control container, and runtime bytecode manipulation to support that level of conciseness. That's the hallmark of a quite consequential framework.

That isn't crafting code; that's a big engineering effort. It isn't local and invisible, it tends to be global and intrusive.

In Java, your only approach to simplifying code in one place is build up a lot of complexity somewhere else.

That is simply not the case in Clojure; by adopting, leveraging, and extending the wonderful patterns already present in the language and its carefully designed standard library, you can reach a high level of readability. You are no longer coding to make the compiler happy, you are in control, because the Clojure languge gives you the tools you need to be in control. And that can be intoxicating.

The source code for this blog post is available on GitHub.

Red Code, Green Code, My Code, Your Code

I had a very odd interchange with my friend Merlyn over lunch; he started talking about red code vs. green code (in the context of supporting both callbacks and promises inside NodeJS). At first I thought he was referring to code coverage of those lines of code ... the implication being that supporting multiple paths of execution may lead to laziness in testing every possible path (with some code going "red" in the code coverage report).

But that wasn't it at all: "red" code referred to framework code, "green" code referred to end-user code, leveraging the framework.

Odder still, Merlyn insisted that he first heard this term from ... me, a few years ago. That's what being in the baby-raising camp can do to you ... I know longer have any idea of what I've said or thought in the past. Actually, this isn't new for me ... I've always had a very vague memory for anything not code.

In any case, this red vs. green terminology is a useful concept ... certainly I move the earth in Tapestry's red code to make end-user's green code as simple as possible.

So, I want to do three things:

• It's definitely worth reflecting on the fact that all code is not created equal, and that long-lived, reusable code ("red") will inevitably grow in complexity to a level that would not be acceptable in client ("green") code.
• Let's promote this term, because it is so handy!
• If I didn't create the term myself, let's track down the originator and thank them!

Tapestry 5.4: jQuery Support now in place

I've spent the last several months significantly reworking Tapestry 5's client-side JavaScript support, in an effort to move away form the tight binding to Prototype. After all of that refactoring, recoding, repositioning, and just-plain-hacking, I was able to do most of the job of introducing jQuery support in just a few hours, yesterday and today.

I'll be producing another preview release pretty soon, or you can get the lastest from Tapestry's master branch.

Currently, the code to switch over from using Prototype to jQuery looks like this (it will get simpler soon):

The first part overrides the provider to be "jquery" (the default provider is "prototype"). In Tapestry terms, the "infastructure framework" provides the APIs for DOM queries, DOM manipulation, and Ajax request handling. If you don't like jQuery, you can easily create your own provider for your favorite framework.

Part of 5.4 is trying to manage all these different compatibility issues at a slightly higher level than configuration symbols; that's the Compatibility service with its Traits. Those two traits disable Scriptaculous support (which isn't needed by jQuery, and won't work without Prototype), and disables support for Tapestry 5.3-style initializers.

Underneath the covers, the way the switch between Prototype and jQuery works is quite simple:

The ModuleManager service is the code that handles requests for modules from the client. It has a configuration that is used to handle edge cases, such as treating traditional JavaScript libraries as if they were AMD modules. The code above sets up a server-side override for the module t5/core/dom. All the Tapestry client-side modules use dom; none of them uses Prototype or jQuery directly (except for a couple that access Twitter Bootstrap functionality).

These contributions ensure that when the client-side requests the t5/core/dom module, what will be sent back will be either the Prototype-specific implementation, or the jQuery-specific implementation. Without this contribution, we'd see a 404 error when the dom module was requested. Instead, the client-side doesn't have any idea that dom is special.

The primary job of the dom module is to wrap DOM elements inside a new object, thereby providing a new API that allows various kinds of manipulation, as well as listening to events, or triggering them. The API is a bit of a mashup between Prototype and jQuery, but leans pretty heavily towards jQuery-style operations and naming. dom's secondary job is to support Ajax requests, plus adding event handlers to the document object.

In practice, this can be very concise and readable (partially, thanks to CoffeeScript):

In this listing (which supports some of the behavior of the Tapestry Zone component), dom holds the export from module t5/core/dom; it waits for the events.zone.update event to be triggered; the callback is invoked with this set to the wrapper around the element where the event was triggered.

The event here is also a wrapper; its a minimal mix of Prototype and jQuery: I like the memo property from Prototype, so that's present. The event handler triggers a pair of before and after events, and updates the content of the zone. Why the before and after events? By default they do nothing, but it would be simple to add handlers to perform some kind of animation when content is added.

In any case, because all Tapestry client-side modules code against this API, they don't know or care whether the page has loaded Prototype, jQuery, or something else. If you are writing Tapestry components for reuse, coding against the dom API will help ensure that your component will work correctly in all kinds of Tapestry applications. However, when coding an application,. you reserve the right to choose what the infrastructure framework should be: you should feel free to use the infrastructure framework directly ... unless you find the dom API to be easier.

How we can be more secure with mail-in ballots than touch screens

Yes, it's election day! I didn't wait in line ... I live in Oregon, where balloting is done by mail. That's a much better system.

Of course, people are suspicious of voting electronically, or by mail. Somehow the mechanical interaction reassures people. However, security has been an issue for the entire history of voting. Ballots can be "misplaced", or electronically stored values from modern voting machines can be invisibly tampered with, after the fact. Certainly, as it is election day, we're hearing stories about devices that make it hard to select specific candidates ... and the conspiracy theorists are certain that the fix is in.

It may be, but the cases I've heard of are simply bad touch screens. I wonder what the conspiracy theorists believe is going on when they attempt to withdraw $25 from an ATM and get $10 instead?

Real attempts to fix the elections don't change what you enter; they change what is stored ... possibly long after you have left the voting booth.

I would say that having fewer centralized counting locations would be more secure. First of all, we could use (as we do in Oregon), optical scanners ... the same, basic, trusted technology that's used for the SATs.

Next, the process of opening the secure envelopes and feeding them to the scanners could be more easily monitored if it occurred in a few locations across each state, rather than in every polling location across the entire country. This process is subject to fraud, yes, but is probably more likely to be affected by simple incompetence.

With physical ballots, it is also possible to do a proper recount. Voting on a touch screen is too ephemeral for a recount to have any meaning.

Ultimately, there is no way to ensure that your vote is actually counted. However, I think we can come up with a way in which we can at least ensure that your ballot made it to the processing center.

Imagine if the ballot mailed to your home had a unique bar code on it, as well as a "tear off" tab with the corresponding unique number. In addition, there is a section on the ballot where you could enter a four or six digit PIN code of your own choice.

Later, at home, you visit a web site: you provide your unique, randomly assigned ballot id as well as your own secret, personally-selected, PIN code. The government web site (built using open-source software, of course) could then identify the status of your vote, just like UPS can tell you where your package is.

Since PIN codes are ultimately guessable, I don't think I'd want the system to do more than confirm the processing of the ballot. However, to someone like me, it would go a lot further towards assuring me that, yes, my vote was counted than anything I've used in the past ... and certainly it would be far more reassuring than the buggy, mis-calibrated, poorly designed touch screen devices in use today.

Further, I like the idea of the vote taking shape over the course of a few days, or even a week. Vote by mail from a week before "election day". Maybe the ballots accumulate but are not counted until the end of the election itself ... again, something that the ballot status check outlined above would help to ensure.

The big weakness of this is privacy concern over linking your identity to a specific ballot. Who would be able to view your vote at a later date? The answer to that should be: nobody; that is not only a sacred tradition, but the fact that your private vote can never be retrieved and used against you in any way at a later date is important to keep voting free and uncoerced.

So imagine if your ballot arrived with four or five unique id stickers, each five or ten digits long: you chose, say, three of them, in whatever order you like and place them on the ballot. Nothing else on the ballot could link that specific ballot to you ... but that now partially-random ballot id can still be tracked to the central counting location. Only you would have a link between your identity and the ballot. I leave it to someone smarter than me to determine just how many "stickers" are needed to ensure reasonably anonymity (and prevent any potential ballot id conflicts) but I think the basic concept is valid.

Zeroing in on Tapestry 5.4

I've had just a bit of time this week to devote to furthering the work on Tapestry 5.4, and it feels like I'm near the turning point. You can follow the progress in the 5.4-js-rewrite branch

I've discussed my plans before; this is a bit of a progress update.

Basically, this release will be all about JavaScript:

• Removing the hard-dependence on Prototype
• Creating an abstraction layer, allowing jQuery to be used instead
• Introducing AMD (asynchronous module definition) modules (using RequireJS)
• Switching to a more declarative style of JavaScript; leveraging HTML5 data- attributes

Prototype vs. jQuery

Tapestry has historically used Prototype as what I call its foundation framework. The foundation is responsible for element selection and creation, attaching event handlers, triggering events, and encapsulating Ajax requests.

It is already possible to have both Prototype and jQuery on the same page; I do it regularly. There's a project for this as well, tapestry-jquery. That being said, there's some costs in having multiple foundation frameworks in place:

First of all is size; there's a lot of code in each framework (160Kb for Prototype, 260Kb for jQuery -- much less after minification and compression), and tremendous overlap. This includes the fact that both frameworks have a version of the Sizzle CSS selector engine.

There's also some occasional clashes; I had a nasty problem where some Bootstrap JavaScript was firing a "hide" event when a modal dialog was dismissed. After much tracing and hair pulling, I discovered that jQuery will treat a function attached to an element as an event handler for the event with the matching name. The "hide" event triggered by jQuery found the hide() method added by Prototype, and my modal dialog just winked out of existence, rather than animation the way I wanted.

Finally, its not just an either-or world; there's also YUI, ExtJS, MooTools ... over the last few years, every single time I mentioned my desire to introduce an abstraction layer, I've received the question "will it support X?" and "X" was some new framework, that person's favorite, that I had not previously heard of.

So the abstraction layer will be important for helping ensure a fast download of JavaScript, and to minimize the amount of network and JavaScript execution overhead.

If you are an application developer, there will be nothing to prevent you from using the native framework of your choice, directly. In many cases, this will be the best approach, and it will enable greater efficiency, access to a more complete API, and access (in jQuery's case) to a huge community of plugins.

If you are creating reusable components, it is best to try to use the SPI (the service provider interface; the abstraction layer). This allows your library to be widely used without locking your users into any particular foundation framework.

Modules vs. Libraries

Tapestry 5.3 and earlier have provided some great support for JavaScript in library form. The concept of a JavaScriptStack makes it possible to bundle any number of individual JavaScript libraries together, along with supporting CSS style sheet files. Even better, Tapestry can create a single virtual JavaScript library by concatenating all the stack libraries together. Tapestry does this at runtime, and conditionally ... so during development, you can work with many small JavaScript files and in production, they become one combined, minimized JavaScript file.

However, this JavaScript approach has its own problems. Page initialization logic is very dependent on the stacks (including the foundation frameworks) being present and loaded in a specific order. No page initialization logic can execute until all libraries have been loaded. If you have multiple stacks in production, then all the stacks must load before any logic executes.

All of this affects page load time, especially the time perceived by the end user. Too much is happening sequentially, and too much is happening over-all.

In Tapestry 5.4, the focus is shifting from libraries to modules. Under AMD, a module is a JavaScript function that expresses a dependency on any number of other modules, and exports a value to other modules. The exported value may be a single function, or an object containing multiple functions (or other values).

Here's a snapshot of one of modules, which provides support for the Zone component:

The define() function is used to define a module; the first parameter is an array of module names. The second parameter is a function invoked once all the dependencies have themselves been loaded; the parameters to that function are the exports of the named modules.

This function performs some initialization, attaching event handlers to the document object; it also defines and exports a single named function ("deferredZoneUpdate") that may be used by other modules.

Yes, I have a module named "_" which is Underscore.js. I'll probably add "$" for jQuery. Why not have short names?

The RequireJS library knows how to load individual modules and handle dependencies of those modules on others. Better yet, it knows how to do this in parallel. In fact, in 5.4, the only JavaScript loaded using a traditional <script> tag is RequireJS; all other libraries and modules are loaded through it.

At the end of the day, there will be less JavaScript loaded, and the JavaScript that is loaded will be loaded in parallel. I'm still working on some of the details about how module libraries may be aggregated into stacks.

Declarative JavaScript

Even in earlier forms of Tapestry, JavaScript support has been powerful ... but clumsy. To do anything custom and significant on the client-side you had to:

• Write a JavaScript library for your functionality
• Monkey-patch a function onto the T5.initializers namespace
• Package up your initialization data as a JSONObject ... typically, client element ids, URLs, etc.
• Have you Java component code invoke JavaScriptSupport.addInitializerCall(String, JSONObject) to link it all together.

Tapestry packages up all those addInitializerCall() initializations into one big block that executes at the bottom of the page (and does something similar for Ajax-oriented partial page updates).

Whew! To make this works requires that elements have unique ids (which can be a challenge when there are Ajax updates to the page). On top of that, the typical behavior is to create controller objects and attach event handlers directly to the elements; that works fine for small pages, but if there are hundreds (or thousands) of elements on the page, it turns into quite a burden on the JavaScript environment: lots of objects.

There isn't a specific name for this approach, beyond perhaps "crufty". Let's call it active initialization.

A more modern approach is more passive. In this style, the extra behavior is defined primarily in terms of events an element may trigger, and a top-level handler for that event. The events may be DOM related such as "click", "change", or "focus", or application-specific one triggered on the element directly. The top-level handler, often attached to the document, handles the event when it bubbles up from the element to the top level; it often distinguishes which elements it is interested using a CSS selector that includes references to a data- attribute.

For example, in the core/forms module, we need to track clicks on submit and image buttons as part of the form (this is related to supporting Ajax form submissions).

That little bit of code attaches a "click" event handler to the document, for any submit or image element anywhere on the page ... or ever added to the page later via Ajax. Older versions of Tapestry might have put an individual event handler on each such element, but in 5.4, the single event handler is sufficient.

Inside the event handler, we have access to the element itself, including data- attributes. That means that what may have been done using page initialization in Tapestry 5.3 may, in Tapestry 5.4, be a document-level event handler and data- attributes on the specific element; no JSONObject, no addInitializerCall(). Less page initialization work, smaller pages, fewer objects at runtime.

The flip side, however, is that the cost of triggering the extra events, and the cost of all the extra event bubbling, is hard to quantity. Still, the 5.3 approach introduces a lot of memory overhead at all times, whereas the 5.4 approach should at worst, introduce some marginal overhead when the user is actively interacting.

There's another advantage; by attaching your own event handlers to specific elements, you have a way to augment or replace behavior for specific cases. It's kind of a topsy-turvy version of inheritance, where the behavior of an element is, effectively, partially determined by the elements it is contained within. Kind of crazy ... and kind of just like CSS.

Compatibility

So you've written a fancy application in Tapestry 5.3 and you are thinking about upgrading to 5.4 when it is ready ... what should you be ready for?

On the server side, Tapestry 5.4 introduces a number of new services and APIs, but does not change very much that was already present.

A number of interfaces and services that were deprecated in the past have been removed entirely; even some dependencies, such as Javassist.

All built-in Tapestry components will operate through the SPI; they will work essentially the same regardless of whether you choose to operate using Prototype or jQuery or both.

The look-and-feel is changing from Tapestry's built-in CSS to Twitter Bootstrap. All of the old "t-" prefixed CSS classes are now meaningless.

On the other hand, what if you've built some complex client-side code? Well, if it is based on Prototype directly, that will be fine as well; just keep Prototype in the mix and start migrating your components to use the SPI.

The devil in the details is all about the Tapestry and T5 namespaces. The Tapestry object is the very oldest code; the T5 namespace was introduced in 5.2 in an attempt to organize things on the client-side a bit better.

It is not clear, at this time, just how much of those libraries will be left. Ideally, they will not be present at all unless an explicit compatibility mode is enabled (possibly, enabled by default). That being said, the conflict between the old active/crufty approach, and the new modular and passive/declarative approach is a difficult one to bridge.

In fact, the point of this post is partly to spur a discussion on what level of compatibility is required and realistic.

Conclusion

I've completely pumped about where Tapestry 5.4 already is, and where it is headed. Although the future of web applications is on the client, there is still significant play for hybrid applications: partly page oriented, partly Ajax oriented – and it's nice to have a single tool that integrates both approaches.

Beyond that, even for a Tapestry application that is a single "page", what Tapestry brings to the table is still very useful:

• live class reloading
• JavaScript stacks (with minification, Gzip compression, and client-side caching)
• the coming runtime CoffeeScript-to-JavaScript support
• best-of-breed exception reporting
• staggeringly high performance

These benefits are all as compelling in a single page application as they are in a traditional application, if not more so. I'm looking forward to building even more impressive apps in 5.4 than I could accomplish in 5.3, and I hope a lot of you will join me.

Not going to Clojure/Conj ... need a ticket?

Unfortunately, I will not be going to Clojure/Conj this year. And it is too late for a refund ... so I'm looking for someone to purchase my ticket at cost. The good news is the reason I'm not going ... I have a new project coming that will use Tapestry, Clojure, and perhaps Datomic and I need to be on-site with them the week of the conference.

So if you are looking to attend the conj, you can take over my ticket, for $475. Please contact me via email. Thanks!

Update: Sorry, the ticket is gone.