I love JIRA, I really do, so long as I never have to do any administration to it.

I’ve got a JIRA instance set up as a household to-do system.  Today, I tried installing JIRA 4.1 as a test instance, upgrading from version 4.0.  It turns out I won’t be making that upgrade for real any time soon, as 4.1 breaks many of the plugins I need for my JIRA workflow.  If I upgraded, I’d cripple my setup beyond usability.

Unfortunately, Atlassian has chosen to leave many obvious and necessary workflow actions out of JIRA, instead relying upon third-party plugins to provide them.  This wouldn’t be a problem if Atlassian didn’t change their plugin API more often than Tiger Woods picks up new sexual partners.

Okay, so I figured I’d at least upgrade to version 4.0.2; I’ve been putting that off for too long.  It’s anything but an easy process, especially when you’re running the EAR/WAR version instead of their “standalone” that comes with its own Tomcat instance.  They don’t have a standalone build for FreeBSD, so I don’t really have a choice.

Although the process is really convoluted and something only a Java wonk could love, it wouldn’t be so bad except that JIRA’s plugin system is so horrible.  JIRA plugins are generally Java .jar files.  There are two kinds of JIRA plugin now, Version 1 and Version 2.  It’s not obvious from looking at a plugin what kind it is.  You have to either go to Atlassian’s plugin repository and find it there, and the page will tell you what version the plugin is… or you have to extract the plugin and go digging in XML files trying to find a setting that will exist only if it’s version 2.

Depending on whether it’s version 1 or version 2, the plugin has to go in one of two widely different directories.  If it’s version 1, it goes in a directory that will need to be wiped out when you upgrade JIRA.

What’s so bad about that?  It’s also the directory that houses dozens of JIRA’s own .jar files, and there’s no way to tell which ones are part of JIRA and which ones are plugins.  In fact, there’s no way to tell what plugins you’ve installed short of doing a file-by-file comparison with a clean JIRA installation of the same version.

Better document that JIRA installation really well as you do it…!

It gets better! Some Version 2 plugins won’t work from their new, plugins-only directory anyway! You won’t find this out until they fail to work, and then you dig through the log file—which Atlassian so thoughtfully creates in whatever working directory you happened to be when you started Tomcat—and see a message that you have to copy the file to the old directory!

Of course, if you merely copy the file, as directed by the error message, you’ll get another warning that there are two copies installed so you can’t use either…

Did I mention that every time you add or remove a plugin, you have to restart Tomcat?  If you don’t have some really rockin’ hardware running your JIRA instance, that’ll bring you a minute or two of downtime.  It’s annoying with three users in one household; it’d be a bloody outrage if I were supporting a large company with this.

What’s particularly galling is that Atlassian’s other major product, Confluence, does plugins so well.  You can search the Confluence plugin repository from within your Confluence instance’s administration screens.  You can see which plugins you’ve installed, and if they’re out of date Confluence will tell you so and offer you a link to download and install the latest version.  There’s even a web-upload link to install .jar files from outside of the Atlassian repository.  Atlassian knows how to do this right.  They just haven’t bothered with JIRA.

So, I do my upgrade.  This entails backing up the current JIRA instance to XML, creating a new database for the new JIRA, installing the new JIRA instance, copying over a number of settings files, tweaking my Tomcat settings, copying over plugins I’d backed up previously, restarting a few times to get the database loaded and the plugins in place, etc. etc. …

…and then the bloody thing doesn’t show any of my workflow actions, because I’m one plugin short.  It doesn’t even clearly tell me “hey, you have workflow actions that don’t seem to be defined” anyplace useful.  It just doesn’t show me any workflow actions. When I go into my workflow editor, the only clue is that some workflow actions look like

Type: class
Class: com.googlecode.jsu.workflow.function.UpdateIssueCustomFieldPostFunction
Arguments:
field.value = Yes
field.name = customfield_10040

instead of

The Needs Parts of the issue will be set to Yes.

Notice how there’s also no hint whatsoever as to which plugin you previously had that’s now AWOL; you’re left to figure that out on your own.

After finally figuring out which plugin was missing—the one that implements the ability to change the value of a field other than the default ones provided with a generic JIRA install—installing it, starting, finding no improvement, finding the log error saying it’s in the wrong place, moving it, restarting… I finally have my workflow actions back, and I’m an hour behind on my other tasks for the day.

Then there’s the security patch. The boys at Atlassian had a rather embarrassing security breach recently. It highlighted a number of XSS attacks against JIRA.  None of the code currently available for download includes the fixes, so you have to apply the patch.  For EAR/WAR installs like mine, this involves:

1. 1. Extracting the patch files on top of the source for JIRA
   2. Rebuilding the WAR file
   3. Stopping Tomcat
   4. Copying the WAR file into place
   5. Backing up your existing Version 1 plugins, which Atlassian doesn’t bother to mention in the instructions, so hopefully you understand this consequence of what they do tell you
   6. Deleting the Tomcat work directory, because otherwise Tomcat may cache the old .jar files
   7. Creating a new Tomcat work directory by hand, which Atlassian doesn’t mention; if you don’t, Tomcat will appear to start but won’t actually run anything…
   8. Deleting the old jira directory, because otherwise Tomcat may not unpack the new .jar files, which isn’t mentioned in the patch directions but is mentioned in a troubleshooting section under new installs for when you upgrade and find upon restart that you haven’t, in fact, successfully upgraded
   9. Starting Tomcat
   10. Waiting for Tomcat to start JIRA
   11. Stopping Tomcat
   12. Reinstalling the Version 1 plugins
   13. Starting Tomcat

That’s one of the more convoluted “security patch” procedures I’ve ever had to deal with in over 15 years as a professional UNIX systems administrator.

If you’re thinking of buying JIRA for your business: If you can’t afford to hire a Java guru to be your full-time JIRA guy, don’t bother.  You will need someone who understands Java and databases and Apache Tomcat and your choice of operating system, and they’ll have their hands full.

[Edit: added one of the biggest examples i forgot while drafting this: the JIRA dashboard.]

I’m not particularly fond of Linux. I’ve used it, and it’s good for many things, but as a professional system administrator I prefer Solaris, or FreeBSD, or even Mac OS X (as a UNIX). Why? A great many of the Linux enthusiasts I’ve had to deal with have suffered from what I call Shiny Object Development Syndrome (SODS).

SODS is characterized by a tendency to concentrate on developing new features that are pleasing and attractive to the developer, with a complete disregard to the usefulness, usability, or basic function of the product itself. It is particularly prevalent amongst open-source developers, particularly those who work on obscure Linux distributions.

Developers who suffer from SODS are often heard replying to legitimate user complaints with some variation on “you can fix it yourself if it’s important to you, the source code is available.”

Sometimes SODS infects an entire organization. When this happens to a company with paying clients, symptoms include: refusal to fix longstanding bugs; failure to supply updates to widely-deployed, stable versions when longstanding bugs are finally addressed; issuing updates that break existing functionality such as extensions, plug-ins, or settings.

Terminal company-wide SODS often reveals itself as intractable bugs pile up because the new shiny features beloved of the developers and the marketing department depend upon so much spaghetti code that it becomes impossible to fix all the problems without starting over from scratch. By this stage, the company usually begins to hemorrhage users as the cost of migrating to another platform becomes cheaper than dealing with the SODS-ridden status quo.

A good example of a company full of SODS is Atlassian.

They make a number of interesting utilities, such as the Confluence wiki and the JIRA issue-tracking system. These programs have a lot of neat features. Because they are commercial programs, big companies feel comfortable buying them—they’re not some faceless open source developer. While Atlassian does offer some nice “starter licenses” at $10 for 10 users, a corporate-sized deployment involves a substantial chunk of money, especially for a small business.

Unfortunately, that doesn’t mean that you get better support than you would for an open-source product. In fact, your support experience will be full of SODS.

Confluence has been around a while. It’s a great place to put all your team’s technical documentation. But now comes the latest version of the software, and Atlassian has stopped supporting two of the most widely used “themes” for Confluence. (Themes are web-page layouts.) If you upgrade, your existing themes may become unusable. The replacement themes aren’t a drop-in replacement, so you’ll need to re-engineer your documents to get back to where you were. Apparently the old code was unsupportable.

That isn’t exactly customer focused.

At least Confluence is reasonably easy to upgrade. Shut down the existing instance, install the new one, alter one or two settings files to change where stuff is pointed, and you’re up and running. JIRA, on the other hand…

Upgrading JIRA is a system-administrator job preservation program. Even a minor upgrade involves dumping the entire JIRA database to an XML file, finding someplace to save it, and reinstalling JIRA from scratch. Atlassian recommends you attach it to a new database instance, while you’re at it. Once you reconfigure everything, from scratch, you can then reload all your data from XML. Obviously, this isn’t a quick process.

Hopefully, you don’t have a very large database of issues in JIRA. Otherwise, you’re going to have a very large XML file that will take a very long time to process, if your system doesn’t choke on it outright.

Don’t forget that you’ll need enough disk space for three copies of your data: two database instances, plus the XML. And that’s presuming you don’t want to back up your data regularly—to XML.

Did I mention that both programs are written in Java? Hopefully, your platform has a fast and stable Java Virtual Machine. (In my experience, such a thing doesn’t truly exist.)

Now that you’ve upgraded JIRA, you get to find out which of your vital JIRA plugins are broken. You see, JIRA doesn’t implement a lot of obvious and useful features. It leaves them for third-party developers to implement via plug-in Java code. Unfortunately, Atlassian hasn’t quite figured out what the plug-in API should look like, as of version 4.1 of JIRA, so there’s a good chance that any given JIRA upgrade will change the API and break your plugins. Broken plugins means broken workflow, which means JIRA is broken until you can find a fix.

Don’t contact Atlassian for that fix, though; they don’t support any plugin they didn’t write, and they don’t even support all of the ones they did write.

But what if you do find a legitimate bug in JIRA? Something that affects a lot of users, is highly visible, is an obvious bug in the program, and is clearly something that your expensive annual support contract should cover? For instance: in JIRA 4.0, if you use Mozilla Firefox, your JIRA Dashboard will fail after an hour of inactivity due to malformed security tokens. You’ll have to manually reload the Dashboard in your browser to continue.

Well, even with hundreds of users, many from Fortune 500 companies, clamoring for a fix on Atlassian’s forums, it took months for a fix to be identified. When it was, the fix was part of JIRA 4.1… which also includes sweeping changes to the user interface and, of course, API changes that break plug-ins. So, upgrading to JIRA 4.1 to get the Firefox fix is a non-starter for a lot of customers.

Atlassian released notes on how to patch JIRA 4.0… which involved downloading Java code and splicing it into the product. If you got adventurous and tried this, you wound up with a fully malfunctioning Dashboard. Atlassian’s response to date has been, to paraphrase, “Oops, our bad. Upgrade.”

Tone-deaf doesn’t begin to describe it.

A more classic example of SODS: JIRA 4.0 introduced an OpenSocial-based “dashboard” that uses JavaScript “gadgets” to display the user’s main interface with the software. Previous versions rendered the Dashboard in HTML created on the server, and it was reasonably speedy. Now, the OpenSocial dashboard looks pretty. It offers nerdnip like the ability to use widgets published by Google.

However, each gadget on your dashboard is a separate JavaScript that gets rendered by the client. The more gadgets on your dashboard, the slower it gets. If your browser has a poor JavaScript implementation (I’m looking at you, Internet Explorer), your dashboard will be dog-slow. And you’re really in trouble if you are on a slow computer, like a PowerPC G4, or an Atom-based netbook, or an iPad, or… Talk about a twofer; you get all the limitations of a web app together with all the slowness of your thin client. To Atlassian, it’s apparently reasonable to require a 3GHz dual Core2 processor to get reasonable performance out of a web-based issue tracker client.

It’s not just JIRA, though. Atlassian’s online documentation is stored in a Confluence wiki instance. I tried reading the JIRA upgrade notes on my iPad. I couldn’t get past the first page, because Confluence refused to let me scroll. It seems that the new themes in Confluence 3.2 don’t play nice with Mobile Safari. Having a paid support contract, I contacted Atlassian customer service. The response? Here’s a list of supported platforms, if the iPad gets popular we might consider adding it.

Well, Safari is listed as a supported browser. I’m using Safari, albeit the mobile version. And is there really any doubt that the iPhone OS is a popular operating system at this point? Is it really unlikely that iPad users might want to use it to read their internal documentation?

It seems to me that, if you’re going to write a Web-based application like a wiki, the very first thing you should do is make sure you’re following the HTML standards to a T. Safari is perhaps the most standards-compliant browser on the planet right now. If your application generates pages that Safari can’t render, you’re very likely doing something wrong.

Atlassian’s technical-support JIRA instance is full of enraged customers. Colleagues of mine that depend on Atlassian software are regretting their decision after seeing what Atlassian’s support is like.

In the meantime, Atlassian rolls out release after release with shiny new features, some of which seem to have little use other than to look cool. This is Shiny Object Development Syndrome, the Internet equivalent of fiddling while Rome burns.

I think Atlassian should call a moratorium on the shiny-object features, and spend a few development cycles working their way through the existing defect list. Start listening to customers and addressing their needs, even if they aren’t sexy… and even if they don’t line up with Atlassian’s vision of where the product is heading. Spend a few releases fixing things so that the APIs don’t have to change with every minor version. Identify the top 10 most popular plug-ins and make them unnecessary: integrate the functionality, buying out the plug-in developer if you have to.

(For programs like Confluence and JIRA, the plug-in libraries are almost like defect catalogues: Here is a list of all the things your customers find lacking in your software, to the extent that many of them have written their own code to overcome the problem. Why not see this for what it is and let it inform development, instead of apparently ignoring plug-ins and breaking them without remorse?)

I would love to be able to recommend Confluence and JIRA. At work and at home, they’ve generally made my life easier.

But then it’s time to upgrade, or something breaks, and I remember that Atlassian has an advanced case of SODS.

I hope that they get some help before it turns terminal.

For now, I’d have to recommend that readers consider alternatives to Atlassian products. The total cost of ownership is too dear.