I love JIRA, I really do, so long as I never have to do any administration to it.

I’ve got a JIRA instance set up as a household to-do system.  Today, I tried installing JIRA 4.1 as a test instance, upgrading from version 4.0.  It turns out I won’t be making that upgrade for real any time soon, as 4.1 breaks many of the plugins I need for my JIRA workflow.  If I upgraded, I’d cripple my setup beyond usability.

Unfortunately, Atlassian has chosen to leave many obvious and necessary workflow actions out of JIRA, instead relying upon third-party plugins to provide them.  This wouldn’t be a problem if Atlassian didn’t change their plugin API more often than Tiger Woods picks up new sexual partners.

Okay, so I figured I’d at least upgrade to version 4.0.2; I’ve been putting that off for too long.  It’s anything but an easy process, especially when you’re running the EAR/WAR version instead of their “standalone” that comes with its own Tomcat instance.  They don’t have a standalone build for FreeBSD, so I don’t really have a choice.

Although the process is really convoluted and something only a Java wonk could love, it wouldn’t be so bad except that JIRA’s plugin system is so horrible.  JIRA plugins are generally Java .jar files.  There are two kinds of JIRA plugin now, Version 1 and Version 2.  It’s not obvious from looking at a plugin what kind it is.  You have to either go to Atlassian’s plugin repository and find it there, and the page will tell you what version the plugin is… or you have to extract the plugin and go digging in XML files trying to find a setting that will exist only if it’s version 2.

Depending on whether it’s version 1 or version 2, the plugin has to go in one of two widely different directories.  If it’s version 1, it goes in a directory that will need to be wiped out when you upgrade JIRA.

What’s so bad about that?  It’s also the directory that houses dozens of JIRA’s own .jar files, and there’s no way to tell which ones are part of JIRA and which ones are plugins.  In fact, there’s no way to tell what plugins you’ve installed short of doing a file-by-file comparison with a clean JIRA installation of the same version.

Better document that JIRA installation really well as you do it…!

It gets better! Some Version 2 plugins won’t work from their new, plugins-only directory anyway! You won’t find this out until they fail to work, and then you dig through the log file—which Atlassian so thoughtfully creates in whatever working directory you happened to be when you started Tomcat—and see a message that you have to copy the file to the old directory!

Of course, if you merely copy the file, as directed by the error message, you’ll get another warning that there are two copies installed so you can’t use either…

Did I mention that every time you add or remove a plugin, you have to restart Tomcat?  If you don’t have some really rockin’ hardware running your JIRA instance, that’ll bring you a minute or two of downtime.  It’s annoying with three users in one household; it’d be a bloody outrage if I were supporting a large company with this.

What’s particularly galling is that Atlassian’s other major product, Confluence, does plugins so well.  You can search the Confluence plugin repository from within your Confluence instance’s administration screens.  You can see which plugins you’ve installed, and if they’re out of date Confluence will tell you so and offer you a link to download and install the latest version.  There’s even a web-upload link to install .jar files from outside of the Atlassian repository.  Atlassian knows how to do this right.  They just haven’t bothered with JIRA.

So, I do my upgrade.  This entails backing up the current JIRA instance to XML, creating a new database for the new JIRA, installing the new JIRA instance, copying over a number of settings files, tweaking my Tomcat settings, copying over plugins I’d backed up previously, restarting a few times to get the database loaded and the plugins in place, etc. etc. …

…and then the bloody thing doesn’t show any of my workflow actions, because I’m one plugin short.  It doesn’t even clearly tell me “hey, you have workflow actions that don’t seem to be defined” anyplace useful.  It just doesn’t show me any workflow actions. When I go into my workflow editor, the only clue is that some workflow actions look like

Type: class
Class: com.googlecode.jsu.workflow.function.UpdateIssueCustomFieldPostFunction
Arguments:
field.value = Yes
field.name = customfield_10040

instead of

The Needs Parts of the issue will be set to Yes.

Notice how there’s also no hint whatsoever as to which plugin you previously had that’s now AWOL; you’re left to figure that out on your own.

After finally figuring out which plugin was missing—the one that implements the ability to change the value of a field other than the default ones provided with a generic JIRA install—installing it, starting, finding no improvement, finding the log error saying it’s in the wrong place, moving it, restarting… I finally have my workflow actions back, and I’m an hour behind on my other tasks for the day.

Then there’s the security patch. The boys at Atlassian had a rather embarrassing security breach recently. It highlighted a number of XSS attacks against JIRA.  None of the code currently available for download includes the fixes, so you have to apply the patch.  For EAR/WAR installs like mine, this involves:

1. 1. Extracting the patch files on top of the source for JIRA
   2. Rebuilding the WAR file
   3. Stopping Tomcat
   4. Copying the WAR file into place
   5. Backing up your existing Version 1 plugins, which Atlassian doesn’t bother to mention in the instructions, so hopefully you understand this consequence of what they do tell you
   6. Deleting the Tomcat work directory, because otherwise Tomcat may cache the old .jar files
   7. Creating a new Tomcat work directory by hand, which Atlassian doesn’t mention; if you don’t, Tomcat will appear to start but won’t actually run anything…
   8. Deleting the old jira directory, because otherwise Tomcat may not unpack the new .jar files, which isn’t mentioned in the patch directions but is mentioned in a troubleshooting section under new installs for when you upgrade and find upon restart that you haven’t, in fact, successfully upgraded
   9. Starting Tomcat
   10. Waiting for Tomcat to start JIRA
   11. Stopping Tomcat
   12. Reinstalling the Version 1 plugins
   13. Starting Tomcat

That’s one of the more convoluted “security patch” procedures I’ve ever had to deal with in over 15 years as a professional UNIX systems administrator.

If you’re thinking of buying JIRA for your business: If you can’t afford to hire a Java guru to be your full-time JIRA guy, don’t bother.  You will need someone who understands Java and databases and Apache Tomcat and your choice of operating system, and they’ll have their hands full.

[Edit: added one of the biggest examples i forgot while drafting this: the JIRA dashboard.]

I’m not particularly fond of Linux. I’ve used it, and it’s good for many things, but as a professional system administrator I prefer Solaris, or FreeBSD, or even Mac OS X (as a UNIX). Why? A great many of the Linux enthusiasts I’ve had to deal with have suffered from what I call Shiny Object Development Syndrome (SODS).

SODS is characterized by a tendency to concentrate on developing new features that are pleasing and attractive to the developer, with a complete disregard to the usefulness, usability, or basic function of the product itself. It is particularly prevalent amongst open-source developers, particularly those who work on obscure Linux distributions.

Developers who suffer from SODS are often heard replying to legitimate user complaints with some variation on “you can fix it yourself if it’s important to you, the source code is available.”

Sometimes SODS infects an entire organization. When this happens to a company with paying clients, symptoms include: refusal to fix longstanding bugs; failure to supply updates to widely-deployed, stable versions when longstanding bugs are finally addressed; issuing updates that break existing functionality such as extensions, plug-ins, or settings.

Terminal company-wide SODS often reveals itself as intractable bugs pile up because the new shiny features beloved of the developers and the marketing department depend upon so much spaghetti code that it becomes impossible to fix all the problems without starting over from scratch. By this stage, the company usually begins to hemorrhage users as the cost of migrating to another platform becomes cheaper than dealing with the SODS-ridden status quo.

A good example of a company full of SODS is Atlassian.

They make a number of interesting utilities, such as the Confluence wiki and the JIRA issue-tracking system. These programs have a lot of neat features. Because they are commercial programs, big companies feel comfortable buying them—they’re not some faceless open source developer. While Atlassian does offer some nice “starter licenses” at $10 for 10 users, a corporate-sized deployment involves a substantial chunk of money, especially for a small business.

Unfortunately, that doesn’t mean that you get better support than you would for an open-source product. In fact, your support experience will be full of SODS.

Confluence has been around a while. It’s a great place to put all your team’s technical documentation. But now comes the latest version of the software, and Atlassian has stopped supporting two of the most widely used “themes” for Confluence. (Themes are web-page layouts.) If you upgrade, your existing themes may become unusable. The replacement themes aren’t a drop-in replacement, so you’ll need to re-engineer your documents to get back to where you were. Apparently the old code was unsupportable.

That isn’t exactly customer focused.

At least Confluence is reasonably easy to upgrade. Shut down the existing instance, install the new one, alter one or two settings files to change where stuff is pointed, and you’re up and running. JIRA, on the other hand…

Upgrading JIRA is a system-administrator job preservation program. Even a minor upgrade involves dumping the entire JIRA database to an XML file, finding someplace to save it, and reinstalling JIRA from scratch. Atlassian recommends you attach it to a new database instance, while you’re at it. Once you reconfigure everything, from scratch, you can then reload all your data from XML. Obviously, this isn’t a quick process.

Hopefully, you don’t have a very large database of issues in JIRA. Otherwise, you’re going to have a very large XML file that will take a very long time to process, if your system doesn’t choke on it outright.

Don’t forget that you’ll need enough disk space for three copies of your data: two database instances, plus the XML. And that’s presuming you don’t want to back up your data regularly—to XML.

Did I mention that both programs are written in Java? Hopefully, your platform has a fast and stable Java Virtual Machine. (In my experience, such a thing doesn’t truly exist.)

Now that you’ve upgraded JIRA, you get to find out which of your vital JIRA plugins are broken. You see, JIRA doesn’t implement a lot of obvious and useful features. It leaves them for third-party developers to implement via plug-in Java code. Unfortunately, Atlassian hasn’t quite figured out what the plug-in API should look like, as of version 4.1 of JIRA, so there’s a good chance that any given JIRA upgrade will change the API and break your plugins. Broken plugins means broken workflow, which means JIRA is broken until you can find a fix.

Don’t contact Atlassian for that fix, though; they don’t support any plugin they didn’t write, and they don’t even support all of the ones they did write.

But what if you do find a legitimate bug in JIRA? Something that affects a lot of users, is highly visible, is an obvious bug in the program, and is clearly something that your expensive annual support contract should cover? For instance: in JIRA 4.0, if you use Mozilla Firefox, your JIRA Dashboard will fail after an hour of inactivity due to malformed security tokens. You’ll have to manually reload the Dashboard in your browser to continue.

Well, even with hundreds of users, many from Fortune 500 companies, clamoring for a fix on Atlassian’s forums, it took months for a fix to be identified. When it was, the fix was part of JIRA 4.1… which also includes sweeping changes to the user interface and, of course, API changes that break plug-ins. So, upgrading to JIRA 4.1 to get the Firefox fix is a non-starter for a lot of customers.

Atlassian released notes on how to patch JIRA 4.0… which involved downloading Java code and splicing it into the product. If you got adventurous and tried this, you wound up with a fully malfunctioning Dashboard. Atlassian’s response to date has been, to paraphrase, “Oops, our bad. Upgrade.”

Tone-deaf doesn’t begin to describe it.

A more classic example of SODS: JIRA 4.0 introduced an OpenSocial-based “dashboard” that uses JavaScript “gadgets” to display the user’s main interface with the software. Previous versions rendered the Dashboard in HTML created on the server, and it was reasonably speedy. Now, the OpenSocial dashboard looks pretty. It offers nerdnip like the ability to use widgets published by Google.

However, each gadget on your dashboard is a separate JavaScript that gets rendered by the client. The more gadgets on your dashboard, the slower it gets. If your browser has a poor JavaScript implementation (I’m looking at you, Internet Explorer), your dashboard will be dog-slow. And you’re really in trouble if you are on a slow computer, like a PowerPC G4, or an Atom-based netbook, or an iPad, or… Talk about a twofer; you get all the limitations of a web app together with all the slowness of your thin client. To Atlassian, it’s apparently reasonable to require a 3GHz dual Core2 processor to get reasonable performance out of a web-based issue tracker client.

It’s not just JIRA, though. Atlassian’s online documentation is stored in a Confluence wiki instance. I tried reading the JIRA upgrade notes on my iPad. I couldn’t get past the first page, because Confluence refused to let me scroll. It seems that the new themes in Confluence 3.2 don’t play nice with Mobile Safari. Having a paid support contract, I contacted Atlassian customer service. The response? Here’s a list of supported platforms, if the iPad gets popular we might consider adding it.

Well, Safari is listed as a supported browser. I’m using Safari, albeit the mobile version. And is there really any doubt that the iPhone OS is a popular operating system at this point? Is it really unlikely that iPad users might want to use it to read their internal documentation?

It seems to me that, if you’re going to write a Web-based application like a wiki, the very first thing you should do is make sure you’re following the HTML standards to a T. Safari is perhaps the most standards-compliant browser on the planet right now. If your application generates pages that Safari can’t render, you’re very likely doing something wrong.

Atlassian’s technical-support JIRA instance is full of enraged customers. Colleagues of mine that depend on Atlassian software are regretting their decision after seeing what Atlassian’s support is like.

In the meantime, Atlassian rolls out release after release with shiny new features, some of which seem to have little use other than to look cool. This is Shiny Object Development Syndrome, the Internet equivalent of fiddling while Rome burns.

I think Atlassian should call a moratorium on the shiny-object features, and spend a few development cycles working their way through the existing defect list. Start listening to customers and addressing their needs, even if they aren’t sexy… and even if they don’t line up with Atlassian’s vision of where the product is heading. Spend a few releases fixing things so that the APIs don’t have to change with every minor version. Identify the top 10 most popular plug-ins and make them unnecessary: integrate the functionality, buying out the plug-in developer if you have to.

(For programs like Confluence and JIRA, the plug-in libraries are almost like defect catalogues: Here is a list of all the things your customers find lacking in your software, to the extent that many of them have written their own code to overcome the problem. Why not see this for what it is and let it inform development, instead of apparently ignoring plug-ins and breaking them without remorse?)

I would love to be able to recommend Confluence and JIRA. At work and at home, they’ve generally made my life easier.

But then it’s time to upgrade, or something breaks, and I remember that Atlassian has an advanced case of SODS.

I hope that they get some help before it turns terminal.

For now, I’d have to recommend that readers consider alternatives to Atlassian products. The total cost of ownership is too dear.

This blog entry was originally published on November 9, 2002.

While The Art of Turboing is all about how to complain with extreme prejudice, sometimes the same technique is useful when you have compliments or constructive criticism. Sometimes, it’s even profitable.

Read on to find out what happened when I shared some thoughts with the CEO of BJ’s Wholesale Club.

The background: I shop at BJ’s Wholesale a lot. They’re nearby, and their prices are quite good. Recently they installed self-checkout machines. After using them, I decided to share some feedback with BJ’s, not as a complaint per se, but as constructive criticism. I want them to succeed. Otherwise, I won’t be able to get so many cheap DVDs!

Here’s the letter I sent to Michael Wedge, the President and CEO of BJ’s:

Dear Mr. Wedge,

My family has had a charter membership to BJ’s Wholesale Club since it first came to Connecticut. As a long-time enthusiastic customer, I wanted to give you some feedback about the club. I’m writing directly to you because your web site doesn’t seem to have any way for customers to send feedback to your company.

Overall, I enjoy BJ’s and I think it’s a good value for the money. As with any warehouse club, the shopper needs to be aware of market prices and accept a relatively limited selection. That’s expected, and BJ’s meets or exceeds my expectations there.

The main reason I’m writing is to tell you about my recent experience with your new self-checkout system.

I’ve recently decided to stop shopping at KMart altogether, in large part due to their increasing reliance on self-checkout. While I like the concept of self-checkout, KMart’s implementation seems designed to enrage the customer. Not only have I never completed a “smooth” transaction at KMart’s self-checkout, but KMart seems to have cut back on its human checkout staff. This means that I either have to accept their poor self-checkout, or wait in very long lines for the few remaining cashiers.

On my trip to BJs today, I saw your self-checkout was now in operation. I was glad to see that you still had a reasonable number of human cashiers. There are definitely times that I would want to use a human cashier, even though I’m a technophile.

The user interface is a bit confusing. I noticed that there are large signs next to each self checkout to direct the customer. I majored in cognitive psychology in college, with a concentration on human-computer interaction. One of the design dictums I remember clearly is that systems which need extra instruction manuals are not well designed. The need for a sign shows that the system isn’t obvious to use.

In general, the checkout process went smoothly. You’ve mostly overcome one of the fatal flaws in the KMart design, which is the weight-check system. At KMart, if you fail to put each item into a bag, the system locks up until you can hunt down a human to help you. This makes purchasing large items very difficult. Your system seems to have a better design in this regard… except that it apparently cannot deal with whole chickens. The system refused to accept a two-pack of whole chickens as a valid purchase.

The process for accepting coupons is much better than KMart’s system. Your system had no difficulty accepting my coupons. KMart’s system used a coupon scanner similar to a dollar-bill acceptor, which had great difficulty accepting coupons. However, the receipt printout for the coupons is a bit confusing.

The credit-card acceptance procedure needs some work. There is a period of time where the credit-card swipe terminal appears to have been reset to the “swipe your card” state without any change on the main display. This is confusing, and made me wonder if the credit card transaction had timed out. The display implied that I might need to swipe my card again. I did not, because I know that other similar systems can do a double-debit of an account in such situations. A few seconds later, the system confirmed the credit card transaction.

The final flaw in the system is that the signature pad is awkwardly placed, very small, and the signature pen is attached with an absurdly short cord. This makes the signature procedure something of a hassle.

I also noticed a much longer than usual wait to leave the store, as the door checker was being far more thorough than usual. I hope that wait won’t be a new trend.

My experience with self-checkout systems in general has been that companies make them too paranoid to function effectively. As a result, they wind up angering customers because they make the checkout process longer and more difficult than it has to be. I hope that BJs will refine its self-checkout systems to prevent this syndrome. I also hope that the self-checkouts will be used to supplement human checkers rather than to replace them.

Thanks for your attention.

I sent this off with no expectations other than the hope that it would prove useful to the company… and that they might take some of my suggestions to heart.

I was very pleasantly surprised when I received the following letter today, a few short days after I sent my letter:

Dear Mr. Levandowski,

Thank you for taking the time to give excellent feedback on self-checkout. Your comments have been passed on to the appropriate Operations and Information Systems folks.

Enclosed please find a $50 BJ’s Gift Card for your time.

Sincerely,
/s/
Mike Wedge
President and CEO

This shows that Mr. Wedge “gets it,” as compared to the many ways that CompUSA did not. The response exceeded my expectations in many ways:

• The letter was personal, on Mr. Wedge’s stationary, and hand-signed by Mr. Wedge, indicating that he personally took the time to review the letter and draft a response.
• The gift card showed that not only was the feedback appreciated, but that it had a tangible value to the company.
• The quick response shows that letters from customers get prompt attention, and prompt replies. That’s a basic part of “customer focused” that most companies fail to understand.

I was already a happy customer of BJ’s Wholesale Club. Mr. Wedge’s excellent response to my letter made me a willing evangelist for the company. An exercise for any corporate bean-counters reading this: Will BJ’s Wholesale Club get $50 worth of word-of-mouth advertising out of this?

Let me be clear: I’m not advocating that everyone go out and start writing letters to companies in hopes of getting gifts. You shouldn’t expect this sort of thing to happen. It’s a wonderful surprise when it does, and I think companies would be well-served in adopting this strategy.

When should you write a letter like this? When you have something constructive to say. You should have a strong opinion that you can express clearly in a brief letter. It should be something that the company ought to be concerned about, but which may not be evident to their focus groups and polling and spot-checks. (If you are fishing for a reward, you’re more likely to get one if the letter is actually useful.)

Something to keep in mind: When a company rolls out a new initiative (like installing self-checkouts), the executives making the decisions may not have all the facts about how consumers like it. It’s the nature of any organization: as comments go from consumers to staff to managers to directors to vice-presidents and so on, they get summarized and prioritized. You may see some little problem that annoys you… and it may be something that would annoy the CEO, too, if it happened to him. If it’s a little problem, though, it might not be in the summaries he gets from management meetings.

Most of all, you should send a helpful letter when you’ve had service that’s just slightly off, and you care about the company. If you’d rather see them fix a glitch than just go elsewhere, it’s worth your time to write a letter. Even if you don’t get a gift out of it, you may still help the company discover the glitch and fix it before it becomes a problem that you, the consumer, won’t live with.

Where to send it? Well, as I stated, I went direct to the CEO because BJ’s didn’t have a customer feedback address. Just as you would escalate with a complaint, escalate constructive criticism: if the company has a feedback address, send it there. If nothing happens… it can’t hurt to send a letter to the CEO. Sometimes you can make a difference for just the cost of a first-class stamp.

The Self-Check Out machines do accept the Gift Cards now. The reason they did not accept them at first was the fact that BJ’s had just introduced the new plastic gift cards (as opposed to old-stlye paper gift cards) a couple months before the Self-Checkouts were put in stores. So the Tech people (where ever they may be) just need a little time to combine the two together.
Now if only they’d get the gift cards to work at the gas pumps. (They are working on this too supposedly)

-A BJ’s employee (who hates dumb customers, but likes the guy who wrote that letter to Mike Wedge)

Oh ya, btw, it doesn’t suprise me he sent you that Gift Card (although 50 is a suprising amount) the company has been trying to promote them like crazy! They even waived the non-member surcharge for non-members who used a gift card up until the end of Jan ’03

Posted by Rob @ 11/10/2002 12:32 PM ET

A followup:

I went to use the gift card today to buy groceries. (When you’re unemployed, $50 can buy a lot of meat for dinners!)

The self-checkouts cannot accept the gift cards directly. How ironic! There’s a small sticker on the credit-card swipe terminal indicating that, in order to pay by gift card, one must hit the button to get an attendant.

The attendant was on the ball, and quickly came to my aid. (She also saw a problem that the system was having when a plastic bag got stuck in the rollers, confusing the equipment, and came over to override it before I asked for her… very proactive!) I had to hit a button to pay at her central terminal, where she called up the order and processed the gift card.

Overall, it was a good experience… but somehow, I don’t think that Mr. Wedge intended his “thank you” gift to expose a new shortcoming in the self-checkout!