Why I hate product comparisons

I read a lot of “Product Comparison” blog posts, a consequence of working in a software firm. Most of them are really not very good. So what’s wrong with many of these posts?

Well generally the bad posts aren’t product comparisons at all. Rather they document someone’s experiences with trying to find a solution to suit a specific organisation or set of needs. This is sometimes interesting and sometimes useful. But interesting only as a subjective point analysis. These posts are far from actual product comparisons.

It’s pretty easy to identify the two categories. Let’s examine these two statements:

"We needed to achieve x and Product Foo had a widget for doing x that we
liked better than Product Bar's widget. This is how we implemented
the x widget."

And:

"When doing x with Product Foo and Product Bar we observed that:
  - Product Bar was y times faster than Product Foo
  - Product Foo works on platforms a, b, and c whilst Product Bar only
    supports platforms a and b.
  - Product Bar has a comprehensive API and documentation whilst Product
    Foo needs more work in this area and didn't cover how to do d, e, and f."

One of these is an organisation-centric analysis of a product and the other is a product comparison.

So why does the difference matter? It matters because it impacts how and why we choose new products. Selecting a product based on someone else’s selection criteria makes for a poor technology choice. To avoid this:

• Gather your requirements
• Find the quality product comparison posts, the ones that use objective criteria
• Use these posts to stock your short list of products
• Test the short list against your requirements and document
• If you do document your choice in a blog post then be clear about your selection criteria and document the sources of your information