Comments on: The 5 Types of Beta Testing Programs and Why 4 of Them Suck

By: Greg Saiz

Greg Saiz — Tue, 14 Jul 2009 05:46:15 +0000

I found this while doing some beta "best practices" research. Funny!
http://bit.ly/ioRwD

By: Saeed Khan

Saeed Khan — Thu, 02 Jul 2009 19:11:47 +0000

Greg,

So my question would be, why is someone building a “marginally interesting” product to begin with? Can you give an example?

By: Greg Council

Greg Council — Thu, 02 Jul 2009 19:06:53 +0000

And what about the “phantom beta”? One where you give customers a long-enough time to test, you recruit a good cross-sample of use cases, and then testing isn’t arduous enough to really ferret-out the corner-case defects or identify those examples you can only see in a “near production” or production environment?

The most popular products will always have anxious hard-core users that really want to pound the crap out of your product. But what about the “marginally interesting” product where generating excitement to really test it is needed? I’ve had to buy steak dinners sometimes to ensure proper thorough testing!

By: sheac

sheac — Tue, 10 Feb 2009 01:12:21 +0000

Interesting article on Beta Testing – http://tinyurl.com/5ypu7r

By: Stacey Weber

Stacey Weber — Mon, 12 Jan 2009 17:10:13 +0000

I’m surprised that no one has asked whether Product Managers really SHUOLD be running the beta. It’s a TEST. If we assume that it’s part of the quality process, the whole topic shifts.

What if there was someone in QA who was responsible for organizing and running beta programs? At one company where I worked, we had one QA person who was a central resource for betas. She would work with the Product Manager and Development team to figure out the entire program — and she was focused on the primary goal of the beta, which (of course, right?) is to improve quality prior to release.

This had two primary benefits. First, beta programs brought in good feedback. The programs were built right into the release schedule, so there was time allotted to fix things. If there wasn’t time to react to the beta feedback, the Beta Program Manager would be shouting. She didn’t want to waste her time if quality would not be improved.

The second benefit was that we ran more beta programs. As a Product Manager, you knew that you would have support, and that betas could be successful. We changed from doing beta as an after-thought to asking very early on, “is there any reason why we would NOT beta this release?”

If you’re not going to effectively respond to beta input, why do them at all? I’m pretty sure you’ve got other stuff to keep you busy ;-)

By: Michael Lundberg

Michael Lundberg — Tue, 30 Dec 2008 18:14:33 +0000

Potential “shameless plug” but actually I could use your collective wisdom…
Disclaimer: We are in the very throes of recruiting beta testers (details at end).

Our product was developed because of frustration on the part of a Product Manager (imagine!) who has a product that licenses code from a third party. Said code is expensive. PM has NO IDEA if it is used by 7% or 93.862% of customers. S/He is responsible for profitability (of course), and would love to make the 3rd Party Code a free downloadable feature and avoid the cost per unit fee.

Sooo… I have two issues I struggle with:
1. Finding product managers who understand the issues and are actual PM’s (I have an serious over-supply of business execs and development types who want this, but it isn’t really targeted at them).
2. Understanding the best way to approach the PM crowd so that I don’t poison my primary audience.

Software details: Specific intent of providing Software Product Managers fact-based metrics on customer product usage, sort of an ‘On-Star’ for Desktop Apps.

Beta details: Product Manager must be primary contact. Software (today) needs to be .NET (not yet ASP.NET), C# or C++ preferred, but others are ok. We don’t charge for the beta, but don’t offer a complete free ride, it can be ‘earned’ in time chunks based on active participation and value of feedback. Deployment is very simple. Once deployed, the PM can re-instrument on-the-fly.

By: Beta get it right or else… « Web Product Management

Beta get it right or else… « Web Product Management — Mon, 22 Dec 2008 22:48:34 +0000

[...] or else… I’m a fan of “The Cranky Product Manager“. In one of her recent posts she talks about beta testing and boy did it ever hit close to home. As I step into my way back [...]

By: Za kraj (početak) godine « Form & Essence

Za kraj (početak) godine « Form & Essence — Mon, 22 Dec 2008 10:46:05 +0000

[...] THE 5 TYPES OF BETA TESTING [...]

By: Teri S.

Teri S. — Sat, 20 Dec 2008 23:09:26 +0000

A long time ago, in a galaxy far, far away…no, wait…a long time ago, I once worked as a tester (yes, I’m one of those) for a smallish company called VM Software (later Systems Center, then Sterling Software, and now CA). Each major release of our products underwent a real beta. We actively recruited beta customers (often from our more “difficult” customers), had a specific start and end date, scheduled frequent follow-up calls with all beta customers, and regularly fixed the problems they found. Since leaving the mainframe world, I have yet to see a real beta program. The feedback provided during a beta (and even alpha) test is invaluable and improved the quality of our products immensely (primarily uncovering severe defects due to configuration and scale differences).

Excellent post, CPM. Even though I’m not a PM, your posts definitely strike a chord. You are spot on.

By: Mark

Mark — Sat, 20 Dec 2008 16:56:08 +0000

I was gonna comment on the “sales beta”, too, but Steve beat me so it. I consider it universally a bad idea to put unready software in the hands of people who are considering a purchase. I have found that the support organization is a strong ally in slaying the sales beta if you need political capital to do it. However, I’ve put my foot down so forcefully in the past that people no longer ask.