Let the Cranky Product Manager remind you all of the purpose of Beta testing programs: To get customers to actually use your about-to-be-released software. Why? To find and fix problems that would not have been found by internal-only testing.
So WHY then, WHY(!?) do so many so-called Beta testing programs seem explicitly designed to PREVENT this type of feedback?
Let the Cranky Product Manager classify and explain the different types of Beta programs, of which the vast majority do NOTHING to improve product quality or identify customer issues.
1. The “Don’t You Dare Complain Because This is Free” Beta
This type of Beta is otherwise known as the “Google Beta”. An Example? Take Google Gmail, which put out its Beta release on April 1, 2004. As of this writing in December 2008, 4.5 years later, Gmail is still officially in Beta, with no word for an official release. Ever.
Clearly, the purpose of this type of Beta is NOT to collect customer feedback. More likely, the “Beta” designation is a euphemism for “No Support. Take It Or Leave It. Don’t Complain To Us If It Doesn’t Work Because It’s Free.”
If there is no mechanism in place to actively contact Beta customers and solicit their feedback (as opposed to just relying on them to send messages to a forum or send YOU an email), and if there is no path to ever release this product, well it’s a “Don’t Complain Because It’s Free” Beta.
2. The Paper Beta
This is where a company has a ridiculously short Beta period, and as a result recruits very few (if any) test customers and thus gets very little feedback. The whole goal here is to NOT uncover issues. Instead it is to push a piece of crap out of the door on a certain date, regardless of (lack of) quality, while still being able to honestly look customers in the eye and tell them your product was Beta tested. Because, yes, savvy customers ask that.
(Even savvier customers ask how big the Beta program is, how many customers participated, how long it lasted, and how many issues were uncovered and subsequently fixed.)
Result? There is no real effort to sincerely, honestly collect feedback. And any feedback uncovered will probably be ignored – because “it’s too late to fix it now.” Bugs will be filed, and possibly, MAYBE, they’ll be addressed in the next production release.
If your Beta uncovers less than 10 issues or is less than 4 weeks long, congratulations, you had a Paper Beta.
WARNING: Running a Paper Beta will DOOM your efforts to recruit customers for any future Beta program – Paper or not. The few customers that do participate be incensed that they spent time installing, testing, and providing feedback on your software, only to have their issues summarily ignored and potentially never fixed.
3. The “Agile” Beta
Allergic to any form of process, some Agile proponents will claim their awesome methodology eliminates the need for expensive Beta programs. Why? Because, supposedly, with Agile the product is “Beta every day,” or at least it is at the end of each sprint or iteration. And part and parcel of the Agile philosophy is getting direct customer feedback throughout the entire development process. Ergo, the claim that no Beta Program is needed.
All this is fine and good, but it still does not obviate the need for a Beta program. Because without getting actual product into the hands of actual customers, you won’t uncover the problems that can’t be uncovered in internal-only testing. It is ESSENTIAL that the customer actually attempt to use the software ON THEIR OWN, for things THEY actually want to do. If they don’t and you just provide a demo or screenshots (as is often the case for Agile-style “in process” customer feedback), well TOO BAD it doesn’t count.
How did you get into this sad situation? Most likely, it’s because you’re letting the Development organization push you around. Because, after all, if the Beta program is eliminated, their jobs become easier and less work. Plus, they can now push their risky bug fixes to the very last minute – as in minutes before that “final build.” This is extremely dangerous yet happens all too often.
4. The Haphazard Beta
The Haphazard Beta is one where you get the product to a particular state, claim it’s Beta, and THEN try to figure out (last minute!) how to actually recruit some customers and get some usable feedback. There’s no organization, and no directed attempt to collect and quantify feedback.
It’s a mess. A big loosey-goosey, panicky mess.
And it’s your fault, Product Manager. Your fault.
If you had been on the ball and actually done some recruiting ahead of time, along with a smidge of planning about how to collect and handle issues, well you might have actually had a REAL Beta. Alas, it’s too late to fix it all now. So hold on tight, pray you don’t lose your job, and promise to do better next time.
5. The Real Beta
This is the Beta program done right. Real customers and prospects actually use the software and identify issues (at the bug level, workflow level, and higher levels) that you could not have found on your own.
And THEN… here’s the kicker…. you actually DO something about the issues they find. That is, YOU ACTUALLY MAKE CHANGES. You fix the bugs. You fix the work flows… You learn how to make your product better.
And it all goes smoothly.
You have recruited enough customers – in both number and variety - to accurately represent your target market. You have enough time to fix their issues. You care enough about putting out a quality product that the final product is something that you can be proud of – something that won’t create a PR disaster requiring a zillion-dollar advertising Band-aid, like the well-known disaster below…
{ 19 comments… read them below or add one }
I think there’s another type of beta: the facade. This is when a company puts on a beta because that’s what you’re supposed to do, but the engineers don’t really pay attention to the honest feedback on the products. Those who participate in the beta feel like no one is listening to them anymore, and the PM works in damage control mode. I’ve been on both the customer side of a facade beta, and it is NOT fun.
Greg, the “Facade Beta” sounds like what the Cranky Product Manager was trying to describe (clearly inadequately) as a “Paper Beta”. You are very right about the “Beta As Meaningless Ceremony” approach.
I guess I was thinking of betas that were sufficiently long, but where some of the engineers really don’t care about the feedback on the product, so they argue with those few suckers – I mean customers – who participate in the beta.
I’m curious to hear how betas are managed at other organizations. The most successful I’ve seen have the attributes highlighted by CPM and are managed by PM but executed through the sales and professional services teams. I think these worked the best for three reasons:
1) Sales gets early exposure to the product and feels engaged to sell it harder post beta
2) Services gets early exposure to the product and feels engaged to support it better post beta
3) Betas are a buttload of work and take lots of people to do properly.
John
http://crankypm.com/2008/12/types-software-beta-testing-programs/
If alleged agilists are saying things like that, then they are doing it wrong.
Releasing early and often is an Agile mantra. At the end of every iteration, the team should have something that they can put in the hands of users. From a product perspective, it may not be feature-complete, but what features are there should work just fine.
You can support that from a product management perspective by choosing features in an order that will get as quickly as possible to the point where it’s useful to at least a small segment of your audience. To manage expectations, you need not even call that a beta. Just so long as people kick the tires.
The sooner you can get real feedback from real use, the better an Agile approach will work. Especially, as you say, if you actually incorporate that feedback into your product plan.
Great post !
However, the problem is often HOW to get quality feedback from “quality” customers.
We really, REALLY want to create a GREAT Windows desktop application (the world has enough crappy products already).
We’ve been through the private beta stages, have received great feedback and improved our product.
However, we are now in the last stage where we have launched a public beta and try to get feedback from various customers in various markets. As the quality improves, it takes more effort from the users to identify problems, and getting dedicated testers is HARD !
The final test will be when we close the beta and officially launch, but the amount of users during the beta period just cannot match the amount of “real” users for an “official” product.
Any tips are appreciated :-) !
Atle
And then there’s the “sales beta”. Kevin, the world’s worst salesperson, says, “The next release isn’t available yet but I can get you the beta version to install and implement.” There’s no feedback to the dev team and suddenly customer support starts getting calls for a product that isn’t supposed to be in the field. Damn, I hate bad sales people.
Hi, shameless plug here, but if you’re looking for guidance on running a “real beta” as Cranky calls it, try this article.
http://www.pragmaticmarketing.com/publications/magazine/4/3/0605sk
Please leave feedback on the article if you find it useful.
Saeed
Steve,
PMs should absolutely put their foot down on the sales beta. There’s no valid reason to put the customer or the company through that.
Saeed
Wow, just wow. This is Geoff after the team holiday lunch and a couple of cocktails, but your post really hit some highlights of bad-ness in Beta land.
I live in Hardware world, where customer feedback means more than just “We gotta fix this widget functionality”, and usually means “oh shit, we get flames shooting out of XYZ module in a physical sense.” Sadly, we are usually so far beyond schedule in shipping that we ship the “beta” with such a short cycle (60 or 90 days) where we KNOW deep in our hearts that unless there is a major SNAFU, the Beta == final release, and beta is really just checking a box off on the official PLC.
I have been railing against this practice for YEARS, but have been told to STFU, and just go with it.
If what we ship as beta is final, we should be honest with our ‘early adopter” customers, and not have the charade that is called “beta”.
Geoff
I was gonna comment on the “sales beta”, too, but Steve beat me so it. I consider it universally a bad idea to put unready software in the hands of people who are considering a purchase. I have found that the support organization is a strong ally in slaying the sales beta if you need political capital to do it. However, I’ve put my foot down so forcefully in the past that people no longer ask.
A long time ago, in a galaxy far, far away…no, wait…a long time ago, I once worked as a tester (yes, I’m one of those) for a smallish company called VM Software (later Systems Center, then Sterling Software, and now CA). Each major release of our products underwent a real beta. We actively recruited beta customers (often from our more “difficult” customers), had a specific start and end date, scheduled frequent follow-up calls with all beta customers, and regularly fixed the problems they found. Since leaving the mainframe world, I have yet to see a real beta program. The feedback provided during a beta (and even alpha) test is invaluable and improved the quality of our products immensely (primarily uncovering severe defects due to configuration and scale differences).
Excellent post, CPM. Even though I’m not a PM, your posts definitely strike a chord. You are spot on.
Potential “shameless plug” but actually I could use your collective wisdom…
Disclaimer: We are in the very throes of recruiting beta testers (details at end).
Our product was developed because of frustration on the part of a Product Manager (imagine!) who has a product that licenses code from a third party. Said code is expensive. PM has NO IDEA if it is used by 7% or 93.862% of customers. S/He is responsible for profitability (of course), and would love to make the 3rd Party Code a free downloadable feature and avoid the cost per unit fee.
Sooo… I have two issues I struggle with:
1. Finding product managers who understand the issues and are actual PM’s (I have an serious over-supply of business execs and development types who want this, but it isn’t really targeted at them).
2. Understanding the best way to approach the PM crowd so that I don’t poison my primary audience.
Software details: Specific intent of providing Software Product Managers fact-based metrics on customer product usage, sort of an ‘On-Star’ for Desktop Apps.
Beta details: Product Manager must be primary contact. Software (today) needs to be .NET (not yet ASP.NET), C# or C++ preferred, but others are ok. We don’t charge for the beta, but don’t offer a complete free ride, it can be ‘earned’ in time chunks based on active participation and value of feedback. Deployment is very simple. Once deployed, the PM can re-instrument on-the-fly.
I’m surprised that no one has asked whether Product Managers really SHUOLD be running the beta. It’s a TEST. If we assume that it’s part of the quality process, the whole topic shifts.
What if there was someone in QA who was responsible for organizing and running beta programs? At one company where I worked, we had one QA person who was a central resource for betas. She would work with the Product Manager and Development team to figure out the entire program — and she was focused on the primary goal of the beta, which (of course, right?) is to improve quality prior to release.
This had two primary benefits. First, beta programs brought in good feedback. The programs were built right into the release schedule, so there was time allotted to fix things. If there wasn’t time to react to the beta feedback, the Beta Program Manager would be shouting. She didn’t want to waste her time if quality would not be improved.
The second benefit was that we ran more beta programs. As a Product Manager, you knew that you would have support, and that betas could be successful. We changed from doing beta as an after-thought to asking very early on, “is there any reason why we would NOT beta this release?”
If you’re not going to effectively respond to beta input, why do them at all? I’m pretty sure you’ve got other stuff to keep you busy ;-)
Interesting article on Beta Testing – http://tinyurl.com/5ypu7r
And what about the “phantom beta”? One where you give customers a long-enough time to test, you recruit a good cross-sample of use cases, and then testing isn’t arduous enough to really ferret-out the corner-case defects or identify those examples you can only see in a “near production” or production environment?
The most popular products will always have anxious hard-core users that really want to pound the crap out of your product. But what about the “marginally interesting” product where generating excitement to really test it is needed? I’ve had to buy steak dinners sometimes to ensure proper thorough testing!
Greg,
So my question would be, why is someone building a “marginally interesting” product to begin with? Can you give an example?
I found this while doing some beta "best practices" research. Funny!
http://bit.ly/ioRwD
{ 3 trackbacks }