Sure, Agile has it’s problems but they are more related to the poor implementation and lack of reorganization that’s required than the process itself.

Man, you certainly have issues but they are probably more personal than related to Agile.

1. Do the most important things first
2. Always be ready to reassess what’s most important

Or, “spend your resources on the 20% of capabilities that will get you the best return.”

Waterfall is more like:
1. Figure out what you want to accomplish
2. Determine the most efficient way to accomplish it

It’s easy to see waterfall’s problems when characterized this way:
a. You have to know up-front what your end game is – which makes it hard to respond to market changes
b. The most efficient way of building the full product is not necessarily the one that front-loads the value, so often low-value items are completed and high-value items end up deferred
c. You do a lot of work up-front to document things that the developers never get to

Now, given my characterization of agile’s key goals, you have to look at agile *methodologies* just as a way of accomplishing them. Indeed, if the most important capability to deliver takes multiple sprints and lots of architecture, you still have to do it. And if the methodology doesn’t give you a way to do it, then the methodology won’t work for that product.

But, on the other hand, most large monolithic capabilities *can* be broken down sensibly into multiple passes through the “smallest thing that could possibly work” approach.

Agile is *not* a silver bullet, and it’s hard to get right, but if it helps you focus on putting first things first and executing on the 80/20 rule, it’s done its job.

That said I think you’ve got a few wrong notions.

First, the problem of people not spending enough time thinking broadly about the right things has nothing to do with agile methods. Teams using waterfall-ish processes fail equally at this, but differently. They spend oceans of time on documenting and arguing, rather than thinking. Or they go off in cloud-cuckoo-land, analyzing, researching, and designing pure fantasies, ones with little relation to actual problems and achievable solutions. Then they stop thinking entirely for months or years, because it’s the wrong phase for thought.

If you want to take time to think, there is nothing in any Agile book that says you can’t. All they say is that if you do have developers around, you need to give them enough to do that they aren’t twiddling their thumbs. So regularly set aside time for research and examining the bigger picture. No Agile guru is making you work at a white-hot pace, and developers need time to think, too.

Second, not hiring enough product managers is not the fault of the method you’ve chosen. That developers are working faster and focusing on what they’re good at isn’t a problem unless you let it become one. If not enough product management is getting done, hire more people. That can include full product managers to share the load, junior product managers to take care of grunt work, full-time researchers to do field studies and report back, and assistants to handle the donkey work of collecting data and reporting it to stakeholders.

Third, plenty of people doing Agile processes take time to understand their customers. At the big Agile conference this year, there was an entire track filled with presentations about how people are making it work. The trick is to take it seriously and devote adequate resources to it. Product managers are the bridge between what people need and what gets made, and a bridge that’s narrow on one end but wide on the other just doesn’t carry traffic well.

Fourth, if your teams are always vaguely panicked, that’s a cultural issue, not a process issue. Sustainable Pace was one of the twelve original Extreme Programming practices. The Yesterday’s Weather practice also acts to force a reasonable schedule. I’m constantly coaching teams to take breaks, go home on time, and generally lead sane lives. If your leaders don’t value a measured pace, no process will fix that, because you’ll just ignore those parts of the process. As many are with their Agile adoptions.

Fifth, introverted developers work just fine in a well-run war room environment. DeMarco’s research didn’t study Agile teams at all, and loud people can run over quiet ones in any environment. Putting them all in the same room can make the problem more obvious, but it’s not like that can’t happen in the meetings, memos, email arguments, and political maneuverings of a waterfall project. The solution is to fix the culture and the relationships, and that’s easier to do when everything’s out in the open.

So in sum, I’d agree: Agile isn’t a silver bullet, and you should stop expecting it to fix every problem a team has. And take the fad-driven wave of dodgy Agile implementations more as evidence that following fads is bad, and less that Agile itself is.

Since agile does solve important software engineering problems well, it seems wrong-headed to throw the baby out with the bath water. Rather let’s identify the mismatch between traditional Product Management and agile IT methodologies and then figure out what is required to effectively interface with an agile engineering group.

In my view there are two aspects or roles of Product Management: Strategic Product Management and Tactical or Operational Product Management. The strategic folks interact with customers and think the “big picture” and long term thoughts about product direction. In my experience, these product managers aren’t the right people to work with IT. There needs to be another layer of Product Management that I refer to as tactical or operational Product Management.

A Tactical Product Manager (TPM) is something akin to what used to be called a systems analyst. The key difference between the traditional systems analyst and the TPM is that the latter is schooled in how to decompose product requirements into agile user stories. The TPM, unlike the strategic Product Managers, has skin in the game – ie, is a participant in the weekly iterations and accountable with the dev team for completing the committed to work package of stories for the iteration.

As part of an agile development team the TPM ensures that there are appropriately sized user stories prepared for each iteration that are aligned with the organization’s strategic product direction. The TPM is also a real-time clarifier of requirement issues and tries to stay a step ahead of the coders by analyzing work in progress for possible requirement gaps or conflicts. As part of the Product Management team the TPM is able to see how selected user stories build into releasable feature sets and serves as the interface between development and Product Management to determine the releasable stories or groups of stories for a release.

My sense of your crankiness is that you’ll mellow some if a staffing strategy such as I propose above were implemented in your organization. The challenge is finding or training TPMs because traditional systems analysts (aka requirements analysts) aren’t always a good fit for agile shops – they often love big, up front analysis and thick MS Word documentation. Finding good people is a challenge on the IT side as well though so it should make you happy, in some cranky way, that the IT manager is struggling with the same issue you are.