The question, of course, is really “is it done enough yet?”.

We finally shipped ProSphere 1.0 at the close of July.  I am not surprising anyone when I say that wasn’t the first release date we had in mind, nor am I surprising anyone when I said we shipped with a backlog of items we wished had made it into 1.0.  But at some point you need to ship it and move on.

Today I have a slightly different team; a few people have changed responsibilities to allow the teams to better meet the needs of our next release.  And while I work to integrate that new team together, I also stew over a laundry list of requirements and enhancements to our product for the next release.  Resolving priority conflicts, providing gross estimates, learning new areas of the product, and making sure nothing slips through the cracks — it’s a busy time.

It’s a totally different kind of busy than when trying to release a product — but it’s just as important.  The context shift is jarring, but the quality of the work done today will impact our working lives for the coming months, and have a direct impact on the satisfaction of our customers with the next version of this product.

This is, of course, the job we all signed up for.  It’s an exciting time.

This post is from: Dave Talks Shop

Ship it, move on!

Fifteen years later, I’m writing about a new big splash for EMC in the SRM space – ProSphere 1.0. I’ll stop you right here and tell you that you need to go read Chuck’s post on the product. I can’t out-do his deep-dive into the industry angles and why it’s such a big deal, so I won’t even try.

What I will tell you is why working on this product was so different from any other product I’ve touched at EMC, and why I’m so proud to be able to announce it here. Just like fifteen years ago, it was a chance to take a look at everything “wrong” while also still looking in new directions at the same time the industry is making another scale leap with Cloud environments. This has been some of the hardest work I’ve done here at EMC. But seeing it get out the door is making it all worth it.

EMC ControlCenter has a decade-plus roller-coaster history.  It’s been through some challenging times, on many fronts, but we’ve never stopped trying to improve it for our customers.  We often talked about things we’d like to do differently, given the chance.  By the middle of 2008, a vision for the future was firming up – and we had a name for it.  “SRM 7,”  reflecting our goal to avoid “growing” ControlCenter to a new major version after 6.x.

We handed out T-shirts with a stylized “7” inside a diamond shape (which might evoke a certain superhero).  The (perhaps dangerous) implication?  SRM 7 was going to fly in and save the day.  We were all a bit skeptical.  But today we’re announcing the first shipping release of that product – ProSphere 1.0.

Why do I think it’s such a big deal?

We didn’t rebuild ControlCenter

Early on, we faced a critical decision – do we rebuild ControlCenter piece by piece, or do we build a new solution from the ground up?  We knew part of the issue with ControlCenter was feature creep.  We wanted to focus on critical customer use cases and build the application that could do those, and resist the temptation to build a giant unwieldy Swiss Army Knife.  That philosophy bled into everything.  We didn’t build a giant infrastructure that could address all our needs; we architected an extensible solution and implemented enough of it to get us through the use cases we were attacking.  We avoided writing “just in case” code to support possible future features.  We didn’t build what we could reuse.  We knew we couldn’t ever finish this in time if we wrote it all from scratch, so we pulled in proven components from other shipping EMC products and integrated them.

Further than that, we didn’t want to be the single best interface for every deep use case – we knew you wanted to use the right tools for those jobs.  So we made a plan — find those tools, link to them, launch them in context, and make your sign-in transparent – and in the process eliminate thousands of lines of code which need to be tested, debugged, upgraded, and so on.  It’s a win on every front.

You’re never going to have a “lean” piece of software to do the giant job of managing the storage for your entire enterprise.  But ProSphere is downright svelte compared to ControlCenter, and we intend to keep it that way.

It was ok to look outside

Another shift we made in ProSphere was to look outside our traditional sources of software components.  We didn’t want to write, maintain, and test unneeded software.  We didn’t want to architect and design unneeded components.  We pulled in open source software, we used open standards, and we got creative.  It was a learning curve for the development team, but in the end we have a product that communicates using known web technologies and patterns, and which we hope will serve as the foundation for an open, extensible management solution.

This extends deep into the product’s DNA, not just a superficial claim about what our APIs look like.  Eliminating traditional agents, using HTTP to communicate between our machines, going to a virtual appliance model, using industry-standard system monitoring  … all these things make the system higher quality and more extensible with less code to carry.

We got Agile

One of our early decisions was to abandon the waterfall software method we had more or less down to a science in ControlCenter, and replace it with an enterprise-scale Agile development approach.  In my personal opinion, two excellent things came out of that decision: we “forced” our development managers to take ownership of use cases within the product, and we created cross-functional (development, quality engineering, documentation, product management, user experience design) teams to tackle those use cases in close collaboration with each other.  In my fifteen years of software development, this is the closest cooperation I’ve seen between functions on a project team of this scale.

We built a safety net

Last, but most certainly not least, this product has the most aggressive safety net I’ve encountered.  Every day, thousands of automated tests run against the system in various forms – unit tests against the code, integration tests against the REST interfaces, automated UI tests against the finished product in a test environment (deployed automatically after every finished build), and even more automated UI tests against the finished product in a “real” environment.  Individual developers had access to a simple web-based tool to deploy a build, patch it, and run a suite of hundreds of automated regressions against it prior to delivery – all from their desks, without ever seeing an installation screen.

We combined this with static analysis tools constantly analyzing the source trees to check for bugs, security lapses, stylistic violations, and undesirable complexity, with web-based dashboards for everyone to see.  Finally, there’s a dedicated team doing manual run-throughs of customer use cases in a variety of real-world environments.  You take all that and combine it with an organizational mandate not to tolerate technical debt, not to tolerate little bugs accumulating in the system, and you’ve got a team that understands what quality means and has an extremely low rate of regressions.

Part of what took us so long was building all that scaffolding, and changing the culture of our development and test organizations to get us here.  And now that it’s built (not that it’s ever “done!”), it will pay for itself for years to come.

In conclusion

I’ve been involved on this product for about three years now … the same amount of time I’ve been a father.  And like any Proud Papa, I can’t wait to show off.

We built a solid foundation – a highly deployable, serviceable, and usable application – and concentrated our efforts on a small, tight family of use cases using that foundation.  We’re already working on what’s next.  I think we’ve changed the game here.  I’m proud to be a part of that.  I was just one manager, with one small team, working on little bits and pieces of this giant project.  I’m grateful to my team for consistently seeing the big picture and working nonstop to get us there.  I can’t begin to explain how proud I am of the work they did.

It hasn’t been easy; a lot of blood, sweat, and tears are flowing in the hallways.  But nothing worth achieving is easy, and there are a lot of smiles today as we finally take the wraps off this.

ProSphere 1.0 is just the first step.  It’s your turn now.  Check it out.  Tell us what we’re doing right, what we’re doing wrong.  Help us take this to the next level.  I promise I’ll listen and do my part to make sure your voice is heard.

This post is from: Dave Talks Shop

Introducing ProSphere

Two colleagues of mine demonstrated to me today a habit which, on its surface, can be difficult to understand.  They really enjoy asking the stupid questions.  We’ll be in a design review, looking at a fairly straightforward diagram showing a few components interacting, and they’ll ask what customer use case this represents and what those components are.

We all know the answer.  It’s fairly obvious. But the person writing the design didn’t label the components and left it open to interpretation.

My first instinct was to jump in, say it wasn’t important, that we needed to get to the guts of the design to review it.  I was wrong.

The benefit of asking the stupid question is that the stupid answer you get just might not be the one you expect.  It’s so obvious that everyone just assumes their answer is right.  They probably are.  But when five people each feel that way, and nobody makes sure all five are thinking of the same stupid answer, you’re probably in a situation where there is more than one stupid answer out there.  And as work progresses, more and more assumptions get built atop those shaky foundations, until someone finally figures out the flaw in communication.

By then, everyone is so invested in their own interpretation that there are bound to be difficult conflicts, messy redesigns, and more.

So ask the question up front.  Don’t be afraid to look dumb.  One of these times you’ll spot the problem a month early and it’ll all be worth it.

This post is from: Dave Talks Shop

Ask a stupid question …

photo credit: Extra Ketchup. This is not me … but it practically could be!

I saw Google last week proclaiming the impending death of the desktop computer, in favor of ubiquitous mobile computing with computing power provided by the cloud.  I see more people replacing their gaming PCs with consoles, their family desktops with notepads, and their notepads with iPads and iPhones.

As a geek who loves gadgets, I’m not opposed to this.  I love progress, and I love the shiny new technology we have access to now.  But I can’t help but look at what happens when a young kid first boots up a device like this.

Imagine a world where the iPad is the ubiquitous platform of choice.  Where do you get applications for your iPad?  From one vendor: Apple.  How do you write an application for your iPad?  You ask Apple for permission.  You apply to be a developer, ponying up $99 a year to do so.  You buy the specific hardware they support, so you can develop and test.  You learn their proprietary SDK, write something, and then want to share it with your friends.  How do you do that?  You ask permission again, before they’ll put your software on their store.

These are not huge barriers to people who seriously want to develop iPhone or iPad software, as demonstrated by the 100,000+ applications currently available for download.  But it’s a moderate barrier to the hobbyist, and an insurmountable one to the middle-school kid with a dream and some spare time, and a further insurmountable one to the person who just wants to experiment and share with friends and has no desire to publish to the world.

I don’t want to make Apple look like the villain here.  Consumers are demanding easier to use devices that “just work” and companies like Apple, Sony, and Microsoft are stepping up.  But I have to wonder what impact this is going to have on the future generation of software engineers who are being born today (or who were born 3 to 5 years ago).  Will we have a generation of people who are expert users but have no inclination to build?  Or will the definition of “build” change in some way?

(Thanks to Michael Surran for the completely awesome photo on flickr which I am using in this post.  His use of the Creative Commons license has made it possible for me to show you exactly the image I wanted for this post with a clean conscience.)

This post is from: Dave Talks Shop

iWorry about our progammers

The concept: write a project where multiple EMC developer technologies can be used in a single program.  Bonus points for incorporating other online technologies.  Win money and fame and the adoration of the world.

I’m waiting for the detailed T&C, but you can read up more on it here.  Innovation through contest is something EMC has tinkered with quite a bit, as you may have read on Steve Todd’s blog last year.

Definitely check out the link for more info. I’m hoping I can find some time in between all my “real work” to put a couple of these tools through their paces.

This post is from: Dave Talks Shop

EDN puts its money where its mouth is

It was hard to create, it should be hard to use (or maintain).

Basically, we worked hard to get this stuff done and we expect you as a user or future maintainer to put the same effort into it.  After all, it took many man-years to write the software, it’s not too much to expect you to spend a few weeks reading manuals and understanding it before you start complaining that it’s hard to use.

As Paul Young recently wrote, though, imagine if wood-chippers took that approach.

Imagine if an author did?  “It took me years to write this novel, you should have to do some research before you read it.”

Some do, I guess.  I’ve read a few novels that require major work to get through.  Sometimes the end result is even worth the work.  But as my fiction writing friends tell me, in general you don’t want your readers to be thinking about your writing, you want them thinking about your story.  Similarly, you don’t want your users thinking about your software design, you just want them thinking about the task your software enables.

I feel the same way about maintaining and testing software.  We want developers thinking about the code, not about the way you wrote it.  You don’t want someone looking at your code, peering at it for a few minutes, and then saying, “Oh, I get it.  Wow, that’s clever.”

There’s a famous quote attributed to a half-dozen different writers (and perhaps originated by Blaise Pascal), that says, basically, “I am sorry I wrote such a lengthy letter; I did not have time to write a short one.”  It takes time to create simple, elegant software.  When we force the issue and compress the time spent on a project, you end up with complex code and complex user interactions.  We should consider this a problem, not a point of pride.

When we present some difficult software to our users, we should apologize to them.  “I’m sorry this UI is so complex.  I didn’t have time to make it easier.”  Instead, we make them feel guilty.  “Ah, perhaps you should have taken the training,” or read the manual more carefully, or attended our seminar.

Think about the people on your team, and ask yourself if they “get” this concept.  Realize, that if they don’t, you’re eventually going to lose your market share to a competitor who does.

This post is from: Dave Talks Shop

Simplicity is a virtue

We’ve staffed this team with people who have worked on successful, shipping products.  Many of them “grew up” in a world where certain things (like, say, continuous integration, or test-driven development) were unheard of.  There are inevitable growing pains as we push for certain behaviors and results.  Sometimes there’s a failure in the software pipeline and the answer from the responsible party is “This wouldn’t have happened if we had XYZ in place,” where XYZ is a practice from some product they used to work on.

As a technical lead, it’s my responsibility to try and educate people and raise the quality of everyone’s work.  I’m supposed to push for the best practices we all need to understand and employ.  But I also need to make sure a product goes out the door.  So when I’m in a closed-door meeting with senior management and they say, “Do you think we should put XYZ in place?” I have to stop and ponder.

The answer, in my heart, is no.  Take away the crutch, make the team fail a few times.  People learn from failure.  In a few months they will adapt and be better for it (or they will have gotten fed up and left).

The answer, in my brain, is yes.  I know that without this crutch, they will be less productive for a few months.  And I know where we’re supposed to be in a few months, and I’m not sure we can get there without the crutch.

I tell management both these answers.  They ask me what we should do.

And this is why our jobs are hard.

(I usually end up leaning towards the practical side here, and hope that we can educate in parallel.  But I’m always left wondering if I should be more extremist….)

This post is from: Dave Talks Shop

Doing the wrong thing for the right reason

My organization made a recent decision to formalize the technical lead role a little bit within their scrum teams.  We did it because we felt there was a gap not being filled, and we felt like our ability to execute and ship a product was being impacted.  A colleague of mine forwarded me an article about the agile “team lead” role, and I was more interested in the comments than the actual text — people discussing whether the role was strictly agile or not.

I’m sorry, but I could not care less.

I’m reminded of a recent post by Mike Cottmeyer (who, by the way, gets paid to help companies adapt agile) where he argues that adopting agile isn’t the point.  The point is to add value to the business.

The point isn’t to play with new technology, to be open-source, to write tests, to measure coverage, or to adhere to standards.  We do these things when the time is right because we believe (based on evidence) that doing those things will lead to specific results.  Scientists can’t afford to be religious about their science, and computer science is no exception.  Don’t stop asking questions.  Don’t stop challenging your leaders, your preachers, your prophets.  If you feel like your practices are set by robed oracles atop a mountaintop, something is wrong with your organization.  Now’s as good a time as any to push back and find out why.

Because in the end, we’re here to sell software that delights our customers and keeps us in business.  Everything else is peripheral.

This post is from: Dave Talks Shop

Keep your dogma leashed

For example, contrast these two anecdotes:

Ben knows his daughter is home sick from school today, and he’s worried about her.  He keeps getting a weird exception coming from apache and he can’t figure out why.  He’s got half the team in his cube, and as soon as they’re done he knows he can go home and check on his daughter.  But no matter what, nothing works.  It takes him until 9 PM, and it ends up being something really stupid and obvious.  He’s mad at himself for missing it, and mad at the team for not finding it.  He leaves frustrated and angry.

Jill has no plans tonight, which is good because she keeps running into problems with the local maven repository.  She’s got half the team in her office trying to figure out why.  5 PM comes and goes, and the team stays on the task.  While they’re swapping stories over cold pizza at 9 PM someone spots a stupid configuration problem.  They all laugh about it, promise not to tell anyone how simple it was to solve, and go their separate ways.

Two years from now, both these developers will remember those nights.  While the end result was exactly the same in both situations, one team will bond closer through the hardship while the other will not.  One developer will think of it somewhat fondly, the other as a time when his work kept him from being where he needed to be.

Jill had the freedom to be in the moment, while Ben’s attention was elsewhere.  This isn’t Ben’s fault, or a credit to Jill.  It’s just unhappy circumstance.  How many of us have been in situations like both those?  I remember staying late into the evening trying to fix a configuration problem for a ControlCenter demo at EMC World — it was annoying and frustrating but in total the team that stayed to fix it grew closer and remembers it with a chuckle.  Why?  We were miles from home, had nowhere else to be, and had the freedom to attack the problem fully in the moment.

I’m not suggesting you hire only mid-20s kids with no families who will lose themselves in their work.  But it’s important to recognize the reasons why the same circumstances can either build a team up or tear it down.  I’m also not suggesting (by far!) that managers manufacture artificial barriers for teams to overcome together.  Software development is full of enough pitfalls that the team will stumble into these situations without your help!

So the question becomes: what can a manager, a technical leader, or a team member do to help the team operate in the moment?  How can we capitalize on those moments when they arrive?  How much does the attitude of the leaders present (and remember: we expect leadership at all levels here … directors and interns share this responsibility!) shape the way the team perceives a difficult moment?

More than I think people realize.

This post is from: Dave Talks Shop

Working in the moment

I recently had a chat with a senior developer working on a very difficult task, trying to achieve demonstrable results in a tight time frame.  I look at his workload and know he’s walking that fine line between “challenging” and “impossible.”  How did he describe its impact on his motivation, his engagement?  As the most fun he’d had at work in over 5 years.  And what was the last one?  Another crazy struggle, when a major scalability issue appeared late in a development cycle and teams were forced to come together and refactor major components with very little time to actually design and lots of pressure to actually produce.

Are software developers inherently broken, looking for painful exercises in futility?  I don’t think so.  I think if you look the tasks that push people to these heights of motivation you’ll find a few things in common:

• Well-defined success criteria
• Insufficient time to enter analysis paralysis
• Developers empowered to pursue their own ideas
• Cooperation of close-knit and competent peers

The question I can’t answer is whether people operating on those projects produce higher quality products.  It’s clear to me that they enjoy their work more, but is that enough?  Perhaps the ruthless attention to continuous integration you see at the heart of Agile/Scrum is a safety valve for that question.

This post is from: Dave Talks Shop

In praise of deadlines