Shifting Paradigms and Disruptive Technology

Why on earth would technologists wish to claim that their favourite innovation is a "disruptive technology", let alone a "paradigm shift"?

Disruptive technologies are like awkward adolescents - rude, unreliable, difficult. As Tim O'Reilly pointed out several years ago

"Disruptive technologies are often not "better" when they start out -- in fact, they are often worse. Case in point: the PC. It wasn't better than the mainframe or minicomputer. It was a toy. Similarly, the WWW was far less capable than proprietary CD-ROM hypertext systems, and far less capable than desktop apps. And developers of both derided it as slow, ungainly, and ineffective. This is a typical response to disruptive technologies." [Lunchtime Keynote at the Apple Worldwide Developer Conference, May 8, 2002]

More examples from the IT world: relational databases, object orientation, anything by Christopher Alexander.

Paradigm shifts are even worse. In science, the new paradigm is supposed to be incommensurable with the old one. As Paul Feyerabend pointed out, even the basic concepts have to be reinterpreted to fit the new paradigm. So if something is easy to understand and quick to adopt, then it probably isn't a paradigm shift. [For a quick introduction, see Steven Shaviro on Against Method.] Bruno Latour's sociological take on science and technology is probably more relevant than Kuhn/Popper/Feyerabend/Lakatos these days, but even his account doesn't exactly encourage us to overuse these terms.

I guess we aren't supposed to take these terms literally. Randall C. Willis, the Executive Editor of Drug Discovery News, reckons the presence of these two terms is a dead giveaway for unfounded hype. His article Hopped up on Hype (October 2005) presently ranks top in an internet search for "paradigm shift disruptive technology".

Presently coming second, somewhat to my surprise, is the piece that made me put aside my other work to write something here - a blogpost called Event Servers, A Disruptive Technology by one Perren Walker of Oracle, extolling the virtues of Event Servers in general and Oracle's Event Server in particular. I arrived at this piece via Opher Etzion of IBM, who responded with a piece called On Event Processing as a paradigm shift.

Opher make the paradigm shift sounds like an exercise in navigating through some topological space - avoiding barriers and finding new avenues. I certainly agree that this is a good source of metaphors for change management, including technology change management.

But why call it a paradigm shift? What's so cool about paradigm shifts, and why are vendors boasting of the disruptive qualities of their products?

Technological Perfecta 2

Following my post on Technological Perfecta on my SOA blog, Tim Bass and Opher Etzion have continued the discussion on Orthogonal Blogging at the SOA Horse Races. Tim writes:

"End users rarely build “SOAs” “EDAs” or CEPs”. End users have IT budgets to solve business problems with the most cost effective technology they can find; and they do not care (if they have a clue) what cute three letter acronyms have been created by analysts to describe momentum in the software market."

Tim's argument here is not about SOA or any other TLA (three letter acronym), but about software innovation in general.

But is anybody asking end-users to build SOAs? To my mind, it is the vendors that are building SOA; they are hoping that their customers will use SOA and related technologies to solve business problems.

But why would anyone want to use new technologies, if the old technologies were adequate? Underlying Tim's post are some fairly fundamental challenges about software innovation.

1. Why should anyone innovate? In particular, why should any individual or organization adopt new software technologies or otherwise change the way they use software to solve business problems?

2. Who should innovate? Should the adoption of new software technologies be visible to developers and end-users, or should it managed by a specialist team of technical architects (either in the organization or in some external supplier) who make sure that everything is transparently efficient and effective for everyone else.

3. When should people innovate? Does it make sense to get in early, or should people emulate the Japanese businessman quoted by Tim: "What we care about are mature technologies with solid reference clients and proven implementations."

4. How should people innovate? Small incremental steps, one technology at a time? Or sweeping changes, adopting an entirely new development paradigm in a single leap? How many simultaneous innovations can an organization cope with, and can (should) this capacity for change be increased?

5. And finally, given that there are so many new technologies to choose from: Which innovations, in which combinations?

To my mind, a strategy for software innovation is not about choosing specific software products, or even classes of software product - it is about managing these five critical questions over time. Specific TLAs will come and go, and software industry analysts will try to create meaningful maps of a complicated and constantly shifting TLA landscape, but there will always be a need for innovation. Won't there?

Five Steps Forward

A recent article by Sean McGrath Five steps forward, four steps back, (ITworld.com. October 2004) proposes a kind of innovation triage. He argues that around 80% of any technological leap forward will later be discarded, leaving 20% to become part of the established State-Of-The-Art (SOTA).

This seems to be a great argument for the late adopters. Why invest in the great leap forward? This looks like a pretty poor ROI, with high risk. Why not wait until other people have wasted energy and resources separating the wheat from the chaff? Let us know when everything has sorted itself out, and maybe then we'll start to consider it.

The early adopters have several possible responses to this argument.

1. Just because some of the stuff fails to achieve widespread adoption doesn't mean it's of no value. We shall be getting technical learning, business benefits and competitive advantage - while you're still reading the brochures.
   
2. The timing of the consolidation is uncertain. By being in at the start, we shall be hot and ready when the consolidation occurs.
3. Early participants get to influence the outcomes. So there.
   

If the McGrath hypothesis is true, it raises interesting questions about timing and dynamics. How long does it take for a large set of technological novelties to settle into a smaller set of technological givens (or "black boxes")? And how does this interact with the broader dynamics of technological development and adoption?

One Small Step? One Giant Leap?

By the way, if you can have 20% of a change, it cannot be a paradigm shift. A true paradigm shift is a discrete and discontinuous event, which cannot be subdivided. Stepwise technological change is incremental, not paradigmatic.