$20 million error: Case study in tech product development (Part 1 of 3)

15 min read

Part 1: Problem (February 2015)

Unwelcome discovery

As of February 2015, CableCo projects its next-generation WiFi router product launch for “Q3 2015”.  The multi-billion-dollar company expects to be first to market, beating out all other internet service providers (ISPs) in the United States. 

CableCo holds a regulatory monopoly on coaxial cable internet service within its geographic territory, but competes intensely against firms offering broadband internet service via DSL, satellite, and fiber optic cable.  CableCo’s marketing message is that it offers the fastest WiFi in its service areas – a claim that would become legally untenable if any competitor were to launch a faster router before CableCo does.

CableCo’s young WiFi Product Manager is leaving for a 4-month sabbatical.  I’m a strategy consulting Principal with 14 years post-MBA experience who has been retained to fill in.  As a hired gun, I’m paid hourly by a staffing firm, which in turn charges CableCo a fixed daily rate.  In sharp contrast to my experience with other large corporate clients, CableCo’s culture disdains management consultants, whom they conflate with “contractors” (i.e., temp staff augmentation). 

In my first week on the job, I discover that the earliest plausible next-gen router launch date is not Q3 2015, but Q4 2016.  CableCo’s hard target reflects a 15-month error by my predecessor.  Consequently, CableCo will not be first to market. 

Moreover, it is evident that CableCo would have lost its “fastest-WiFi” marketing claim even if the imagined launch date had been attainable.  CableCo’s direct US competitors are poised to launch next-gen routers in Q3 2015 – using a well-known component source unfamiliar to CableCo’s Product Department.  

This unwelcome discovery transforms me into the bearer of bad news to a resistant audience.  With diplomacy and constructive problem-solving, I will steer the company on a corrective course – but not without the messenger being proverbially shot.

WiFi technology background

In February 2015, the next generation of WiFi technology is the “Wave 2” implementation of the 802.11ac networking standard.  New features promise a major performance leap over the current generation:

  • More antennas. “Four by four” or “4×4:4” devices have 4 radio chains (4 transmitting + 4 receiving antennas) and thus up to 4 spatial streams.  Available Wave 1 wireless chips support a 3×3:3 configuration for the 5GHz frequency band.  Wave 2 chips support 4×4:4 or 8×8:8.  More antennas = more spatial streams = more speed.
  • MU-MIMO. The Multi-User Multiple-Input Multiple-Output algorithm improves network utilization (and therefore speed) when multiple high-bandwidth users share a single WiFi access point (AP).  Several kids can successfully stream different videos in separate bedrooms.  Current-gen devices use MIMO (multiple antennas sharing same channel), but without the impactful new multi-user aspect.  MU-MIMO’s rate gains are dramatic at close range, and drop off further from the AP. 
  • Broader broadband. 5GHz-frequency WiFi signals are broadcast on 20MHz-wide channels.  11n devices can transmit over 2 bonded channels (40MHz channel width), and 11ac Wave 1 devices can use 4 channels.  Wave 2 devices can bond 8 20MHz channels together (i.e., 160MHz channel width).  This improves speed for high-end client devices in low-density environments that have little WiFi signal congestion or aviation radar activity.  Because 160MHz-wide channels necessarily overlap spectrum used by aviation, devices must meet FCC Dynamic Frequency Selection (DFS) requirements.  As with MU-MIMO, rate gains from the 160MHz feature are best for client devices at close range.

The Federal Communications Commission (FCC) has recently increased the maximum indoor signal transmission wattage on U-NII-1 spectrum channels.  When implemented in existing device firmware, this will increase signal range, and the Received Signal Strength Indicator (RSSI) for connected client devices will increase.

WiFi technology generations follow an 18- to 24-month cycle:

  • 2009: 11n wireless chip samples first shipped to device makers (“prior-gen”)
  • 2011: 11ac chip samples (“current-gen”)
  • 2013-2015: 802.11ac Wave 2 chip samples (“next-gen”)
  • 2016 (projected): 11ax chip samples (“next next-gen”)

Three semiconductor companies manufacture the wireless and network processing chipsets used in WiFi routers.  CableCo maintains relationships with Qualcomm and Broadcom.  However, nobody from CableCo has ever spoken to Quantenna.  At my request, the Legal Department executes a Non-Disclosure Agreement, enabling me to meet with Quantenna on behalf of CableCo.

Semiconductor manufacturers have historically traded off being first to market with successive generations of technology.  As of February 2015, Quantenna is the technology leader for the 11ac Wave 2 WiFi generation:

  • Quantenna
    • 4×4: September 2013 delivery of chip samples to device makers.  Summer 2015 (estimated) MU-MIMO software.  160MHz not available.
    • 8×8: Early 2016 (est) chip samples, including MU-MIMO and 160MHz software.
  • Qualcomm
    • 4×4 v1: November 2014 chip samples, including MU-MIMO software. 160MHz not available.
    • 4×4 v2: Summer 2015 (est) chip samples, including MU-MIMO and 160MHz software.
  • Broadcom
    • 4×4: January 2015 chip samples.  Summer 2015 (est) MU-MIMO software.  Fall 2015 (est) 160MHz software.

New WiFi technology is introduced first to the retail market, and later incorporated into ISP devices when it’s more mature and thus less likely to provoke costly customer service calls. 

Back in July 2014, a device maker had launched the first Wave 2 retail device.  Just after the CableCo engagement begins, Bell Canada becomes the first ISP to offer a Wave 2 device (March 2015).  Both devices use Quantenna Wave 2 wireless chips.  My conversations with vendors reveal that – unbeknownst to CableCo — four US-based ISPs already have Wave 2 devices in testing.

Company situation

CableCo offers new residential internet customers the choice between (a) leasing a CableCo modem and CableCo standalone WiFi router, or (b) providing their own retail-purchased networking equipment.  For strategic and monetary reasons, CableCo prefers customers choose CableCo equipment.  About 40% of new internet customers do so.  A next-gen router that keeps pace with the timing of retail device innovation will maintain that adoption rate.

CableCo launched its in-market 802.11ac Wave 1 WiFi router 10 months ago.  It’s proud to have been the first to market with that generation of technology, beating out the much larger second mover by 5 months.  The router is a sleek, rectangular, black box, branded by the device manufacturer.  It operates with 3 radio chains for the 5GHz frequency band and 3 for the 2.4GHz band (3×3|3×3).  Commercialization took 8 months, from testing the certification-ready prototype to field launch. 

This product superseded CableCo’s prior-generation 802.11n 2×2|2×2 router, and is now being deployed to all new customers (i.e., people moving into a new residence + people switching from DSL) and to existing customers reporting technical issues with older devices.  The Wave 1 router is a big success, as measured by a record low volume of customer calls to technical support.  CableCo hopes to enjoy similar success with its Wave 2 router. 

CableCo’s commercial and residential divisions have decided to consolidate customer-premises equipment (CPE) with the Wave 2 generation.  The in-market residential WiFi router doesn’t include the static IP capability demanded by a third of commercial customers, whereas the Wave 2 router will.  Because the in-market commercial WiFi device is causing headaches, the commercial team is particularly eager for the promised Q3 2015 Wave 2 launch and would be harmed by any delay.

Implementing public WiFi is also a priority.  Customer routers will broadcast a separate signal for shared community use.  CableCo customers moving through a dense urban area can thus experience continuous WiFi coverage.  WiFi phone calls become practical.

CableCo does not have visibility to customers’ WiFi signal strength, channel utilization or connected client device capabilities.  Customer Service sometimes appeases signal range complaints by sending customers a second WiFi router – though this is known to potentially create more problems (signal interference) than it solves (signal range extension), and is an expensive practice.  Once CableCo implements the Broadband Forum’s newest data model standard in its router software, it will have remote access to CPE performance data.

Corporate Product and Engineering teams famously clash across a chasm of technical expertise and market knowledge.  In CableCo’s case, the Product-Engineering divide is particularly extreme.  The WiFi Product Manager going on sabbatical self-identifies as “not at all technical”.  I track down basic information about her product that she’s unable to provide:  How big is the WiFi business I’ll be managing for her? [900k residential customers, $80m annual gross revenue]  How much bigger will it be if the pending merger closes? [60% more customers, 8x more unique device models on the CableCo network]  What WiFi technology is on CableCo’s network now?  [60% prior-gen / 40% current-gen]  How many Wave 2 units will be deployed over its life, before CableCo introduces its successor device? [~1.5 million].    

The Product Manager is plainly overwhelmed, and has been managing the product passively and reactively.  Once word circulates that a replacement’s on board, people from Engineering, Supply Chain, Customer Service, and Network Operations approach me with unaddressed urgent and important matters: “We’re glad to have someone in Product who will actually respond.”

Barriers to innovation

Consumer technology companies are primarily market-driven.  Product strategists define the experience a product should deliver, in what timeframe, and at what target cost.  Engineering figures out the technical path to deliver that performance, within time and cost targets.  It is Product’s job to resolve thorny feature-cost-timing tradeoffs based on market intelligence and strategic business analysis.

At CableCo, the Product Department leadership vacuum has invited idealistic engineers to treat Wave 2 router time-to-market and cost as dependent variables.  Six months before her promised WiFi product launch date, the WiFi Product Manager cannot provide me with a copy of the Product Requirements Document (PRD).  The Engineering Department’s 50-page draft PRD is a catch-all wishlist, describing an over-featured and expensive router.  Instead of market-optimal launch timing informing device feature selection, desired device features are dictating launch timing.

CableCo doesn’t have a WiFi product strategy.  Internet and WiFi service accounts for about one third of CableCo’s revenue, while video service accounts for half.  But, data services are higher margin and faster-growing than video.  CableCo isn’t the only multiple system operator whose historically preferential focus on its video division no longer matches its business trajectory.

Without a strategic plan, defined target market, or business unit financial objective, Product lacks a framework to resolve feature-cost-timing tradeoffs:

  • Feature vs cost: Is a 4th radio chain at 2.4GHz worth $5 in wholesale per-unit device cost?  Is a second 5 GHz band to separate client devices worth $15? 
  • Cost vs time: Is possible price negotiation leverage gained by an open-field RFP process worth probable months of launch delay?
  • Time vs feature: Is DFS spectrum operation (necessary to enable 160MHz channel width) worth a 2-month launch delay for mandatory FCC certification?

Similarly, the relevance of innovative smart speaker/hub devices from Amazon and Google is unclear when not considered within an overall business strategy.  The Product SVP casually asks the User Experience (UX) team to re-design the Wave 2 router custom casing from a rectangle to a cylinder (not yet realizing the 15-month launch date error, which this will exacerbate).  Without seeking input from Engineering, UX produces a high-style design that hardware vendors point out is thermally unfeasible.  Marketing isn’t consulted either, and wonders which customer segment cares about router aesthetics enough to pay substantially more.

CableCo’s hierarchical communication norms inhibit employees from speaking up.  Mistrust between departments abounds.  Product won’t approve funding for the equipment Engineering needs to certify Product’s devices.  Supply Chain is required to use Marketing’s sales volume forecast, but is held solely accountable when actual need outstrips product inventory.  Procurement summarily vetoes sourcing from a hardware supplier favored by Engineering.  Video Product hasn’t apprised WiFi Product that it expects near-term video-WiFi hardware convergence.  Procurement distributes an over-broad intellectual property agreement not reviewed by Legal; when device makers refuse to sign it, Procurement removes them from the candidate vendor list.  When a senior executive requests evaluation of an obscure device brand mentioned to him socially, Engineering dutifully interrupts its already-backlogged testing schedule rather than push back. 

Hardware vendors confide privately that they long knew CableCo’s expectation of a 2015 Wave 2 launch was impossible.  But, they haven’t spoken up, for fear of CableCo shooting the messenger.  

Product development process

Once product specifications are finalized, commercializing a new router takes approximately 9 ½ (best case) to 19 months (realistic case), or longer: 

  1. [1 month] Device maker designs circuit board layout and mocks up product casing, based on CableCo’s finalized specs in the PRD. Any substantive future changes to product specs or casing design will re-set the development clock back to this point.
  2. [1½ – 3 months] Device maker receives chip samples from chip maker, assembles product prototype, optimizes antenna configuration. Timeframe depends on how new-to-market the chip is.  If CableCo imposes a shorter testing period on device makers, CableCo’s own testing period will be commensurately longer.
  3. [4 – 12 months] CableCo engineers test and certify product prototypes from multiple device makers, iterating with chip and device makers to fix software bugs. Timeframe depends on how young and buggy the silicon is, how many prototype devices must be evaluated parallel, and how motivated device makers are to turn around bug fixes quickly. 
  4. [1 month] CableCo executives approve product and negotiate price with device maker(s). Market split negotiations and internal approvals would take longer with multi-sourcing.
  5. Preparation for launch
    1. [2-6 months] Device maker orders bulk chips, creates tooling for product casing, ships product. Timeframe depends on whether CableCo’s RFP process supports device maker justification of risk buy of chips and investment in third-party certifications prior to CableCo purchase order.
    2. [2 months] CableCo makes device labels and manuals, trains customer service and field tech team, updates website, updates billing and other IT systems, creates advertising

ISPs commonly pre-qualify a shortlist of hardware vendors before requesting prototype device submissions for testing.  However, for the Wave 2 router, CableCo’s Procurement Department envisions an open-field process involving over a dozen vendors.  And, unlike CableCo’s current-gen router, Wave 2 will have a custom feature set and a custom-designed casing.  Thus, candidate vendors are expected to invest large sums in speculative custom development — in exchange for a low probability of winning a supply contract.  Moreover, given CableCo’s announced plan to multi-source the Wave 2 router, a candidate vendor can only hope to be awarded some fraction of CableCo’s projected device purchase volume.  And, in the eyes of the largest device makers, that projected volume is relatively small with relatively low profit margin.

Revised launch date Q&A

Given CableCo’s current processes, Wave 2 commercialization is projected to last 19 months.  The clock starts when the PRD (including custom casing design) is finalized.  If the PRD is issued to vendors by May 2015, the projected launch date would be late Q4 2016.

Why expect a 19 month process, when the Wave 1 process was only 8 months? 

  • Wave 1 launched 8 months after testing began — not after specs were finalized. Because CableCo didn’t request custom device features, vendors could immediately submit their off-the-shelf devices for testing. 
  • Wave 1 didn’t have custom casing.
  • Engineering only had to fully test one Wave 1 device. The Wave 2 process envisions Engineering testing numerous devices in parallel.
  • Wave 1 chips were deployed in retail devices 1 ½ years before CableCo tested one. Therefore, most of the bugs intrinsic to “young silicon” were eliminated before CableCo began its process.  Wave 2 chips will have been deployed in retail devices 1 month before CableCo begins testing them.

What could make the launch date slip even further, beyond Q4 2016?   

  • Technology to test MU-MIMO and 160MHz features doesn’t exist yet. CableCo’s current testing policy will suspend the development process to wait.  
  • Timeframe for device interoperability testing is controlled by CableCo’s third-party auto-configuration server (ACS) vendor, which is historically slow-moving. Because that vendor knows it will likely be replaced in the pending CableCo merger, it may be less cooperative. 
  • The custom casing design is in flux. The development clock can’t start until casing design is final.     
  • When Procurement decided to multi-source the Wave 2 router, Product didn’t consider that would result in multiple Graphic User Interfaces (GUIs) across devices. Customer Service has vetoed this as unfeasible.  A custom GUI must now be specified in the PRD.
  • Because device makers know they are part of a large field of potential suppliers, they may not move as quickly with iterative software bug fixes as the Wave 1 vendor did.

How did 4 other ISPs get so far ahead of CableCo?  

  • Given 18-month technology generation cycles and a 9-19 month development period, CableCo needed to specify its Wave 2 device immediately after launching Wave 1. Nearly one year later, it hasn’t done so.
  • The inexperienced WiFi Product Manager has had limited oversight. The Product VP who presided over Wave 1 became focused elsewhere and left.  The position was vacant for a while, and the new Product VP hasn’t asked questions yet.  The Product SVP sits down the hall from the Product Manager, but hierarchical culture limits engagement.
  • CableCo misunderstood the semiconductor market, by being unaware of a third chip maker – who happens to be the technology leader for the Wave 2 chip generation.
  • CableCo’s product development process doesn’t support product agility. In addition to starting earlier, competitors have developed Wave 2 devices in less total time than CableCo can. 

Where did the implausible “Q3 2015” launch date come from?  

The date originated from the Product Manager, who doesn’t recall how she arrived at it.  There is no Gantt chart, timeline, or project plan.  Nobody added up the development process steps.  And, nobody asked whether anybody had added up the steps. 

The high cost of being late to market

Speed to market is high on the list of best practices in technology strategy.  Launch delays are notoriously costly – and most costly for consumer products and fast-moving technology areas where time between technology generations is short.  On the other hand, CableCo’s quasi-monopoly status somewhat attenuates the tremendous damage usually ascribed to a year-long lag behind the category technological leader. 

  1. Lost revenue
    1. Lost market share. Competitors with newer technology steal market share.  During the launch delay time period, many would-be new customers and high-churn-propensity existing customers are siphoned off.  Depending on stickiness of the product/service, they don’t come back.  In CableCo’s case, closing a merger when its WiFi offering is one technology generation behind the market could greatly impact customer retention.  Additionally, a Wave 2 launch delay puts the commercial business at risk of not having any WiFi device to deploy to a third of new customers.
    2. Shorter product life. Fewer total units can be sold before the next next-gen product must be launched.  Customer expectations evolve inexorably.  When a product is late to market, its in-market window of time shortens, because the subsequent upgrade’s timing stays fixed.  CableCo will have spent a longer-than-expected period of time developing a next-gen product that it will deploy for a shorter-than-expected period of time.  (If margin on the old and new products is similar, the meaningful impact is on development efficiency, not profit.) 
  2. Higher expenses
    1. Development costs. Development costs are higher simply because the whole process took longer.  Employees aren’t allocated to other important tasks, or new resources are hired for those other tasks. 
    2. Marketing costs. For any late entrant that had expected to be an early entrant, the additional effort of adjusting an existing/planned marketing message drives up marketing costs.  In CableCo’s case, market evolution may have soon necessitated a revised messaging angle.
    3. Life extension of current product. Continuing to manufacture the last-gen product beyond what was planned can be costly.  In CableCo’s case, its hardware supplier has already discontinued the last-gen router model.  For router supply during the unanticipated year-long delay until the eventual next-gen router launch, CableCo now must identify an alternate vendor of the last-gen model – and put that new hardware through lengthy pre-market QA testing.  Because CableCo’s desperate position limits its price negotiation leverage with a replacement vendor, WiFi business operating margin could suffer.  In addition, adding another vendor’s model to the installed fleet of router models means substantial complexity (i.e., added cost) for the Customer Service organization.

Time to market isn’t the only objective function.  Technology product developers must balance competing ideals of quicker time to market, lower cost, and better product performance.  However, in fast-moving technology markets, the perfect is often the enemy of the good.

Considering sources of lost revenue plus higher expenses, this product development error may cost CableCo $20-$30 million in gross profit.

How would you address the problem?   Read my solution (Part 2)

  |   Email Email

This entry was posted in >10 min read, Business topics, Telecom, [All posts]. Bookmark the permalink.

One Response to $20 million error: Case study in tech product development (Part 1 of 3)

  1. Pingback: $20 million error: Case study in tech product development (Part 2 of 3) | The Idea Factory OnlineThe Idea Factory Online

Leave a Reply

Your email address will not be published. Required fields are marked *