Bad stats at stats firms

5 min read

Convergys Corporation does data analytics to identify operating cost reduction opportunities for its call center clients.  They ask prospective analysts to do a take-home analysis of 120 rows of call center data.  I’m 20 years past being an analyst, but a networking acquaintance encouraged me to apply as she believed her firm would love to have a quantitative expert on staff.  And I would have been extraordinarily happy to have the work.

I asked this acquaintance if I should dumb myself down, but she assured me that this firm is an ego-less meritocracy for which there’s no need to self-censor.  So, I made a 10-page PowerPoint with data visualizations so slick that I’ve since used them as samples of great data visualizations.  Storyboard structure, trend summary (by agent, agent tenure, day of week, over time, by topic, etc), hypothesized causal relationships to investigate outside the data, Tufte-style clean design, and concrete next steps to pitch a meaty engagement to the fictive prospective client.  I hit this thing so far out of the park it landed in the next stadium.

In addition, I explained to the reader that all inferences are preliminary, because the low sample size translates into large margins of error, even at an aggressively low confidence level.  In fact, there were no statistically significant differences in the data Convergys provided.  None.  All of the blips in those 120 rows of data have a strong chance of just being random noise.  The margin of error is bigger than the effect size we’re trying to identify.

This is a situation where they either haul you in to fix the whole department or reject you in terror at realizing they don’t understand their own career domain.  In Convergys’ case, I never heard back – not even a “thanks-but-no-thanks”.  And, – perhaps needless to say – that acquaintance never returned email or call again.  

Twenty-ish years ago, after I got the figurative Golden Ticket of a premier Wall Street internship, I discovered a calculation error in the Goldman Sachs M&A Group’s Excel valuation model.  This was the model that the most elite department in the most elite investment bank used to value every multi-billion-dollar, front-page-WSJ transaction.  They were proud of it, and asked us 8 carefully-selected interns from the top business school programs to check the model in our spare time – issuing a challenge they believed was ironclad.   I was the only one who found something.  I wrote a gentle email explaining the issue – and in obsequiously non-threatening, self-deprecating, know-one’s-place-in-the-world female fashion, gushed that this is not a material error in almost any case but an error nonetheless, relayed to them respectfully in direct response to their own request.

Needless to say, I wasn’t popular.  My male boss at Goldman Sachs told me to “go shopping” since there were no deals available for me to be staffed on.  

That same summer in Manhattan, I got a glass of lemonade with a floating piece of lemon that still had its sticker attached.  Meaning they didn’t wash the lemons to make the lemonade, and so not likely the fruit for the fruit salad either.  Arguably, since produce stickers are food-grade and edible, it’s not a material error.  But, it tells you about the level of care and competence in the kitchen, and makes you wonder what more egregious issues you didn’t catch.  And, finding paper in a mouthful of food is gross.

But, – again, perhaps needless to say – the waiter was pissed at the request for new lemonade.  

The next year, I visited my new fiancee’s parents’ house for the holidays.  He had been a Harvard math whiz.  He thought it would be fun to get out one of his old statewide math competition tests and have the two of us compete to do the test.  I got the same score as he did – despite the fact that he was a math major (I was literature), he went to Harvard (I to one of the “lesser” Ivies), and he had done the test in high school (it was sight unseen to me).

In a flash of rage at the outcome, he threw his pencil across the room.  (Needless to say?)

Once I did get hired by a consulting firm explicitly to revamp their numeric capabilities – after I pointed out that their Fortune 50 pharmaceutical client was making global business decisions on data “trends” indistinguishable from random noise.  Neither the consultancy nor their clients knew what a margin of error was or how to calculate one.  I brought in several new tools to improve decision quality and lower decision analysis costs: Bayesian methods, lower confidence levels, better survey design, reliably automated calculations, hypothesis testing to replace data mining, and framing research questions within business strategy. 

My boss was embarrassed, needless to say.  But he was eager enough for a competitive edge to stoically ride through his discomfort.  It was my own loyalty that faltered in this case:  when I found him cooking the books to his subcontractors, I blew the whistle.

But, then there was the time, at age 26, I explained Monte Carlo simulation to an insecure, hot-headed, community college-educated boss, and he punched a wall in frustration that he couldn’t follow me.  The CEO sought me out directly, bringing little ol’ me to the Board meeting to explain it directly to the Board.  (And he arranged a fat bonus for me, and he gave me a promotion.)  Impressive leaders don’t feel they need to know more than their quant analysts.  But, those leaders are the rare exceptions.  In twenty-some years since, I’ve never again met someone so unthreatened by a smart employee.  (I would have eagerly defined my career by following that CEO, but a few months later he died of cardiac arrest while jogging.)

The story at hand — of not getting a callback from Convergys — has many themes:  It’s a story about the consequences of being overqualified for a position and of being smarter than the boss, a story of a would-be academic mind forced to grovel for acceptance in the daft corporate world, a story about the psychological barriers to innovation and learning, a story about humans projecting destructive competition onto situations that should be about productive collaboration, and a story most concretely about innumeracy among self-proclaimed numerati.

Falling over oneself to apologize for one’s own competence isn’t usually enough to mitigate other people’s terror at confronting their own incompetence.  However, hiding your competence won’t get you a good job, a promotion, or a satisfying career either.  The solution, as with most issues in life, is finding a strategic middle ground.

The unfortunate reality is that many firms using and/or selling data analytics mess up basic statistics.  And yet the world keeps turning.

  |   Email Email

This entry was posted in <5 min read, Business topics, Math is everywhere!, [All posts]. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *