Customer Feedback Surveys Considered Harmful

 

Customer satisfaction surveys are hot. Every time you go to the bank, open a support ticket with an online vendor, go on a trip or hire a freelancer for any task, you will subsequently be dogged relentlessly with email requests to rate your experience.

On the surface, this is good idea. Given the near ubiquity of the internet and email combined with the low cost of sending these emails and the valuable insight your firm can garner from your customers, what’s not to love?

The shop/hire -> rate -> reward feedback loop has become baked-in to some systems. Many live marketplaces incorporate these feedback transactions into ratings, which then become a score which then impacts future prospects of whomever is being rated.

And that’s where the trouble starts.

There is a point where this stops being useful and the knock-on effects of a ratings system predicated on feedback results becomes counter-productive. That point is when the ratings become targets.

When a company decrees “All customer feedback ratings must score a minimum of X, or else…” the company has just commenced the process of invalidating and corrupting all useful information to be gleaned from that feedback/survey process.

A label which captures this concept is “Goodhart’s Law” – after economist Charles Goodhart, who posited in essence that “when a measure becomes a target, it becomes useless”. He meant it in an economic context in his critique of Margaret Thatcher’s attempt to stimulate the economy by using measurements to set targets in monetary policy (sound familiar? That’s what the “data driven” Federal Reserve has been doing for nearly the last 10 years.)   It’s what super-investor Howard Marks described in detail in the  “Second Order thinking” chapter of his book “The Most Important Thing”.

Goodhart’s Law  generally comes under the umbrella of “unintended consequences”:

“The most famous examples of Goodhart’s law should be the soviet factories which when given targets on the basis of numbers of nails produced many tiny useless nails and when given targets on basis of weight produced a few giant nails. Numbers and weight both correlated well in a pre-central plan scenario. After they are made targets (in different times and periods), they lose that value.”

This anecdote from an examination of the phenomenon wherein that author sums up Goodhart’s Law as “People and instituitions try to achieve their explicitly stated targets in the easiest way possible, often obeying the letter of the law. “ and gives a great example of the outcome of this in his related work “Guessing the Teacher’s Password”:

Suppose the teacher presents you with a confusing problem involving a metal plate next to a radiator; the far side feels warmer than the side next to the radiator.  The teacher asks “Why?”  If you say “I don’t know”, you have no chance of getting a gold star—it won’t even count as class participation.  But, during the current semester, this teacher has used the phrases “because of heat convection”, “because of heat conduction”, and “because of radiant heat”.  One of these is probably what the teacher wants.  You say, “Eh, maybe because of heat conduction?”
This is not a hypothesis about the metal plate.

This is not even a proper belief.  It is an attempt to guess the teacher’s password.”

And this is what happens, en masse, once surveys are condensed into ratings which are then set as targets.

When this plays out in feedback and ratings systems the same dynamic is at work. Rather than the data conveying signalling information that can be used to refine the customer experience, they become targets in themselves and in doing so they cease their signalling function. Participants regear their efforts toward optimizing the actual ratings responses instead of the underlying actions which are supposed be measured…

“Did I satisfactorily address the reason you called today Mr… Jeslovich?”

“Yes, thank you”

“Then you’ll score me a 10 out of 10 if The Corporation should select you for our customer satisfaction survey then?”

“Uh, ok I guess so”.

I don’t know about you, but I am always fidgety through that exchange. If not visibly then mentally. It takes the entire exchange into a begging/pleading power construct which if it makes me uncomfortable, has to be at least subconsciously, repeatedly humiliating for the person who’s livelihood depends on my scoring them a 10/10 in some b/s survey. It changes the nature and the timbre of the exchange, bringing it out of a useful realm where I would feel comfortable enough to relate useful information back to The Corporation.

Example: I rated my customer experience as an 8 out of 10 because…..

• The robot said my hold time would be 3 minutes and it was 7 and a half
• The agent was courteous but unfamiliar with my product’s firmware version

The above feedback conveys useful feedback.

The reality:

“So is everything ok now Mr. Jeftovic?”

“Yes, thanks”.

“Ok great. Listen, when the survey pops I need you to score me a 10 out of 10 or it’s my ass, ok?”

“Sure, no problem”.

Now I rate my customer experience thusly:

10/10.

Amount of useful information communicated back to The Corporation: nil.

What happens if I’m genuinely, ideologically committed to honestly scoring my customer experience?

I’ve encountered full-on harassment, whining, various shades of drama and hysteria when I’ve done that in an open market live marketplace context. I’ve had such horrible experiences in live marketplaces from leaving genuinely considered feedback that I routinely write in my job specs that “Any attempt to coach my feedback rating will result in a lower score”. It actually drives down the number of bids I get on any particular job but conversely, I find I’m hiring higher calibre people.

God forbid what would happen if I left honest scoring in a corporate feedback scenario.

In fact this week I had cause to rate a technician whom I felt was just goddamn outstanding. He blew my mind. He worked for a big clueless corporation and he was over the top on my problem. He stayed late  (and didn’t get paid to) until my job was done, and fixed.

He was so good that I even made a point to call in to The Corporation to make sure his supervisor was notified that this technician was fabulous. Then the online survey came in so I did what any normal person who was totally blown away a technician who went above and beyond to fix a situation The Corporation had kind of messed up in the first place and I rated my customer experience 9  out of 10.

That’s good right? No. Actually I found out later 9/10 is the required rating these technicians need to score on a customer feedback survey to not get in shit. It’s the bare minimum. This guy was new on the job, so he never told me that. He wasn’t yet inculcated with the routine to implore me “when the survey comes in you need to rate me 10/10, ok?”.

The (non)logic that goes into a company creating a customer feedback ratings system wherein the minimum score required in order for the agent to “pass” is 9 out of 10 is ridiculous. There is no spectrum for useful information. It’s Lake Wobegon logic  “where all the women are strong, all the men are good looking, and all the children are above average.”

Circling back to Goodhart’s Law. If you want honest feedback from your customers, stuff you can actually use to refine your business processes and improve things, you need to decouple those feedback results from the employee scoring.

Your employees will serve the customer more diligently if they aren’t worried about extracting that 10 out of 10 from them on the ensuing survey.

Your customers will provide more honest, accurate, useful information back to you if they aren’t worried that expressing their opinion will result in undesirable reprisals against the unfortunate employee.

That may seem counter-intuitive but tying employee’s scores, or even worse, their compensation to these feedback mechanisms is akin to the old joke “The beatings will continue until morale improves”.

You won’t get anything valuable from those surveys other than a loose pecking order of who is the best at cajoling your customers into leaving manipulated and worthless feedback. In that sense it may be useful for identifying The Corporation’s future CEO but beyond that it’s noise.