Qualitative research provides the eyes, but quantitative research provides the legs of validation.
By Dr. Jodie Monger
Are your customer surveys accurate? Put another way, are you guilty of
survey malpractice by giving your company faulty information based on inadequate
research methods?
Malpractice is a hard word – it directly implies professional incompetence
through negligence, ignorance or intent. Doctors carry insurance for malpractice
in the event that a patient perceives a lack of professional competence. But,
for contact center professionals, there is no fall back for professional
incompetence, whether intentional or not.
You should never put yourself in a position where your competence can
be called into question. Many call center managers are skating on thin ice
when it comes to typical practices used to measure customer satisfaction.
By definition, an
ineffective measurement program generates errors from negligence, ignorance and/or intentional wrongdoing.
You have a fiduciary responsibility to your company – and recommendations made based on erroneous customer data
do indeed meet the definition of malpractice.
Measurement programs must meet certain scientific criteria to be statistically
valid with an acceptable confidence level and level of precision or tolerated
error. Without these considerations, you are guilty of Survey Malpractice. To
defend your program with "it has always been this way" or "we were told to
do a survey" is not sufficient. Research laws adhered to in academia apply to
the business world. A deficient survey yields inaccurate data and results in
invalid conclusions no matter who conducts it.
Before assuming that Survey Malpractice does not apply to your program,
consider the following tell-tale signs of errors and biases:
Measuring too many things.
Your survey evaluation of a 5-minute contact center
service experience takes the customer 15 minutes to complete and has 40 questions
on it. While everyone in your organization has a need for customer intelligence,
you should not be fielding only one survey to get all of the answers.
Should the contact center be measuring satisfaction with the inhome repair
service, the accounting and invoicing process, the latest marketing campaign,
or the distribution network? Not all on one survey.
Not measuring enough things.
An overall satisfaction question and a question
about agent courtesy does not a survey make. Without a robust set of measurement
constructs, answers to questions will not be found. Three or four questions will
not facilitate a change in process, won't enable effective agent coaching and
is unlikely to be a valid measure to include in an incentive or performance plan.
Measuring questions with an unreliable scale.
In school, everyone agreed on what test scores meant: a 95 was an A,
an 85 was a B and a 75 was a C. Everything
in between has its own marks associated with it as well. Yet, when it comes
to our service measurement, we tend to give customers limited responses. What
do the categories excellent, good, fair and poor really mean? Should you
translate them into numbers for a grade from the customer? Giving limited
options does not permit robust analysis and statistical analysis is generally
applied incorrectly. Using a categorical scale or a scale that is too small
(like many typical 5-point survey questions) is not adequate for the evaluation
of service delivery.
Measuring the wrong things or the right things wrong. Surveys should
not be designed to tell you what you want to hear but rather what you need
to hear. Constructs that are measured should have a purpose in the overall
plan. Each item should have a definitive plan for use within the evaluation
process. The right things to measure will focus on several overall company
measures that affect your center (or your center's value statement to the
organization), the agents and Issue/Problem Resolution.
Asking for an evaluation after memory has degraded.
When we think about time, 24-48 hours does not seem like a long time. But when you are measuring
satisfaction with your service, it is the difference between an accurate
evaluation and a flawed one. Do you remember exactly how you felt after you
called your telephone company about an issue? Could you accurately rate that
particular experience 48 hours later after other calls to the same company,
or others, have been made? Yet, you are asking your customers to do just that
with a delayed measurement and are opening the door to inaccurate reporting
and compromised decision-making for your organization. This is also an unfair
evaluation of your agents.
Accuracy and credibility of service providers and product vendors.
As with any technology or service, the user assumes responsibility for applying the
correct tool, or applying the tool correctly. You can purchase software to
schedule agents but if you do not apply the functionality correctly, you will
be responsible for the error.
There are plenty of home-grown or vendor-supplied tools to field a survey, but
again if you do not apply the functionality correctly, you will be responsible
for the error. You can locate a service provider that is interested in selling
you something, but it is usually something that fits into their cookie cutter
approach which will not be to your specific requirements.
Wiggle room via correction factors. If you are using correction factors to
account for issues in the data or to placate the agents or the management team,
some aspect of the survey design is flawed. A common adjustment is to collect 11
survey evaluations per agent and delete everyone's lowest score. With a valid
measurement that includes numeric scores as well as explanations for scores
and a rigorous quality control process, adjustments in the final scores are
not necessary. Making excuses for the results or allowing holes to be poked
diminishes/ undermines the effectiveness of the program and highlights an
opening for Survey Malpractice claims.
How to do it Right
In addition to adding important insight for management, customer score
explanations (survey comments) enable a more accurate level of individual
accountability. In research, the collection of comments is referred to as
qualitative data. This critical element is often omitted because it may be
difficult to collect, to process and to analyze. Without this feature, a
measurement program is immediately susceptible to errors that are not corrected.
The most common error when customers respond to surveys is that they reverse
the scale (scale flipping) which means they want to be giving you positive
scores, but are instead giving you negative ones by mistake. Therefore,
comments can be reviewed to ensure correct interpretation of the scale and
of the question. Proper assignment of the survey is evident as well. Survey
scoring or assignment errors account for an average of a 5-7% variation in
scores pre- and post-quality control. This is the amount of negligent exposure
from errors in the data. Controlling for a 5% score variation lessens the need
for correction factors, insures more reliability in the results upon which to
base management decisions, and allows your QA team to coach with confidence.
The bottom line for quality control is to avoid a need to defend this
performance metric as a human resource tool. By instituting stringent quality
control, all surveys pass the "is it fair?" test and then the right agents
are held accountable for the right data. Your center's Quality Assurance and
Monitoring program requires continuous calibration, rules and guidelines. In
other words, a stringent process is needed to be successful. Your survey
program is no different.
Research is a science as is the application of the results. Garbage in
equals garbage out. Dirty data is the equivalent of garbage for your measurement
program. Considering the implications to personnel management and the overall
management of the center and its ultimate value as the relationship manager for
the company, it is better to do NO surveys than to do poor surveys. Critically
evaluate your program or proposed program on the following criteria:
1. Use of the correct measurement methodology. The customer chooses the
communication channel to your organization and your survey of that experience
should match the channel. Avoid survey channel slamming by employing immediate
post-call surveys to evaluate telephonic customer service delivery and email/web
surveys to evaluate the electronic channel. Mixed methodologies are likely to
create Survey Malpractice exposure. To avoid Survey Malpractice, insure that
you are measuring the rights things through the right channel.
2. Appropriate length. Protect your response rate and survey validity by
fielding a survey that is just right, not too short and not too long. Monitor
the point at which respondents drop off to determine if the survey is too
long. Consider running more than one survey if the scope of the measurement
need grows. Measuring becomes an addiction for those who can benefit from your
continuous contact with customers. Do not attempt to measure everything in one
survey. Run additional surveys for them and do not allow excessive additions
to your Voice of the Caller survey.
3. Actionable results. Information must be more than just interesting to know
and what management wants to hear. Customer intelligence drives the organization
and should be highly sought after. If your team and company executives are
not clamoring for the information, you have room for improvement. Managing
the customer experience is continuous and must be proactive rather than
reactive. The contact center is an effective intelligence partner for the
company, beyond its own measurement needs. The value the measurement program
in the contact center brings to the organization can erase the misconception
of being a "cost" center, as long as it is done correctly.
4. Accurate results. Too much time, effort and resources are wasted chasing the
wrong things from questionable survey results. More time, effort and resources
need to go to the design of the survey and its implementation. Again, NO survey
is better than a bad survey. Action on results is expected so the direction
for action must be accurate.
5. Measurement should complement existing programs. Surveying is part of a holistic
approach. Call monitoring is your view of quality and surveying is the callers'
view of quality. It should not be one or the other, but how to use them each
to complement quality assessment with a holistic view of the caller experience.
6. Retain the customer relationship. Measurement is intended to understand and
then to re-engineer the experience to be successful for the company and for the
customer. When service interactions fail, quantification is not enough. It is
irresponsible of the management team to not have immediate triggers to serve as
a safety net for the relationship. Build your survey tool to include a request
for contact if the customer needs additional attention. This way, you will be
able to save a customer relationship that might otherwise have been lost. Delayed
measurement programs do not include the benefit of an immediate trigger.
7. Request-Resolution and Problem-Resolution must be quantified. The various
call resolution calculations such as First Contact Resolution (FCR) are most
important from the customers' perspective. Just because your team or the
agent views the call as "closed" does not mean the customer has the same
impression. Reporting and acting on any other FCR metric is negligence. A
series of questions around the resolution topic must be included in your survey.
We all know that cheap can sometimes be very expensive. Cost-only decisions
are rarely a comparison of apples-to-apples, although you intend that to be
the case. Finding the least expensive survey option does allow you to check
the measurement requirement off of your list of things to do. However, with a
low quality and value plan, the Survey Malpractice implications could easily
be more than 100 times your annual measurement budget. Biased and inaccurate
results affect the contact center's position within the organization as the
momentum of effectiveness breeding efficiency never occurs. The right dials
are not tweaked and improvement is not made and certainly cannot be quantified.
Unfair measurement used for performance evaluation causes low agent morale and
higher turnover. A lack of quality control enables scores that could be invalid
and/or assigned to the wrong agent to be included in the scoring model. And
with all of these situations in play, unhappy managers equals unhappy agents,
which in turn equals unhappy customers.
Ignorance about measurement contributes to Survey Malpractice. Design a customer
intelligence plan and secure a greater ROI from the right measurement. Use
the customer-driven quality monitoring to be less of a report card summary
and more of a diagnostic tool to achieve best-in-class. Agents respond to fair
and equitable feedback and productive coaching and this translates to happier
customers. Educate and leverage all parts of the company about your measurement
program and help them to achieve their goals. If the data are sound and analyzed
properly, the results will be sound and therefore defendable. This leads to
a greater ROI, but more importantly to greater value – to your organization,
to your employees, and most importantly, to your customers.
