People have short term memory (or selective memory), when
they can’t remember things they will resort to how they think something should
be. Recently Facebook was in the hot seat because of this very reason.
viewed if it was watched more than 3 seconds by the viewer. In other words, if someone watches a video
for 2 seconds then that video view won’t be counted as a view in this metric.
Video views”, the “standard” definition of it should be Total Time spent watching
video divided by Total Viewers. However, that’s not how Facebook defined it. In Sept Wall Street Journal reported that Facebook “vastly overestimated
average viewing time for video ads on its platform for two years.” This lead to an apology from Facebook
it rolled out this metric two years ago. So it was not actually doing anything
wrong. It was a case of short term memory
critics put it, is a problem of omission. While Facebook very clearly states
that it’s only counting views as any video-play event that lasts longer than
three seconds, it does not go out of its way to explicitly beat readers over
the head with the fact that this definition of a “video view” applies
equally to the calculation of average duration of video views.”
- Make sure everybody understands clearly how metrics are defined. I talked about this in my post in 2012, Standard Definitions of Metrics: Creating a Culture of Analytics. In that post I said
Lack of standard definitions for the metrics causes people to report different numbers for supposedly same metrics, leading to confusion and total lack of trust in data. No trust in data means that nobody is going to use the data to make strategic decisions and there goes all your efforts to create a culture of Analytics.
Having standard definitions is not as easy as it sounds. It starts from you and your team having a clear understanding on how to calculate various metrics. Some seemingly simple metrics can be calculated in various different ways and all of those ways might be right but getting one standard way of calculating those removes any confusion and gets everybody on the same page.
- People have short term memory. In my 2012 post, titled Dealing with Short-Term Memory: Creating a Culture of Analytics, I wrote:
We all make assumptions from time to time; sometime we state them clearly and sometimes we just assume in our own head. We then operate under those assumptions. In context of Analytics, one such assumption is that everybody knows what the goals and KPIs are. We might have defined them on the onset of the program, campaign, beginning of month, quarter, year etc., but once those are defined we start to assume that everybody knows about them and is operating keeping those goals in mind.
Well the truth is that people have short term memory. They do forget and then start to interpret the KPIs, defined based on those goals, in their own way. As the Analytics head/analyst/manager, it is your job to constantly remind stakeholders of the goals and KPIs.
two great lesson for all the Digital Analytics teams.
Clearly define your metrics and make sure the underlying metrics and calculations are clear in your definition.
Don’t make any assumptions, people have short
term memory. Just because you stated a definition of a KPI in past does not mean everybody will remember it and know how tit was calculated. It is your job to make sure anybody
using your metrics/KPI can get to the definition and calculations right away.