Are Spiders and Bots your customers?



Those who know what internet bots and spiders are, know that
know that spiders and bots visit your site very day and multiple times a day.  For those who don’t know here is the definition
of bots according to Wikipedia: 

An Internet bot, also known as web robot,
WWW robot or simply bot, is a software application that runs automated
tasks (scripts) over the Internet. Typically, bots perform tasks that are both
simple and structurally repetitive, at a much higher rate than would be
possible for a human alone.

According to a recent survey, on an average about 51.8% of the website visitors are
bots.
One assumption many digital analysts make is that the JavaScript
based analytics solutions such as Google Analytics, Adobe etc. either stop bots
from executing the script or filter them out before calculating the metrics.  Well, this assumption was true years ago, it
was one of the selling points of JavaScript based solutions as compared to log
file based solutions.  However, things
have changed over past few years, many bots are now capable of running JavaScript and hence polluting your reports.  Likes
of Google Analytics and Adobe Analytics will filter out the spiders and bots to
an extent but considering the number of new bots that emerge every day, it is
not an easy task, neither for them nor for you. 
So what do you do?  Since you are
not going to get 100% bot free report filter out as much as possible so the
effect on your reporting and analysis is minimal.
I recently had a conversation with one of the attendees of
my Digital Analytics Association (DAA) workshop at Chicago eMetrics. She told me that she gets a report every
week from their digital analyst. When asked, Digital Analyst confirmed that the report does
not filter out any activity from spiders and bot becuase he does not have time to
remove them. She was wondering if she should worry about it and push to remove
the bots from report or just accept it.

What do you think? Do you think the report she is getting is
worth anything?

Unless they are selling to spiders and bots, she is not
getting an accurate picture of website usage by real customers.

Make sure to ask your analytics team if they are removing
bots from reporting. If not, then do not accept the reports till they have done
cleanup and are paying attention to it on ongoing basis.

Comments? Questions?

This entry was posted in Data Analysis. Bookmark the permalink.