Featured Miscellaneous Search Engines Website Traffic

How to Stop Bots from Messing Up Your Traffic Analytics

Not all bots out there are good bots. Trolling your sites are malicious automated machines that can cause a number of problems. These bots inflate analytics data and can be responsible for attacks on your site. They are even handy tools for scraping and copying your content. Before you can properly use analytics to accomplish anything you need to have good analytics data. The old saying “Garbage in, garbage out” is doubly true in this case.

Content traffic data is important in determining how successful your content marketing strategy has been over a period of time. It also allows you to develop new strategies built around improving your content marketing.

These bot visits skew the data so that you can’t trust the numbers to gauge your performance. It’s crazy – the 2014 bot traffic report by Incapsula found that over 50% of all website traffic was due to bots. It was also discovered that if you build your brand and become recognizable, it was likely that you would see more of an influx of bots to your site.

3 Key Ways to Deal with Bots & Win

Sounds like a “lose-lose” situation, doesn’t it? If you improve your visibility and market more and get a wider outreach then bots will also take more notice of you and the chances of a bot attack go up along with your popularity. Luckily, you can control bot visits relatively easily. The idea behind blocking bots is to find out where they come from and stop the source.

This might seem like a complicated piece of internet wizardry, but the reality is as simple as the following steps:

  1. Locate your Log Files: Servers all have a series of log files that make a record of visits to the website from each user based on their IP location. These logs are usually stored on your server. If you have a hosting company that gives you cPanel as your hosting front-end you can simply access your logs by clicking the link in the main cPanel window (the first window you visit). If you use Apache for your front-end, your logs will be in the /var/log folder. IIS users can configure their logging through the local computer’s control panel. In the control panel, you select administrative tools, then internet services manager, then select website, right click and then select properties, select website in the tabs then available, then on to properties and finally the general properties tab. In typical fashion, the Microsoft logs are the hardest to get your hands on.
  2. Figuring Out the Most Visits by IP and User Agents: When you get your log files downloaded, it’s a simple matter to consolidate them into a single text file and then import them into Excel (or whatever you prefer to view your log files in). Excel is a very innovative way to manipulate data so that you can make sense out of it. When you import your data into Excel you can select the space delimiter to get the right data into the right columns. With a little cleaning up you’ll have usable data almost immediately.

Utilizing Excel’s Pivot Table Builder, you can create a pivot table to link number of visits to Client IP and then get a feel for the counts of visits from malicious IP’s. Client IP’s determine where your visitors are coming from and can easily give you insight into where most of your visitors are based. Renaming the table headers to Client IP, Hits and finally a User Agent column gives you a setup to determine which IP’s have visited your site the most. The User Agent determines the browser version and the operating system version that your visitor was running. Obviously, bots would have none of these so it’s just a matter of determining which ones are blank to pinpoint the presence of bots.

  1. Blocking the IP: After you’ve figured out which IP is the bot location, you can now move forward in blocking reference to the bots in your analytics reports. Additionally, if you’re concerned about security, you can also block the bots from accessing the site altogether. Google Analytics gives you the option to block individual IP’s. It also comes with a built in bot-checker that you can enable in the Admin panel under View Settings and by selecting “Exclude all hits from known bots and spiders.” A handy tool for filtering your analytics to get a more realistic view of your outreach. Omniture gives you a bit more control about your analytics viewing and tabulation by giving you the option to exclude individual IP’s, exclude a set of IP’s (if you have a large number of bot entries) or create a processing rule that ignores certain IP’s and IP ranges.

On the server side of the spectrum, you can limit the availability of your site to certain visitors based on their IP. CPanel includes a handy IP Deny manager which allows you to enter IP’s that you can deny access to. In Apache, you can utilize either the mod_authz_host module or the .htaccess module, but the former is the more preferred method for controlling access. Open IIS Manager allows blocking through its features view, then navigating to the IP4 Address and Domain Restrictions, then to the actions pane and finally adding the IP address of the bot into the Add/Deny Entry list.

The Threat of Bots and How to Combat Them

Even though bots can be a nuisance, being aware that they exist and are an active threat allows you to deal with them. These methods, although they do work, are not 100% effective in determining whether an IP is a bot. Constant traffic monitors via third party solutions can give you a more secure method of streamlining your traffic and keeping bots away from your pages.

Similarly, the introduction of ReCaptchas can stop bots and prove to be a viable measure, although it can be quite annoying if you’re a human to prove your humanness. At the end of the day the security of your site is of the utmost importance. It’s better to prevent bots from accessing your site than having to deal with the fallout of their malicious actions. Prevention will always be better than cure, and when it comes to websites, bots can deliver quite a blow to the viability of your content marketing strategy with very little effort.

 

About the author

avatar

Julia McCoy

Julia Spence-McCoy is the CEO of Express Writers, an online copywriting agency that began in 2011 with thousands of web content pages written to date and more than 50 talented writers on the team. Her passion is copywriting and all that pertains, including the ever-changing game of Google algorithm updates.

22 Comments

Click here to post a comment
  • Good comment, Julia.
    A new client of mine came to me with an issue around Google indexing. They had previously been getting a lot of attacks, hack attempts and unwanted comment spam. It came to light that they had installed an country blocker on their website and had set a massive list of blocked countries including USA (this business is located in New Zealand). You can probably guess what the result was? Googlebot has a USA IP address and was blocked from accessing the site. A simple case of over zealous blocking took the entire website out of the search results cache, with exception of some content files that were not in a blocked directory.

    Blocking IPs can be a minefield. Tread carefully, and think it through before reacting. IP blockers are a brutal and merciless line of defense. That said, there are usually plenty of countries you just don’t get good traffic from, ever.

    If the issue is around Analytics, consider setting filters in GA views and keep an eye on any new ones. They all have either IP or domain name footprints. When setting a filtered view, remember to create a new view first to test the filter, so that you don’t accidentally mess up your data, then apply the filter to a master view (but not your raw data view – which you should leave unfiltered as a base).

    For enquiry form submitters, use a ‘honeypot’ type filter that is visible to bots, but not humans. If the field gets filled, it’s a bot.

    For comment spam, use moderation systems and also disable or close comments when not required to be open.

    Perry Bernard
    Cranked SEO Personal Trainer.

  • Some really good information on controlling fake visits from bots and spamming sites.

    I have been frustrated many times seeing this in my analytics reports. Finally I can fix it .

  • Thanks for sharing this information ……… can you show me how to block any country traffic or websites using wordpress …..

    I am waiting for Your reply ……

    Thanks,
    Pradeep

  • Hey Julia,
    Really awesome article with noteworthy information. The tips you have given in the post are really helpful and worked for me.
    Many thanks for the tips and the information.
    keep sharing 🙂

  • Everybody has a problem with this, but why shouldn’t Google try to solve this problem instead of us. I try to block some spam bots, but it is useless because the next day there is a new one plus google says that my report is incorrect because of filtering.

  • Thanks for the highlight you have just shared on the subject matter. This has really become a subject matter of interest lately to webmasters. Google has provided a way around it by means of filter in GA. There are many suggestions on the internet on how to go about filtering those bots since they are classified as either Ghost spam or Crawler spam. I have found this article as an effective approach to filter out the bots from GA http://www.ohow.co/what-is-referrer-spam-how-stop-it-guide/

  • Thanks for sharing information, You really made a great effort to share your knowledge about spamming in analytics data.

    I also suffered from this type of spam, called crawler spam and ghost spam.

    I applied some filters to get rid of the issue, also checked the “hits from known bots and spiders” option and applied blocking IP address of spam bots and sources of spam traffic.

    Now I rarely see traffic through those sources.

  • This is a great post that can help Google Analytics practitioners. Thank you for writing about how to stop bots. I’ve tried filtering them but they keep on coming back.

  • Elvira immoblienmakler munchen provides the best deals in buying or selling the best manner. If you are looking for property and their deals, elvira immo is the right place for you.

  • Hii Julia, You have posted awesome information here to stop bots and save websites from any spam activity. It would help me alot as a Google Analytics practitioner. Thanks a lot for sharing this kind of content.