NotProvided.eu

Why you should not use Google’s WMT data

in ,

Google data comparisonGoogle proudly announced that they will show more detailed search query data in Google Webmaster Tools. Since the introduction of the query dashboard, Google has been showing rounded and summarized data points which where really useless, especially for low volume search query data. The only useful insights you could get out of it where the ratios between different types of queries, branded versus non-branded impressions for example. The SEO community actually reacted very positive but I think you always should keep in mind that Google is never doing something just to serve webmasters and SEOs. So did Google really improve the data quality and usability for analyzing purposes?

Last Tuesday I already did a quick calculation on some domains and I saw more people complaining about the erroneous data Google is presenting within the WMT dashboard. Most online marketers already are familiar with the fact that you can never trust the data Google is giving you for free so I decided to dive into the numbers and try to detect cause or general pattern within the data discrepancies. I have used Google Analytics for now, but I’m gathering some log file data to set up a more reliable comparison.




Since Google states the data improvements have been applied retroactively, I will take the month December (31 days) as the period. First of all, I had to collect all the data I needed:

  1. WMT: Top Search queries, per day and the total of December
  2. WMT: Top pages, per day and the total of December
  3. Google Analytics: organic search queries from Google. Add the secondary dimension Source / Medium and include a filter containing [google].
  4. Google Analytics: Landing Pages, origin from organic Google traffic. Add the secondary dimension Source / Medium and include a filter containing [google / organic].

organic-search-traffic-ga-filtered

Within Webmaster Tools, make sure you use the right filters depending on which dimensions and filters you use within Google Analytics. The default filter is on “Web”. Make sure to use the “All” setting to get all your data. This can cause big differences if your website has a lot of images.

Update 14-01: I got some questions how to make use of these filters through their API. If you have a look at the URLs, you can adjust your code accordingly:

https://www.google.com/webmasters/tools/top-search-queries?hl=nl&siteUrl=http%3A%2F%2Fwww.notprovided.eu%2F&db=20131215&de=20140114&qme=false&qm=keyword&prop=ALL&region=&more=true

Available variables within Search Query dashboard:

  1. QM: filter keyword lists based on input. If you want to exclude the keyword, add qme=true
  2. Prop: choose between different type of Google search engines: All, Image, Mobile, Video or Web
  3. Region: you can choose a specific region
  4. More: All queries OR Queries with +10 impressions

webmastertools-filters

And for my own website I have compared analytics with logfile data, to check the data validity of Google Analytics (which should be OK for the smaller accounts). To automate this process, I recommend using the APIs from WMT and Google Analytics. For people using PHP the following libraries are useful:

  1. PHP Server-Side Google Analytics Client
  2. PHP script for automating downloading of data tables from Google Webmaster Tools

I have only used pages that got more than 10 clicks, either from the WMT or GA dataset. The data I used is not absolut correct for sure, but I just wanted to show that there is no logic wihtin the differences between the data sets. Another factor you have to take into consideration is that WMT is also not showing all the data if you’re having a popular website, so make sure you take into account the ratio between shown / not shown pages or queries:

not-all-queries-visible

Some aspects to consider when using Google Analytics data for example.

  • I haven’t checked all the profile settings, so for some profiles IP filtering was included for example.
  • Sampled data versus unsampled data, more information at Blastam.com
  • GA is only tracking when Javascript is enabled

Also Webmaster Tools has some disclaimers:

  1. “Some processing of our source data might cause your stats to differ from stats listed in other sources (e.g., to eliminate duplicates and visits from robots). However, these changes should not be significant.” by Google
  2. “To protect user privacy, Google doesn’t aggregate all data. For example, we might not track some queries that are made a very small number of times or those that contain personal or sensitive information.” by Google

Comparison of the datasets

I also have analysed the differences on keyword level, but because of the (not provided) data within Google Analytics, the results are even more alternating than on page level. I did have a look at the total clicks volume per month per domain, WMT versus GA data:

keyword-data-per-month

Top pages per URL, WMT vs GA:

I only have used pages with more than 9 clicks, so the absolute difference in percentages is not going to the moon because of the bigger absolute differences:

differences-per-landingspage

Top pages per day, WMT vs GA.

WMT = blue triangles, GA = orange squares. If there is a structural reason behind the differences, the trend lines should be parallel. As you can see for domain 2, they are definitely not parallel. For domain 3 they are almost identical. These are the numbers for the three domains:

day-to-day-overview

day-to-day-comparison-1

day-to-day-comparison-2

day-to-day-comparison-3

Even considering the fact that most of Google Analytics data is sampled data for the bigger accounts, the differences between the datasets are really inconsistent which makes the data quite useless. I have tried to look to the ratio’s because there are a lot of differences between accounts and the bigger the accounts are, the bigger the individual data errors will be. I have analysed 25 domains in total, and every account has unique differences between the data presented in Webmaster Tools compared to the data visible in Google Analytics.

My message to everyone: think about data quality and validity before using Google Webmaster Tool data for your research, predictions or reporting purposes. It is ok to analyse trends and movements, using individual data points is not recommended.

Reply by Google’s John Mueller

John’s answer starts at 20:02. He pointed out that you should be aware of deselecting the web only option within WMT, as explained above. So I think I should e-mail John some examples 🙂

Comments

30 responses to “Why you should not use Google’s WMT data”

  1. Ammon Johns Avatar
    Ammon Johns

    While it may not, necessarily, explain all of the discrepancies, one huge factor that most overlook is that Google are using two very different data sources to produce this data.

    One of the sources is interaction on Google itself, which it can track exceptionally well using its page views and clickthrough data.

    The other is Google Analytics, which is commonly deplyed as a javascript-based tracking code, and is blocked completely by several anti-spyware products, including by default by Norton’s biggest home-user internet security package – a package pre-installed by default on many computers sold by high street computer stores. I believe that Norton’s 360 still completely blocks GA inside any browser on the machine. Other security and anti-spyware apps and software have similar approaches.

    Reply
    1. admin Avatar
      admin

      Interesting comment Ammon. If that would be the reason between the differences, it would be the same average difference (with some diversity between domains because not everyone is using the same software) don’t you think? But this got me interested so will have a look at this tomorrow, thanks.

      Reply
  2. Ammon Johns Avatar
    Ammon Johns

    Tracking of any sort, even right down to your own server logs, is a very inexact and completely unpredictable thing. Even broad trends can be completely illusory. Caching, pre-fetching, etc all throw tracking off by miles, and of course, the entire strength of speed of Google Chrome is in how it pre-fetches other pages and resources you *might* click on, even when you don’t, creating a hit with no click and no actual pageview, that will be reported as both…

    http://www.cre8asiteforums.com/forums/topic/90941-how-accurate-are-your-analytics-tools-and-data/#entry338093

    Reply
  3. Alan Ng Avatar
    Alan Ng

    Hi,
    Have you tried any segmentation or did you just download all data from WMT? We did some similar analysis and it varies by site, especially if you have a lot of image results. Maybe you should try the same exercise, segmenting for web queries in WMT and then exporting just desktop data in analytics and see how that looks?

    Reply
    1. admin Avatar
      admin

      Hi Alan! For some domains I’m tracking image search and some country specific engines separately and still there are irregular differences between the datasets.

      Reply
  4. Grant Simmons Avatar
    Grant Simmons

    I wish all data was *definitive*, but ultimately we’re relying on a third party dataset that can provide us *trends* over time / segments… and that can be sufficient to justify channel investment, inform strategy / tactics and demonstrate success.

    2c

    Reply
  5. Dennis Sievers Avatar
    Dennis Sievers

    Also don’t forget the cookie consent which might infect the outcomes, depending on your sample set.

    Reply
    1. admin Avatar
      admin

      Good point to consider, in this case I didn’t used domains on which there is cookie consent system present.

      Reply
  6. David Iwanow Avatar
    David Iwanow

    Good start on trying to see how much the data varies as far as I understand it after speaking some folks that are smarted on this stuff than me the issue is that GWT is not giving us apples to compare against GA apples so it’s almost impossible to make it match up.

    Measuring Clicks (GWT)
    Measuring Visits (GA)

    Never the two shall meet 🙁

    Reply
    1. admin Avatar
      admin

      But if that is the case, you alwyas should get a relative difference which would be around the same within a set of domains you compare. Every issue with tracking, will be around the same for every domain, no matter if it is WMT or GA. So overall, trends should be usable but I will try to follow up on this because some datasets also show me that trends are not the same either. The good thing is, trends can be statistically analysed, using raw data right out of WMT and GA is maybe to erroneous.

      Reply
      1. Gordon Avatar
        Gordon

        Good post and its good that you pointed out the difference in the two data sources. However, would you really expect a relative difference?

        Obviously I don’t know which sites you have checked it against but surely each site will attract slightly different audiences who may/may not have be from a demographic that have different software installed and different browsing behaviour, so although it seems logical to think that you will get a relative difference, it may not actually be the case…

        Cheers,
        Gordon

        Reply
        1. admin Avatar
          admin

          Thanks for your reply Gordon. Fair point, but even if I compare two domains, which have the same target group, differences are not logical. For instance, I made sure only to included domains for the Dutch market. What you could do is sum up all the causes for particular data interference (Do not track, cookie blocking, antivirus software, JavaScript disabling etc.), investigate the number of people that are making use of these concepts and plot that against the data of both WMT and GA. What do you think of that?

          Reply
  7. Bernt Johansson Avatar
    Bernt Johansson

    Interesting post.

    I have done some analysis of the GWT click data and Google Analytics visit data my self (but a very small test though) and on a domain level I agree that it is hard to see why we should trust any of the data GWT gives us. Because of the fact that GWT only tracks the top 1 000 queries each day and that visits and clicks are not the same thing, a fair comparison seems like a very hard thing to do.

    What I did was to look at organic traffic (GA) and organic clicks (GWT) to specific landing pages for a short date interval to see if the data was more aligned on a more granular level.

    I did this analysis for 4 different pages and for the full domain. The front pag, a category page (1 click from the front page) and two detailed pages (we can call them product pages if you like).

    I measured my data by looking at how many visits (GA) per click (GWT) each page got. So v/c = 1 means that the number of clicks and the number of visits where actually the same.

    Results:
    All pages: 0,96 (GA: 351 924, GWT: 365 069)
    Front page: 1,65 (GA: 14 494, GWT: 8 776)
    Category page: 1,13 (GA: 5 164, GWT: 4 552)
    Product page 1: 1,01 (GA: 12 062, GWT: 11 946)
    Product page 2: 1,01 (GA: 11 573, GWT: 11 397)

    So as you can see from my data, the two product pages actually aligns quite well and that the front page data is way off the charts. Then you could always ask what the correct ratio between clicks in GWT and visits in GA should be. Any ideas?

    I was thinking that myabe using ratio between Adwords clicks and GA visits might be one way to see what should be “normal”.

    Reply
    1. admin Avatar
      admin

      Thanks for sharing your own experience Bernt! On the end, we have to search for a reasonable ratio between the two datasets. My advertising collegues already asked me to also compare Keyword Planner, Impressions and Clicks data from AdWords as we have seen some discrepancies in that data too.

      Reply
  8. Stephane Hamel Avatar
    Stephane Hamel

    Here’s the analogy I use when speaking at conferences: “If you have a toddler you know there are many ways of taking temperature: ear, mouth, armpit… or rectal… Each one will provide the same outcome, but it’s important not to mix the instrument”.

    What is reassuring in your graphs is those results match up pretty well but the scale is different. Now you know both tools are directionally the same. The big question is: if the scale is different, will it change your decision?

    Reply
    1. Midas Avatar
      Midas

      Great analogy Stephane and furthermore a really solid point on the scale situation. As you allude to they match up pretty well.

      There’s also a number of valid points about GA tracking here, it’s missing a lot of data!

      The fact that WMT and GA are matching up a good thing because it makes the trend and scale meaningful, and certainly in our eyes, useful.

      Reply
  9. Errioxa Avatar
    Errioxa

    I tried a test.

    With an invented word which has no results in the Google serps.
    A link from my blog to Google search with KW, this link launches an event in Analytics.
    So you can count how many impressions should be in “search querys”.

    Then I compare how many impressions are in the WebmasterTool and how many impressions in Google Analytics

    In three days showed me how 29 prints in WMT and 45 events in Google Analytics (35%). In WMT should be at least 45 because each user that caused the event has also led to a Google search for that word.

    The ideal test would be to create a cookie on each user who executes the event in the post (click link), and check when I get a visit to this post if it contains my cookie, so we would know how many users clicked the link and went back into after searching the KW.

    Sorry my english 🙁

    Reply
    1. admin Avatar
      admin

      Thansk for sharing this interesting test! But there could be some problems, what if the user is not loading the SERPs fully so WMT tracking is not getting the impression tracked? But for such a number, I would expect the difference to be lower indeed.

      Reply
      1. Errioxa Avatar
        Errioxa

        Could be a cause, but I do not think the percentage of users who do not fully load page Google results is as large, because Google’s pages load very fast.

        In WebmasterTools could be that loaded data are not yet visible or Google Analytics data are sampled.

        Reply
  10. Tommy Landry Avatar
    Tommy Landry

    Thanks for taking the time to dig into the data and make some comparisons. I’d expect there to be discrepancies, sometimes rather wide, between the two data sets. Aside from the above mentioned issues regarding the “ghost” visits, where a pageview logs but never actually happens due to pre-loading, Google Analytics is not a 100% measurement solution. They use sampling, similar to how market research firms survey hundreds of respondents and scale that up to the population using statistics. I’m unsure how GWT measures the visits, clicks, etc., but if it is also sampled, then we find ourselves with margin of error on both ends of the equation.

    Say both data sets have a margin of error of 5% (round number picked to simplify the math), that equates to a potential 25% error for the combined or compared data sets.

    As with all Google data, it’s crucial to look at trends over time. Without GWT search queries, we have no detail to analyze on the keyword side.

    Curious to see what else you learn after you do the additional research regarding Ammon’s feedback. This is a topic of great interest to me. Cheers!

    Reply
  11. Keesjan Deelstra Avatar
    Keesjan Deelstra

    @Jan-Willem Bokkink: Interesting stuff. I don’t see you talking about the location filter. Our study showed the data compare quite well when comparing between WMT all/1country with GA organic/Google, 1 country.
    You start to tell Google is doing this in its own interest and not for webmasters. My opinion on Google changed for the worse last years but on this improvement I don’t see what is the caveat? For sure they should not have mystified keyword data with ‘not provided’ in the first place. But why are you suspicious on this one?

    @Stephane Hamel: You hit the big question: “The big question is: if the scale is different, will it change your decision?” I don’t think so. These data are the best we have now. I’m glad WMT is doing a better job now, so we also have clickdata under 10 clicks that are less bucketed. We still mis the GA engagement and conversion data though.

    @jan-Willem. the first link in the article is not working

    Reply
  12. Ralf Avatar
    Ralf

    Oh no, it is not getting easier )o: But also with one thing in relation to the WMT´s we fight for years now: The data, belonging to the (1000) Links a Website has/ is shown. In our experience, it takes at least 6 or many more month until not any longer existing backlinks are as well not longer listed in the WMT´s! So we have examples, taking more than ONE YEAR between losing/ deleting a link and beeing confirmed in the WMT´s. Do you share this experience?

    Reply
  13. Spook SEO Avatar
    Spook SEO

    I agree with you Ammon, Google is getting data from different ways and there is not a single way to collect the data the one is the interaction with Google plus that is more helpful if you have Google authorship and it will rank you more perfect if you are doing good. I like the way you have written a post and your way of presentation of data is great.

    Reply
  14. Andres Tapia Avatar
    Andres Tapia

    Hi I did another analysis that maybe is interesting.

    In 2013 Adwords implemented the paid&organic report, which, basically, pulls the data from WMT and gives you an overview of clicks, queries and ranks on organic and paid results.

    The good thing of that report is that it wasn’t rounded as WMT on that time.

    So I decided to use that report to have an overview of the Brand vs non-Brand ratio for organic results.

    The thing is, in 2014 January we all know that WMT started to not be “as-rounded” as before. Kind of, more precise. Therefore I decided to compare the paid&organic of Adwords with the WMT report. Turns out that the different of the data is huge. In WMT sometimes I got on a week 3x more kws that on Adwords, other times, the number of clicks is just 2x times bigger for a kw on Adwords than on WMT (remember I am filtering out the paid clicks on Adwords).

    I wonder why it is happening, maybe any of you have some input.

    Thanks

    Reply
    1. admin Avatar
      admin

      Hi Andres,

      This sounds recognizable! Same with keyword planner data versus AdWords data showing discrepancies.

      Reply
  15. Bryan Avatar
    Bryan

    Interesting post – thanks! I’m analyzing a fairly-new website and have started questioning the GWT data.
    I thought I knew what “clicks” and “impressions” were, specifically I thought that if someone did a search, and Google showed my website as one of the search results, that would be an “impression”, and then if that user then clicked on that result, that would be a “click”.
    Apparently, that is completely wrong. For a certain period, my GWT is showing 0 clicks over a five day period. Zero. I go to Google Analytics, and under Acquisition, it tells me that for the same period, there are some 40 sessions coming from Google with a search term of (not provided), but I can also see specific search terms like “best donuts atlanta” that resulted in hits (with source Google), and totaling 11 sessions and 7 users.
    Does that make sense? How are people reaching the website with source Google and a specific search phrase, yet not “click”ing? Confused….

    Reply
    1. Bryan Avatar
      Bryan

      Ok, it turned out it was my stupidity (or maybe GWT’s). I had switched my site to use https, but Webmaster Tools was only showing http. Why it can’t show both automatically is strage, but in any case when I added the https version of the site then the results look more realistic.
      It took me several weeks to stumble upon this issue – I hope maybe this comment can save someone else some time.

      Reply
  16. Dominikos Avatar
    Dominikos

    Hi

    I just noticed that there is a big difference in the average positions of every keyword between GWT and GA. I cant understand why, maybe because the delayed data of the last days.

    Do you have any explanation?

    Thank you in advance

    Dominikos

    Reply
  17. Cómo afecta el (not provided) al SEO en medios de comunicación – Blog de Christian Oliveira

    […] son precisos. Esto es un defecto de cara a cualquier web. Hay varios estudios demostrando que los datos en muchas ocasiones no se corresponden con la realidad de la […]

    Reply
  18. Inaccurate Search Rank in Google Search Console | HARTENSTINE

    […] Why you should not use Google’s WMT data […]

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *