November 2, 2016

How to Get the Whole Number Search Volumes that You Miss

Tired of the Keyword Planner's antics? Walk through our process on how to get whole number search volumes. There's also a free tool!

Mike King

Google’s erosion of keyword data continues to feel increasingly unfair. It started with not provided (aka Google Secure Search) on the measurement end and now the AdWords Keyword Planner has become more useless for determination of keyword value on the strategic side.

Perhaps this is Google’s way of forcing Organic Search practitioners to shift their focus away from keyword targeting toward targeting concepts and generally thinking of the channel as more opaque. The combined series of moves that they’ve made would suggest so. Today, I want to talk about the alternatives and share an internal tool we use for getting whole search volumes quickly.

Not provided, Not Surprised

To be fair, I’ve always felt like Tom Cruise doing some super secret maneuver to infiltrate an enemy’s base and steal intel when I’d pull search volume data. I’d often think, I can’t believe they give this to me when I don’t run any AdWords campaigns! So, I’m not surprised the change was made in the AdWords Keyword Planner.

However, I’m an entitled user of Google products just like anyone else, so I want my data the way I have always had it. So what are the options?

The Available Options for Search Volume

Clever folks in the SEO industry, of course, have reacted in kind. For example, the Screaming Frog team noted that if you move your keywords to the Forecaster within AdWords then you’ll get whole numbers. However, in my experience that has been hit or miss. So that’s left me to examine some of the other available SEO tools that provide search volume.

There have been indications that Google made this move to stop scrapers from snatching their data. So that begs the question: where the different SEO tools are getting their data from? Is it fresh data? Is it accurate?

So, in preparation for this post, I reached out to the different tools that we use to get a sense of accuracy and freshness. This is not an exhaustive list of all the places you can potentially get search volume as part of your keyword research process; rather, it is a list of tools that we actively make usage of here at iPullRank.

KeywordTool.io – We primarily use KeywordTool.io for researching on the long tail. While I do not use the Pro Features myself, the site indicates that they can provide search volumes. Via email, they confirmed that they can indeed still provide accurate search volume, CPC and AdWords competition.

SEMRush – We use SEMRush for a number of things, but mostly for competitive analysis. Of tools tracking providing this type of intelligence, they have, perhaps, been doing it the longest. So their dataset is one of our go-tos for calculations of rank difficulty and otherwise quick research.In SEMRush’s response, they indicate that no one has accurate search volume data and they were always delivering estimations, but they have not continued to update the data month over month since Google made the change. However, they have a big update in the pipeline.
Ahrefs Keyword Explorer – Similarly, Ahrefs has a Keyword Explorer product, but they have not been able to update it since the change in Google. As one of the quick movers in the SEO space, it’s not surprising that they have an updated product coming soon.

Moz Keyword Explorer – Moz is perhaps the farthest down this rabbit hole of keyword estimation. They’ve been incredibly transparent about their Keyword Explorer and how it’s built from models from clickstream data.Just this week, Britney Muller put out a detailed examination of accuracy between KWE and AdWords Keyword Planner. However, I’m having difficulty wrapping my head around how volume ranges are meant to be used. More on that in a second.

All in all, everyone seems to have their own brand of solve for this. While I’m a strong believer in modeling and extrapolation of this kind, it is difficult to make a shift to something imprecise when something precise is available. For the same reason, estimations of not provided never really got a foothold in the industry, I wonder if we’re

How do YOU use Ranges?

Perhaps, I’m being short-sighted, but switching to using volume ranges doesn’t quite make sense to me in terms of the many things that keyword research powers in organizations. Granted, the idea of forecast modeling for Organic Search is somewhat ridiculous to begin with. Nevertheless, it is still something that clients require to get buy-in from upper management. So how are we expected to use ranges? Are we meant to average them? Take the upper bound to show potential opportunity? Take the lower bound to be conservative? Present them as just ranges and throw the idea of forecasting entirely out of the window?

Frankly, I am unsure how both Google and Moz expect us to use these numbers. So I polled my followers.

In my incredibly imprecise survey (n=120), nearly half of my followers indicated that they present the numbers as-is.

I guess you could build forecast models from both the min and max of the range so you can determine the best and worst case scenarios, but I anticipate that opens the door to a wealth of questions that are difficult to answer. Also, there is the challenge of scaling any of this to thousands of keywords.

While I haven’t taken the time to think critically about how to effectively work ranges into our keyword research strategy, I do know that we have a very strong approach built on whole numbers. So, I, like many of you sought out a way to keep getting those.

Hello, GrepWords.

GrepWords has long been a quiet response to the difficult prospect of attaining an AdWords API key. You see, to get and maintain an AdWords API key, your application has to offer bid management capabilities. In other words, you’ll only maintain the key if your app helps Google make more money. In the past, many SEO tools had an AdWords API key, but Google regularly audits those using those keys and often use their leverage to strongarm SEO companies into removing features. Famously, Moz did not back down and as a result, lost their key.

I tried to check in with Russ Jones, who is involved in both Moz’s Keyword Explorer and GrepWords (separately) to get a sense of whether or not they have been actively updating the database. At the time of this writing he was OOO, so I’ll follow up and update this once I hear back from him.

GrepWords acts as a data provider that makes that data available to users without that requirement. While there are other options out there, such as SEMrush, GrepWords offers the data in a no-muss, no-fuss JSON format via an API. The Builtvisible team’s SEOGadget plugin also makes it trivial to get the data within Excel.

That plugin is fantastic, but it’s not my personal preference. I don’t enjoy wrangling API queries from multi-cell responses into the format that I want. Rather, I prefer to manipulate my data via a scripting language and make it return exactly what I want. So naturally, I’ve written a script that the team can now use to get their search volumes.

GrepWords is pretty light on documentation, but that is a result of its beautiful simplicity.

There is no indication, in the documentation, of how rate limiting is managed, or what response codes represent, so without some testing, it’s difficult to really set up granular fault tolerance. That being the case, I just default to assuming anything that doesn’t return a 200 response code is an error.

Upon playing around more with the API, it seems that every response is a 200, but the JSON returned may give an error such as [{“error”:”july- doesnt exist in fresh”}[] or simply a text response of “query too short” when you try a query such as “m” or “mi.”

Nevertheless, there are two API endpoints: Keyword Lookup and Related Keywords. Their respective behaviors mirror that of the Keyword Planner. The Keyword Lookup endpoint allows you to provide a series of keywords and it returns their local and global monthly search volumes (in whole numbers), cost per clicks and the search volumes for the past 12 months. The Related Keywords endpoint allows you to provide a keyword and extrapolate the related keywords and same values noted above.

Everything is a GET HTTP request, so you don’t have to go through the trouble of managing a POST request. In fact, because there is a documented option for the API to return a CSV of your results, you could potentially get your data directly from typing a URL into your browser. Note: Despite it being documented, I have not been able to get it to work.

Let’s try it, go ahead and register for an account, despite the fact that there is no messaging to this point on the site, every account level offers a two week free trial.

Keyword Lookup Endpoint

You should see a result like this in your browser:

Let’s break down the Keyword Lookup query above:

The apikey parameter is your API key. No surprises there.
The q parameter is your series of keywords, delimited by a pipe (“|”). The API can handle up to 1000 keywords at a time (Moz’s Keyword Explorer can return 1000 keywords at a time as well), even more than the 800 maximum that the Keyword Planner would return. However, you will be limited by the length of query that you system allows. Being that this is a GET request, most clients allow URLs of up to 2083 characters long.

And that’s all you really need to know. The optional parameters are:

output – This allows you to specify if you want the response in json, xml or csv. By default it’ll be in JSON which is great because JSON is better than either of those other formats for manipulation and passing between the front end and backend. I’ve not been able to get the API to return the XML or CSV files though.
loc – GrepWords provides keyword data for 32 different locations. It defaults to US, but you could for instance run that same query with loc=united_kingdom on the end of it and get very different data.

Notice how this query returns estimated clicks, impressions, cost, CTR, average CPC, average CPC, country and kwp_date.

Related Endpoint

The apikey, output and loc parameters are above exactly the same in this endpoint. The following parameters are the ones that differ here:

q – In the case of the Related Endpoint, the query parameter allows you to input a keyword or a regular expression.
start – This optional parameter allows you to specify which number result you should start from.
results – This optional parameter allows you to specify up to how many results to return. The default is 1000.
sort -This optional parameter allows you to sort (in ascending order) by any of the given variables that the API returns. For instance if you want to sort by keyword from A-Z you’d add &sort=keyword to the end of the URL.
regex – Allows you to specify whether or not your query is a regular expression. I suspect that if you regex equal to anything it will work as the documentation does not specify it. My guess is that the system just looks for the presence of a regex variable in the parameter.

Building a Tool

Our Manager of Content Strategy, Fajr Muhammad was in the office on a Saturday getting caught up on some keyword research when she was struck by the volume range issue.

I caught wind of her issue and decided to whip up something to help out. Using Google’s Material UI for the frontend and building a quick AJAX handler script in PHP, I was able to throw something together quickly.

Leveraging the DataTables plugin for jQuery, it also made it pretty easy to persistently collect a set of keywords and then export them by copying to the clipboard, making CSV, Excel doc or PDF.

Just Give Me the Tool, Mike

At this point, you’re probably like “just give me the tool.” No problem.

You can find the latest revision of the code I worked on for Fajr at our public GitHub repository. In order to install it, you’ll need a web server running PHP 5, either online or on your local server. You’ll also need an API key from GrepWords.

Here are your steps to get started.

Download/Clone (https://github.com/iPullRank-dev/grep-volumes.git) the Git Repository
Extract it into a folder on your local or remote server
Open it in your browser (https://localhost/grep-volumes)
Put in your API key
Put in keywords
Enjoy search volumes

Alternatively, you can check out the live version of the tool here. You’ll still need to bring your own API key.

Also, I haven’t gotten around to working in the related endpoint, but if people get some good usage out of this tool, I’ll make it happen.

Where Does Keyword Research Go From Here?

I’m excited to see the SEO industry respond both reactively and proactively to Google’s move to obfuscate search volume data. While there are features that all of the different keyword research tools have that make them valuable, I’m not quite ready for the shift that ranged data requires. All that said, I’m curious as to how others are approaching keyword research in this new range-driven environment.

I’d love to hear your thoughts. How have you changed your approach to keyword research due to Google’s recent changes?

Author
Recent Posts

Mike King

Mike King is the Founder and CEO of iPullRank. Deeply technical and highly creative, Mike has helped generate over $2B in revenue for his clients. A rapper and recovering big agency guy, Mike's greatest clients are his two daughters: Zora and Glory.