[1:30] The Semrush Ranking Factors Study – What is it?
[3:20] The methodology to address the correlation implications.
[6:20] How should SEOs remain skeptical yet find value in these SEO correlation studies?
[10:15] Highlighting limitations and the nature of correlation.
[12:20] How did you decide on the controversial title and the marketing of the study? The metrics created behind ‘content quality.’
[19:45] Big insights and takeaways from the Ranking Factors Study
[22:00] What’s your take on SGE and GenAI based on the study’s findings?
[26:30] Do you think Brand recognition should trump content quality in search rankings?
[30:05] Rapid Fire Rankings
In episode 129, Erika Varangouli shares the time, effort, and marketing discussions that went into Semrush’s 2023 Ranking Factors study. Too often, uninformed SEOs will use statistics from these correlation studies and use them to serve their own agendas despite them not proving causation.
Erika argues that the data still has value to inform and guide us. We can apply the insights to hypotheses and experiments to our own websites without using it purely as a prescriptive playbook.
She explains why the team landed on a study called ‘Ranking Factors’ and use their own metrics quantified as ‘Content Quality.’
Title: Former Head of SEO Branding - Semrush
Rank your best SEO marketing win:
Rank your top 3 SEO tools:
Rank your best SEO or trick or tactic:
What’s the biggest scam in SEO that you’ve recently seen?
Rank what you love most about the SEO industry:
Rank your top 1-3 marketers:
What’s your Generative AI hottake?
Garrett Sussman: Welcome back to another episode of the Rankable Podcast. I’m your host, Garrett Sussman, of iPullRank.
I am really excited to dive into the ranking factors study at Semrush. Last year, Google was ridiculous with so many algorithm updates.
And yet, Erika Varangouli is joining me today. We’re going to talk about her role as the head of SEO branding at Semrush. She has been a regular speaker, a webinar host, and an awards judge on SEO and content marketing topics, working with all sorts of companies, including Capterra, Symantec, Travelex, ASICS, HSBC—you name it.
She’s been involved. She is a friend of the community, cares so deeply about all of us in SEO, and she doesn’t deal with the BS either. So, I’m excited.
Thanks for joining me today, Erika.
How are you doing?
Erika Varangouli: I’m good, I’m good, Garrett. Thank you so much for this intro. I mean, I need the recording, and I need to start selling myself more like that in the future. Thank you so much.
We got to hype ourselves up. You know, nobody will know. We all have cheerleaders in our background, but like, you are all over the place talking about this ranking factor study.
This is a massive analysis, a correlation study, if you will.
Can we talk before we dive into the questions, just give us a breakdown of what it is, what you aim to achieve at Semrush with it?
You and Marcus, well, Marcus, you know, and the whole team put so much work into it. So, tell me about it.
Erika Varangouli: Yeah. So first of all, yeah, I would want to start with that. It’s, I have the privilege of talking about the study and sort of raising awareness about things we found and having very interesting discussions, but this is the result of over six months of like some very, very capable and much smarter people than me working on this.
So, you know, Marcus Tolber, who’s our SVP of enterprise sales, of enterprise solution, Yulia Ibragimova, who’s the head of our research team, and her team, all working on that, our brand campaign team. It’s like those awards moments where you, you know, you get the Oscar for the film, you say, “I want to stay with it.”
But I’m saying that to show that there’s a lot of people from different backgrounds and different kinds of knowledge and skills working on that for a really long period of time. So the idea behind that, and I don’t know how many from our audience may know of these studies, like in the past, for example, Marcus used to do them a lot when he was at Searchmetrics, and looking at what are the relations between factors or signals we know because someone from Google has confirmed they’re using it or because many of us in the industry talk about them and have seen some sort of relation.
But what can we observe in terms of those factors in relation to higher rankings of sites?
The tricky part being that relation, and thank you for highlighting the correlation part at the start, which is, you know, there has to be a methodology into how we assess it.
Obviously, we’re not in the algorithm. We cannot directly kind of say, okay, we know it works exactly like this. We’ve tested it across like at a huge scale. We’ve also done correlation.
Everything points that this is 100% the cause.
So what you do instead is you try to measure that correlation. We use the Spearman model just because it seemed and it has been the most appropriate through time to sort of test this kind of thing to see how do rankings move when you check for a specific factor across the SERPs.
We did that.
We started with a big set of keywords. So that’s our starting point, which we cleaned up terribly though. It shrank 10 times. That’s why we started with a really big sample as well. And that allows us to study over 300,000 listings on the SERPs. So that’s a decent sample, right?
Obviously, how much is too much?
It could be more, but it’s a decent sample to be able to get to a stage where you can present some things. Like we saw that when we studied, for example, content relevance and how we measured it, we saw that it correlated to a higher degree than when we studied having a keyword in a URL. And then it’s just presenting the sets of factors.
I have to say the final set was like we examined 60, 65 factors. Again, we had started with more, but
We excluded a lot of them as well. And then, for those factors, grouping them, so some are content related, some are backlink related, and sort of come to marketers, come to business owners, come to people who work on a website and are curious to see, okay, so of all the things I hear as good or bad ideas, often contradictory opinions as well, can I see something?
Has someone studied any relations at scale?
And that’s what we aim to do is say, okay, nowadays we studied these factors and we found these relations between higher rankings per factor. It’s fascinating.
So we’re talking like 16,000 different keywords. You call out, it was specifically the US market. It’s for mobile results. So mobile shows up differently than desktop.
We know, that’s why we joke in SEO. There’s so much like it depends on different types of queries, different types of industries, different types of search features all affect all this. So to your point, it’s a fascinating correlation study to see what aspects of content, what aspects of linking influence it, but we don’t necessarily want to take away this directly causes that.
Garrett Sussman: How can SEOs from your perspective, how should they use your study to inform their strategy while at the same time maintaining a healthy skepticism about the direct impact of the identified factors on search records?
Erika Varangouli: That’s a good question because also obviously I’ve been involved in conversations online about it, and I’ve heard people’s objections or I’ve seen like posts being more on the causation side of interpretation and I’ve stepped in to say, well, I think this is wrong.
But here’s where I’ll start because when we started, obviously we all had our experiences of previous ranking factor studies, right?
Marcus, myself, the team at Semrush, we’ve all done it before. And this is a big part of our fear. It’s like, go out there, publish this, then you get all the backlash because something doesn’t resonate or because many people interpreted it as causation and then you get the backlash, if this is not what you did, you’re lying, we’ve been there.
But we all agreed that the starting point is that when you’re in SEO, like relying on data, using data as a basis to make decisions is crucial, right?
It’s not the only thing or the only angle you have to come at things, but it’s crucial. And we have tons of data.
So it didn’t make any sense for us to sort of decide to do nothing because the something we could do is not perfect or is not the answer to everything. So that’s the starting point. And the other thing also for me, which I’m a bit precious about is like, I often like I’ve worked with small businesses, solopreneurs, and big brands, right?
And people are interested, like their livelihoods are on their website. So when they go online and research or they go on social media and they see what people post, they end up being very confused at some point, right? Especially if they don’t practice SEO.
So they hear, this is a tactic, this is my strategy and it worked amazingly well. And then they might find something else that says we tried it and everything went wrong. Or they may see, Google said this, like, I don’t know, John Mueller said this or Gary said that. And then see tons of tweets or posts about it saying they’re lying to our face.
This is not true. Don’t believe them.
So I think this is a very tricky, these are like weird waters to navigate for anyone in this.
And for us and for me, the idea in particular, and that’s why you’ll see throughout the study, not like just in the methodology or we added a huge limitations part at the end, but throughout the study, which I know was a bit long, so apologies to anyone who suffered through reading it all. Garrett, I think you were one of them. And I think you were one of the first people to read it.
We made sure to highlight that, you know, we found this, but we’re not interpreting it as, okay, do this then and you’ll rank or don’t do this or do this and you’ll suck. You’ll tank if you do this. And we make sure that every factor we analyze, we add like a practical element of what does this mean for you?
And when we were writing it up, we were thinking, okay, the person we’re talking to is not Garrett, right? It’s not Erika.
The person we’re talking to may be a junior SEO, like in their first role in a team trying to propose to their manager what to work on next, or a small business owner who pays a freelancer and is not sure what they should allocate their budget to like, okay, what to work on next, or a small business owner who pays a freelancer and is not sure what they should allocate their budget to like, okay, what? So making it clear that this is correlation, for me, the ideal scenario is that regardless of who you are, when you read it, it pushes you to really think critically about the things you are told, the things you see online, the things you read by Google, and your experience with your own results, like with your own program, your own strategy. I feel that anyone who is an SEO or is practicing SEO content, I don’t think they would see something in this study that would surprise them.
Like I’m saying that and I’m not really selling it. I should be saying like, really, you won’t believe what we found. But factor number seven will surprise you.
But I mean, I feel for many of us, it was just a positive reinforcement of the things we believe or we know to be true or we see a lot. For everyone else, I think it’s just a good direction.
Okay, I saw that content quality came really high. What is my content doing on my side? How good is it? What are the guidelines about what good content means nowadays? And try to start testing it and improving and iterating on your strategies more and more.
Garrett Sussman: Oh my God, there’s so much great stuff to unpack there.
And I really do agree with you because I feel like in a lot of ways, SEOs have responsibility almost as scientists to hypothesize, test, and then work, move forward based on the results. And I think a lot of the ranking factor study, the correlation study really gives you at least a guide of what to test. And then you point out the different audiences, enterprise, small business.
In one of the highlights of the study talks about reputation and the star ratings on a site. And obviously, that applies a lot more to a local business, which is a completely different algorithm than your global SEO. So it really can be confusing for people who are getting started.
But one thing we were talking about that I want to tap into is right before we start recording, it’s talking about how intentional you and the team were with the phrasing, how you worded different aspects of this. You just mentioned content quality. And that was something that you actually created a metric for around keyword density and relevance and word embeddings. It’s more complex, but choosing content quality has a certain intention that people think content quality just isn’t good content. It’s not as scientific as what you did.
The other one I would love for you to address is maybe the controversial part is deciding to call it a ranking factor study in the first place when we know how complex Google is and implying it’s a ranking factor.
So you’ve done these in the past almost implies that we want to drive a narrative from the data, which is always risky. So can you speak to those intentional decisions in the wording and what the mindset was there?
Erika Varangouli: Yes, of course. So I wrote down both of them because I think I’ll forget while answering the first one, but you get me back into track if I digress.
So let’s say the quality, the content quality score. A true problem with content quality is that it’s not really measurable. So we have what I call proxy metrics. You can have metrics that you find anywhere from using Grammarly or Semrush SEO writing assistant, like they give you a score based on Flesch-Kincaid based on phrasings.
But when we look into Google guidelines, what is helpful content? What is quality content? What is quality user experience?
This is something way more complex than any of those metrics can get. And again, we do not profess to have quantified content quality, but what we had to do was either one of two things, either completely ignore it and sort of say, okay, we cannot measure content quality with any of the metrics we have on hand, right? Or we could try to come as close as possible within the context of a study and the resources available and all of these other things around us and say, okay, here’s a valid proposition as to how we can quantify because it’s an observational, it’s based on quantitative analysis, right?
Host: Garrett Sussman
Title: Demand Generation Manager
Garrett loves SEO like the 90s loves slap bracelets.
Each week he interviews the most interesting people in the world (of SEO). When he’s not crafting content, he’s scouting the perfect ice coffee, devouring the newest graphic novels, and concocting a new recipe in the kitchen.
Get insights, stories, and strategies from a range of practitioners and executives leading the charge in SEO.
Enjoy this podcast? Check out Garrett’s video show round-up of everything search engine optimization: The SEO Weekly