How To Interpret SEO Correlation Studies

It’s like, how can we quantify it so at least we can see if there is a correlation there.

So we created a metric and this is a metric as well that Marcus was building for the enterprise platform. It can help you come closer to understanding how good content is.

That was intentional because we all know, we all read, we hear Google, we know that this is probably something important that you don’t want to ignore in a study of this type. It’s not like we’re ignoring something that used to be massive 10 years ago, but no longer is.

And so, okay, maybe no one cares. And so we approached it this way. Plus for relevance, we used embeddings, which took us beyond just the phrase matching kind of approach, the traditional keyword analysis of does it have that in the text?

We also did study those. And for me, one of the great things and just to iterate is like, okay, they came up not as strong signals, whereas relevancy, when we measured it with embeddings, allowed us to actually see that the relation is there. So for me, that is just doing a study like this in 2023 and 2024 versus 2020, for example, or 2017. We all have to progress. And for me, we have to do a lot better in the next one, since we all survived it. And I’m hoping there will be a next one, which will be even better.

But we can come back to this if you want, but I really want to address the calling it ranking factors because I know also that is a topic that triggers some emotions and I understand them. So my answer to that is obviously, we are not Google.

We do not know like here’s a bucket of factors. Here’s a bucket of non-factors. We declared that the way we approached it is we used things, we explored things or signals based on two criteria. Other Google has said that they are a signal or the discussion in the industry around something being or not being important for us made it interesting to look into because we know for people to be talking about it, there might be something there. It will trigger their interest.

This study is done by a company, is a marketing content that we put out. So how you market it is also part of it.

So when we were discussing, okay, how do we call it?

Like, do we call it something else? And when I saw some people saying, but you’re lying to our face, this is not a ranking factor, this is misleading.

I’m thinking, okay, let’s come up with something that is equally appealing in terms of marketing, but can also disclose quite succinctly like what it does. I couldn’t find anything. I don’t think, maybe I’m not a good marketer. I don’t think I could market a study that is titled correlation exploration of 300,000 SERP listings with the view of understanding if something like it’s, this is a naming convention that has been used in our industry for years, which means that people have already formed an idea of what to expect when they see that naming convention.

So all joking aside, I hear that kind of criticism. We make it very clear that, you know, it’s like the factors in the signals we’ve selected in this way to explore X, but how we name it in a way that is clear and perceived like in a consistent manner, I think that does a pretty good job. However, if anyone has a better idea on this, we debated long enough about this internally, but yeah, I think it’s fair as long as we are not tricking anyone into believing we did something we did not.

Garrett Sussman: Yeah, no, and I think that’s fair. I think it’s really hard. I think that intentions matter. And obviously, as we’re discussing, you put a lot of thought into that because the last thing you want to do is mislead and you really are focused so much on building community.

And to your point about Google, A, it’s not like we’re going to have a Yandex situation where someone’s just going to release files and we’ll have this list of factors to parse. And on top of that, that said, we do have the DOJ lawsuit where there’s been reveals about Google talking out of both sides of their mouth in terms of being deceptive.

And it’s funny just reading through the correlation study of how you address user signals and bounce rate, like not being correlated with rankings and things like that. So it’s still interesting, regardless of how the narrative plays out, just to at least look at that data.

Was there anything that surprised you from the study or that gets you excited confirms what you believed?

Erika Varangouli: There were tons of things, like the amount of commerce, the volume of conversations on Slack and on calls with the team.

Honestly, I cannot even describe. I think if I kept something in the end, so initially I was more hopeful that some correlations would be higher. So I was expecting things like the authorship or branded share traffic, like those correlations I thought would be higher. However, once you remind yourself of the limitations, this is not broken down by type of query or by industry. There’s a lot of things at play there.

I don’t think now that you ask me like this, I don’t think something was hugely surprising for me. It was just mostly signals around brand trust, authorship, things like that. But again, and that made it into the studies, we made sure to say, look, our study didn’t show a correlation. We believe it might be also due to its nature and its limitations.

So the one thing I don’t want is an SEO going to their manager tomorrow after reading it and say, look, we need to scrap all our schema. We should stop anything we do there. Or conversely, we should just do schema from now on.

That’s the answer to everything. So I hope that has not been the case for anyone. And yeah, I would be happy. I mean, that’s a great point. I saw that too.

It doesn’t surprise me that schema doesn’t play a bigger role because the truth is most people don’t, unless you have a plugin that basically does it for you, I’d say nine out of 10 websites are not going to even think about schema in the first place, even though it’s so valuable as we move forward towards an AI-driven, LLM-driven sort of web.

Garrett Sussman: I am curious speaking of AI, a little bit of a pivot, but the last year has been AI this, AI that. I know SEMrush has incorporated a few minor AI features you’re looking to.

What’s your perception on Google results and how AI, you know, like what future trends in SEO should specialists be preparing for based on the study’s findings?

Erika Varangouli: So, again, with Mark, as we addressed it in a sense of, you know, as we were preparing for the study as well and analyzing all the data, the whole conversation was and still is around SGE and whether Google is launching it.

Will our traffic tank? No, it’s a great opportunity.

So at some points we were even doubting ourselves. It’s like, should we do it now? What if we launch in the next day? SGE launches everywhere. What’s going on?

But the idea is that, you know, we’ve been in SEO for a while to know that even in the last 10 years, right, Google has gone through a massive transformation.

And when we started, for example, I started in 2012, many things were very different. I haven’t seen the roots though of what good SEO, what effective SEO is like changing dramatically.

What I’ve seen, my take on it is actually Google becoming more capable of going towards a model that what they preach is actually what they can support. And I know there will be a lot of like, but look at the SERPs now, but look at what’s happening with spam. But I feel like this is probably what’s been going on because of AI, like entering everyone’s laptops and everyone just prompting someone to write something. And it does go through that.

So my take on it is that what Google is telling us they want to achieve and what makes sense for them to want to achieve as well, by the way, is to create experiences for users that really satisfy them, right?

They want to keep users, keep using their products, right? So the more we go to personalization, the more we go towards these things. But what is the idea behind doing SEO well?

The idea as well is that you offer such a great experience that people will find you and then engage with you and then do what you expect them to do, right? So that main focus on offering an experience for a user, the information they’re looking for, not pissing them off, waiting for something to load for three hours, knowing where to go to convert or trusting that you’re not going to steal their money and run off.

Those things, I don’t see them changing. AI in that sense, language models and generative AI and how it is used has been around for a while. We were perceiving it less maybe as even daily users of it, but in reality has informed the things we see for years now and their development.

So I don’t see that becoming irrelevant as we go ahead. I don’t see like, okay, SGE launching tomorrow and all of a sudden you don’t have to have good content. Like that’s not the case. And I think that’s the basics and the principles we have to look at. And that’s how I look at the study as well.

Garrett Sussman: Yes, there are some factors that are very specific and more like minor, but the principles of like, okay, we have some large areas in SEO that affect performance.

So you have content, you have backlinks, you have user experience, all of those things remain.

Last question I got to ask, and I agree with you about the AI stuff, like the fundamentals still change and it’s really great to pull it back to Google’s goal of serving the user the best experience to answering their search intent.

You are the head of SEO branding.

Brand is a big conversation. The DOJ that I mentioned lawsuit where click through rates and brand affinity seem to benefit larger brands even if their content is the best. Your personal opinion, do you think recognizable brand should trump the best content? Or do you think that if it’s the best content that serves the user intent, in theory, Google should find a way to surface that over the most recognizable brand?

Erika Varangouli: I hate to be Google nowadays. Okay, so here’s my personal opinion. I’m not speaking for anyone else or saying someone has said it and it’s official.

My personal opinion is that it’s very hard for a search engine to achieve that level of excellence with so much content out there and so many signals from different sources to achieve that level of that almost complete objectivity based on that content is great, it should do better. But here’s the plot twist for me.

Having a big brand doesn’t happen by accident. Having a brand is not just a result of an Erika somewhere coming up with funny messaging that people love and that kind of makes you a brand.

Let’s consider websites like Amazon, for example, or let’s go with Airbnb. You can look at it from many perspectives. And it’s like, okay, they have amazing content on social or they have this, they have that.

But look at Amazon. Their content is not the best on product pages. But what I get as a user and that is consistent through the year, so consistency matters, is I have a big index of products. So as a consumer, that matters. Like, if I go to a shop and I find three products versus being able to compare 300, I personally feel lost after a bit, but many times that has helped me.

And then I have all this trust, you know, where are they? Is someone stealing my money? The amount of times I’ve found something really appealing online?

And then my worry is that is someone in another place trying to scam me to pay them money and never send what I’m intending to buy.

So all of these things are collective signals that build a brand. I would argue that yes, if you’re a new company, you haven’t had the time to build a brand yet, right? So you have to over-rely on other things that you can control more imminently. But over time, it’s almost for me, it doesn’t make any sense not to have a reputation.

If brand is your reputation, like what others say about what they think about you, it’s kind of weird if after years of doing something and doing it well, you don’t have that kind of reputation. So I would think it’s impossible to get to that level.

Well, you know, I’m a small retailer looking Amazon in the eye. And maybe I shouldn’t outrank them just because I have better content on my site, because the element of trust with users goes beyond just, you know, a great landing page. That is such a fantastic answer. I’m with you. And it’s very compelling.

And obviously, when you bring in something like e-commerce, that’s a really great example of where the trust signals matter so much more than the content and just highlights all the different business models, all the different listening on SEO.

Garrett Sussman: Erika, this has been a pleasure. I really highly recommend anyone who’s had a chance to dive into the ranking factor study, it is free, it’s available, there is a link to it in the description notes of the show. And obviously, look at it with a critical eye with critical thinking, but use it to potentially build your own hypotheses.

Is there any parting words about the ranking factor study before we dive into the rapid-fire rankings?

Erika Varangouli: No, just read it with an open mind and use it to build hypotheses review your things you felt strongly about or not at all and just test new things.

Garrett Sussman: I love it. Okay, so time for some rapid-fire rankings. I’m going to put the music on. Are you ready for this, Erica? I don’t know. Okay.

No pressure, quick answers. Here we go.

Diving in rank your best SEO or marketing win.

Erika Varangouli: Rank my best?

Garrett Sussman: Yep, your best SEO or marketing win.

Erika Varangouli: Okay, growing organic traffic to the SEMrush blog in the first year by over 600%.

Garrett Sussman: That’s amazing. Okay, you are allowed to say Semrush, but rank your top three SEO tools.

Erika Varangouli: Okay, thank God I’m allowed to say Semrush. So I will say Semrush. Google Search Console counts, right? It counts. Okay. And I really like Ahrefs as well.

Garrett Sussman: Oh, that’s a great product. A great tool. Rank your best SEO trick or tactic.

Erika Varangouli: Okay. I don’t have tricks. Like my only trick is to be very, very dubious and very skeptical when people sell a trick or the only tactic you need to be successful.

Garrett Sussman: I love that. Right. Okay. What’s the biggest scam in SEO that you’ve recently seen?

Erika Varangouli: Okay. It’s not recently, right? So I’m sorry, Garrett, I’m breaking the rules. But I really, really dislike when people share success stories out of context.

So, you know, all these fancy graphs of zero to 100 million search traffic in 12 months with only this one trick. So this is something that I consider dangerous and not honest. And this is why I really dislike it. I agree. And we see this so often.

Garrett Sussman: Okay. Rank what you love most about SEO.

Erika Varangouli: That you have to be equally analytical and creative and curious to be good at it.

Garrett Sussman: That’s so good. That’s well said too. Rank your best learning SEO resource. You can say Semrush. You guys have a great academy.

Erika Varangouli: No, but like you’re killing me. Like it cannot be one. Can you be honest if it’s only one…

We would not know as much as we do without Barry Schwartz, you know, just continuously reporting and putting information out there and keeping us all in the loop with what is going on.

It’s ridiculous how much he publishes, and Aleyda is fantastic.

Learning SEO is an amazing site. Now, an even more unfair question is to rank the top one to three SEOs or marketers that you most admire or look up to.

Okay, I’ll put Aleyda again first. I’ve followed her for years. I happen to also know her now a bit more. And I think she’s amazing. And she is pushing the industry forward and that collective level of knowledge. I really, really admire Mark Williams-Cook and all the information and the knowledge he shares.

Also bizarrely, I’ve never disagreed with him on something, which is very weird for me.

And if I can bring back someone who was paramount to me and my learning and my growing but is no longer around was Bill Slawski. But I think he made us all better. And I still find myself going back to his content to find things that are still relevant and useful for me.

Garrett Sussman: Bill, that’s such a good call. I can’t even imagine how much fun, as much as fun as he has, would he be having over the past year of what was happening with Google?

Erika Varangouli: I’ve thought about it so many times. I was like, what would Bill say? What would he write right now? So yeah. He’d be writing so much more. Anyway, so sad. Anyway.

Garrett Sussman: Okay. This is the final one. What’s your generation AI hot take?

Erika Varangouli: Okay. So get ready for that. It’s that we’re not training AI; it’s training us. It has already become much smarter than we are. And it’s just either having a laugh or training us for something that we cannot even understand yet.

Garrett Sussman: I completely agree. I actually feel like in a lot of ways, SEO is the same way. The way we search, we were trained by Google, not training Google, but maybe that’s changing. And then finally, rank your number one cause or charity that you want to promote.

Erika Varangouli: Okay. Thank you. Yes. I want to promote Médecins Sans Frontières, which is Doctors Without Borders, essentially. That is quite dear to me for a number of reasons. So yeah.

Garrett Sussman: That’s a great cause. This has been such an amazing conversation, Erika. I really appreciate your transparency and the thought and intention that you put into your work doing a fantastic job at Semrush. If people do want to get in touch with you, the best place is on LinkedIn.

Erika Varangouli: So just, you know, with my name, they will find me and I live there nowadays a lot more. I’m also on Twitter because you know, I would feel like a fake SEO if I wasn’t on Twitter, but I’m not as good with Twitter nowadays. Like I’m, I’m, I’m kind of trying to stay away, but many people don’t stay away from it. So I can’t afford to.

Garrett Sussman: Hey, that’s what community is all about is you got to be where the community is. I appreciate it. Thank you so much for joining me.

Erika Varangouli: This has been awesome. Thank you, Garrett. It’s been my honor. Thank you.

Garrett Sussman: My name is Garrett Sussman of iPullRank. Thanks for checking out another episode of the iPullRank Rankable podcast. We will catch you next week. See ya.