Garrett Sussman: Watching the NBA playoffs. Mr. Will Critchlow is the founder and CEO of Search Pilot. He founded Distilled. He founded SearchLove Conference. He is leading the way in agile changes and innovation solutions for enterprise level websites. Let’s get to it. Presenting Enterprise Scale ECommerce and Retail SEO Lessons from thousands of experiments. Please welcome Will Critchlow.

Will Critchlow: I am hoping, I’m hoping that you all are ready for a presentation that isn’t all about AI. Because even the bit of my presentation that’s about AI is mainly not about AI. So we’re going be talking about some tactics. That’s kind of generally the popular bit of what people want from our data and our work. The what to do about AI, which is mainly not about AI. And some stuff on the business side around how we can get some great results, but also get the credit that we deserve, get the budgets that we’re looking for, and prove the value of our work to the organisations that we’re talking about here today. And I hope it’s going be a really optimistic conversation because I’m optimistic. I’m talking about ways that we can have agency, that we can get results that we’re excited about, that we can get those budgets, and that we can get really great outcomes. But I want to start with the maybe less optimistic end of things. The worst year of my business career. And I hope you don’t mind going with me on this little dark journey. I’ve never told this story in large scale in public before. So I hope you’ll be kind to me.

It’s more than ten years ago. We’re about 2014. And literally, will get on a plane, fly to New York City, get on stage at an SEO conference instead of going to therapy. So 2014, we’d opened up an office here in the city a few years previously. And this was the toughest year. In the intro, Garrett talked about how, before running SearchPilot, I ran a company called Distilled. Distilled was an SEO agency, and by this point had been going for almost 10 years. And we’d had 10 solid years of growing every year. In fact, I thought that’s how business worked. You just get bigger every year, just natural growth. And 2014 was a year where we were struggling with some of our biggest clients, struggling to get some of the results that we wanted, that we expected of ourselves, the things that they expected, and struggling to prove the value of some of that work. And it was the first down year in revenue in Distilled’s history. And revenue is one thing, but if you talk to anybody about what’s the most important thing in their business, they will tell you it’s the team. And we started that year with 51 people in our global team. During the course of that year, 26 of those people decided they didn’t want to work for me anymore. More than half the company. Sure, I screwed up a lot of things. During the course of that year, I think we hired 19 people. So we ended up just a little bit smaller than we’d started. And the energy came back. But the reason I’m telling this story is all of that soul searching, all of that trying to figure out how can we get better results, how can we prove the value of the work that we’re doing, how can we demonstrate that we’re getting the results that we’re hoping for, that was the genesis of what became SearchPilot. So this darkest year was when we formed the R & D team.

Some of the guys who I still work with went over into the R & D team, including my co-founder, and started building the platform that would become SearchPilot, which let us run enterprise scale SEO testing. And this was a confluence of a few things coming together. The decade of consulting experience, the fact that at that point AWS had got to the point that we could actually do some of this stuff technically. And we had this amazing platform. Really quite quickly, you get the first version of these things. And everything was easy from that point onwards. We do this not because it’s easy. We thought it would be easy. It was not easy from this point onwards. It was a lot of grind, a lot of enterprise sales, a lot of RFPs, a lot of ISO 27,001, SOC 2 compliance, and PCI and HIPAA, and all kinds of other exciting things. But the point is that over the next decade since then, we’ve got to work with a whole bunch of really exciting major retailers in particular, but a whole bunch of great brands. Some of them we can these multibillion dollar, multibillion pound e-commerce sites. And some of them we can name. We work with Marks and Spencer, iconic British brand. We work with Petco, for example. And the reason I’m bringing this up is what I want to talk about today, I love stuff that comes from research. I love stuff that comes from surveys. But what we’re presenting is from thousands of tests run with these folks. And I’m really excited be able to move the needle at that scale. And I’m hoping that even if your site’s much smaller than that, or you work with much smaller clients and customers, that there’s hopefully some interesting, useful tips that I can give.

Anyone here collect baseball cards when they were a kid? Or still. That’s fine. No judgment. Well, we’ve got the retail SEO testing pack that you can and I want to turn this into physical swag. So this is popular enough. We’ll get these things printed up in like little foil. You can rip it open and what. But for now, it’s digital and virtual. So there’s a QR code, or you go to bit. Lytesting pack. And there’s going to be a couple of stages through my presentation today where I pull up QR codes And you can obviously snap pictures of those. Feel free to share them. They all open up on links of our resources that will be helpful, useful to you. I will also share the slides. I know they’re going out in the Slack. And if you follow me, I’m willcritchlow on all the social places. I’ll put it out there as well. 

Okay. So 20 cards, 15 winners guaranteed in every pack. And this is far too small to read. I’m not running through this, but this is just the contents page. So in the tactical side, I wanted to talk to a few of what I think are some of the most interesting, some of the most confounding, some of the contradictory results that we’ve had and that I’ve certainly found interesting. So this first one is an interesting one. SEO text. I love things that are called SEO text. Anybody have SEO text on their website? Bonus marks if it’s in div class equals seotext. Right? That’s just yeah, let’s just make this really clear what we’re doing here. It’s probably nice and small, probably in gray. No human has ever read this stuff. We tested removing some of this terrible content from the PLP, from the listing pages on one of our e-commerce customer sites, and saw an uplift from removing that content. In other words, it was hurting the performance of that website. This is not universally true. Your SEO text may be better than theirs. Your competitor’s SEO text may be worse than their competitor’s SEO text. Who knows? But this was a positive result. I included one that was just a very basic title tag test because I think title tags are still the Halliburton SEO tests. That’s underrated for anybody who has been following the MBA this year. No, wait. He’s the overrated one, right? Anyway, moving on. This was just a basic title tag test, a 24% increase in traffic to these pages by removing the comparison reference. And titles are way underrated because they’re one of the very few bits of your website that affect both rankings and click through rate in the search results. And for those of you who are as old as I am, you’ll remember when folks on the ad side used to have it was like a whole part of their job was optimizing adverts for click through rate. Computers do it these days, I think. But writing those adverts was like a whole job. And on the SEO side, we’ve criminally underinvested in having really great and compelling snippets.

Not all of your content is bad. So we’ve repeatedly heard from lots of places that the layout of the page shouldn’t matter whether something is expanded or not by default on mobile. It’s one of those things that apparently isn’t a ranking vector. Well, we’ve repeatedly run tests that show that it does matter. And in fact, on mobile, having content exposed by default, in other words, not collapsed into an accordion or tabs or whatever else, exposing it by default on page load can result in this was a 7.5% uplift in traffic exposing that content. For the geeky SEO types in the room, a canonical tag test, 22% increase from changing the way the canonicalization worked across a bunch of PDPs. So there’s more of these. And you can take your time with them. I’ll just pull out this one. This one’s an interesting one, where on one particular website, we saw a significant traffic increase from removing meta descriptions entirely. And this was surprising and confounding because what Google was doing when there was no meta description was writing its own, and they were terrible. But it turned out they were better than the ones we thought were good because Google has an unfair advantage. We can only write one meta description per page. They can show a different snippet for every query that that page ranks for, and it can be highlighted in bold related to those query terms. So even the kind of not very well written, not very grammatically correct stuff could, in fact, outperform because it was more topically relevant, because it was pulling the bit of the page that was most important. And it turned out that having no meta description forced them to present that. So anyway, tons more of them.

That’s the QR code again, if you want it. That’s the tactics bit. I promised a little bit of AI that isn’t really AI. Well, again, thinking back over that time period that I was talking about of going all the way through from starting Distilled in 2005 through the bad year, that first bit, one thing that happened in the middle of that time is the incredibly fast growth of social media. Do you remember 2007, 2008? Launched the Facebook news feed. Twitter is taking off everywhere. Everybody’s getting a Twitter account. And there was this kind of narrative. This is my lovely hand drawn description of what everybody thought was going to happen, which was search was going to die. Social was going to take over. People weren’t going to Google best Thai restaurant in Midtown. They were going to tweet their friends and say, where shall I go to eat? And this is what was going to happen. Social was going to take off and search was going to crash. This was one of the many predicted deaths of search. Of course, that’s not what happened. Search continued to grow fast and social was incremental on top of that.

And this is my bull case for what’s happening with AI. I think volumes are queries, broadly defined, query volume is exploding, And some of those things are searches. And more and more of those searches are still going to be needed to be fulfilled by things that look more like what we’re used to in a search context. I go deeper into this in a separate presentation, which you can get at SEO Week-AI-SEO. A couple of things that I found interesting as I was pulling this bit of the presentation together. I’m thinking through what do we know about query volumes? What’s Google saying? What do we think we can trust or not? And there’s a couple of quotes that I present these with initially without comment. Obviously, on stage. I’m going to comment. But Sundar, on the Hard Forked podcast, saying that year over year, they’re sending more traffic outside the Google ecosystem than ever before. I don’t know if believe it, but he said that. Liz Reid, head of Google Search, said we see that the links included in AI Overviews get more clicks than ten blue links. Who laughed? Come on! This is a very serious executive giving an on the record statement. 

I suspect there’s various ways that these things are technically true and this is an e-commerce talk. And e-commerce is the area where they are probably most true and it’s most exciting because query volumes are exploding and does this all mean Google and e-commerce are both very, very excited about high commercial intent clicks. Right? That’s where e-commerce lives, but it’s also those queries are where Google makes all of their billions. And neither ChatGPT nor Google nor Perplexity nor any of the others are actually going to ship you stuff. And so at some stage in this technological whirlwind, users are still going to need to reach some variation of a PDP, a product page. And shout out to Aleyda’s talk this morning. We obviously coordinated, but she set me up nicely. She hovered all of that stuff that was amazing. If you didn’t see her talk, get her slides around changes to the search result page, all the stuff around Merchant Center, all of those evolution of Google wanting to be your PLP and the filters and all that kind stuff. I’m not going into any of that, but I’m totally aligned that Aleyda covered that wonderfully earlier on.

And so what should we do? What should e-commerce sites do about AI right now? AI as a discovery channel, mean, not like you can use it in your business, wherever you like. Right now, channels as they exist. Next, be accessible. Finally, extend your experiments there. What do I mean? Right now, channels as they exist. What is your biggest AI channel right now? It’s Google. Google is your biggest AI channel, still. And I don’t just mean AI Overviews. All of the bits of Google that are AI powered, all the regular ranking stuff that’s AI powered. If we’re talking about AI powered discovery, we are overwhelmingly talking about Google. And so when your executives are asking for your AI discovery strategy, your answer should be, here’s our Google strategy. You probably want to change the title to our AI strategy because they like that kind of thing. That’s a pro tip about dealing with executives. 

I do have a couple of specific AI things that I think are very important around accessibility. Again, throwback to 2005 SEO. Robots.txt. was robots.txt a thing in 2005? I can’t remember. Anyway, I love robots.txt. Make sure you’re not blocking the LLM crawlers. This is an e-commerce specific thing. It’s debatable in media or some other places, but if you’re in e-com, much like we just heard, you want the LLMs to learn about you from you.  So you definitely want to be there. I think that’s probably unarguable, but if anybody wants to argue about it, we can have that conversation later. Definitely 2005 SEO. No JavaScript, please. In fact, just generally. Can we just get rid of Port of JavaScript. HTML is where it’s at. But the major LLM crawlers do not use JavaScript. And so if you require JavaScript to render the content on your website, you’re not going to be visible in the LLMs. And eventually, we at SearchPilot and you as e-commerce, either consultants or marketers, are going to want to extend your experiments to them. I believe this is going to be possible. 

Bear with me here. A little convoluted argument. So Jeff Bezos, you probably know him. What he said about Amazon is we’re taking the bet that even over long time periods, users are going to want low prices, fast delivery, a vast selection. That is not going to change. And the big bets that the massive organizations take is what are the things that are not going to change? And I think Google’s equivalent is freshness, completeness, and relevance. Nobody says, I wish I could get last week’s news. Nobody says, can you just miss out some of the sources because I’m not interested in them? And nobody says, show me the crap bit. And where some of the LLM stuff is falling down is specifically in the freshness and completeness part. And I think if we’re going to get to the point that those channels are genuinely competing with Google, they’re showing up in your analytics, and you’re getting sales often at scale, at volume, then they’re also going to have to crack some of this stuff. And if they’re fresh, and if they’re complete, you can run experiments against them in much the same way that we run SEO experiments. And so eventually, we’re going to do that. So my prediction is that by the time AI recommendations get to a scale that they’re meaningful to the business, they will also be testable for these reasons. They’re going to continue sending people to product pages. The need for freshness means that we’ll be able to test to learn what works best.

Okay. That’s kind of the AI bit. So in summary, what I’m saying is essentially for the people to buy from you, the machines have to recommend you. But that’s been the case. That’s all of our job. That’s just Google. And these machine preferences are becoming more human-like. And so we need this hypothesis driven testing. And I think this is going to stay true. I haven’t had a chance to chat to the speaker yesterday who was talking about you have this one VIP customer, and it’s the LLM crawler because my view is actually those preferences are going to stay human-like. So my agent is going to have different preferences to your agent and we’re going to need to test in that world. Okay. But anyway, the end of the AI bit. So a little diversion before we get into the final part of the presentation, which is a section I’m calling stuff CFOs say. And one of the things that’s been quite interesting to me, so I was never brave enough to really specialize in the Distilled era. Some agencies are really great at saying, we specialize in this particular area. SearchPilot’s been much more natural to specialize. So we’ve specialized really closely in e-commerce, for example. And that’s why we have all this data. And one of the unexpected benefits of that is being able to go deeper into those customer organisations. So I have met more CFOs, more non-SEO marketers, the CMOs and the like, more product people, more engineers from e-commerce companies. I’ve met more of them in the last six months than the previous 20 years. And the finance conversations have been some of the most interesting, honestly. Hence the segment. Stuff CFOs say. One finance team told me, we’re still seeing double digit year over year cost per click inflation. Our paid search is getting more and more expensive faster than everything else in the world. And you know what that means? That stops a marketing plan working real quick. You extrapolate that 15% a year inflation out two years, three years, and your marketing plan doesn’t stack up anymore. That was out of the mouth of a finance team, a major e-commerce player. One of them told me 60% of our new customers come from organic search, but we spend 80% of our budget on paid search. Finally, they told me that. They told me that. I was like, I know. I know. 

Mike was talking about why is this? It was you, I think. Yeah, you put out a thing. Why is this? Anyway, finally, I’ve met the people who hold the purse strings and they’re telling me this and they’re saying it’s a problem. Said I know it’s a problem. Let’s do something about it. Give me some of that budget. One of them told me, PPC overspent by an amount larger than the SEO budget. No, actually, no, no. It was worse than this. PPC overspent last month by the entire annual SEO budget. Overspent. I was like, you know what? I have a great plan. I can help you. I can help you with this. From the finance team, organic search is my favorite channel. Not if you do it well. I’m not arguing that it should be free or that it should be cheap. Quite the opposite. But I am saying it can be hugely efficient and you’re building an asset that is going to perform for you over the coming years.

But I heard this from them. For all of these reasons, SEO has to be a big part of the plan. So the final eight minutes today is going to be all aligned with the stuff CFOs say. How do we get better at talking business language? How do we position, pitch, describe, and do our work in a way that it’s legible to the whole business and that it gets the results that they really want to see. So I’ve called this turning SEO into a performance channel. Again, I dive deeper on this at this QR code, SEO week performance on the Bitly link. Okay. What do I mean by performance channel? So a marketing leader looks at a performance channel when they can understand how it works. 

They have a model of how the channel functions and operates. They can understand and prove the value of that channel. And most crucially, they can put more budget in and get more results out. This is fundamentally my main understanding of why paid search is such an amazing channel and why it gets such huge investment, is that lever, right? Or that dial. You put more in and you get more. Obviously, I’m oversimplifying tons around the profitability question and how you actually operationalize that, but that is the fundamental of it. So how do we turn SEO into a performance channel? And I think it really comes down to this question. Do you have a good answer to what you would do with an extra million dollars in your budget? Good. Whoever said that. Great job. Yeah. I hope so, right? I think those finance conversations that I was having, they have actually genuinely led to people starting to say we should be testing what we can do if we really invest much more in these channels. And I’m deliberately talking about millions of dollars here because our budget should be in that scale. This is not $100 here for a tool, $100 there for a tool. Most large organizations are already spending millions, whether you know it or not. Because by the time you look at your headcount, by the time you look at the demands you make on engineering and product, we’re into the millions, if not tens of millions, straight away.

So I think we need to have great answers to these questions. And part of it is about understanding the channel. So this is why I’m still banging on about AI stuff is cool. I’m very excited. But Google, people, organic search, this is anonymized, aggregated, real e-commerce data. Biggest channel, probably no surprise. You don’t get new customers very often from direct and email. Direct, they have to know who you are. Email, they have to have already got on your email list. And on this particular set of anonymized data, about 60% of organic search is unbranded. In other words, people searching for the thing you offer rather than just specifically for you. And that bit of unbranded search is bigger than any other new customer acquisition channels. I don’t think I need to evangelize this to this room, probably, but it’s surprising how much we still do need to evangelise this within the organisations that we work for and that we represent. Showing them the importance of this channel still today. Yes, the AI stuff is exciting, but biggest channel. And part of building that mental model in e-commerce is saying, where does this traffic land on our website? Because it’s mainly not the home page. On this data set, 4% of home page traffic was unbranded, 70% of PLP traffic, and 85% of PDP traffic was unbranded. Again, you probably know this. You’d guess these numbers. But there are marketing executives in your organisation who haven’t internalised how this dynamic works and how this plays out. 

And so we take this kind of thinking, this kind of model, and we say, okay, we want more of this, the unbranded organic search traffic, to our scalable site sections, our PLBs, our PDBs. And we’re going to build a model that explains to you how we’re going to make that investment and models some variations of the results that we’re going to hope to get from that. This sounds terrifying to most SEOs. We’re going to build a model that says how well we’re going to do. It’s good. But I think it’s really important. And a key part of this is understanding the baseline. So I called this SEO depreciation. I talked about you’re building an asset in organic search that lasts for years. But assets depreciate. They get less valuable over time if you don’t maintain them. And again, I dive much deeper in this in the video that you can get at the QR code. But the basic question is, what do you think would happen if you just stopped doing SEO? No SEO team, no agency, no data, no tools, no effort. It would go down, right? Your organic search traffic would go down. I don’t think that’s controversial, but it’s amazing how many models I’ve seen that assume flat year over year baseline. Right? If you’re doing that, you’re doing it wrong. It’s not a flat year over year baseline. You have to invest to stay flat and then invest more to grow. And I’ve validated this with some real data. We’ve got a couple of examples that we’ve seen historically where we happen to know teams were cut, investment was cut off, and then we’ve been able to measure the performance as a result. And sure enough, it might stay there for a year, but then it starts tailing off. I think roughly you should be modeling this as something like a 10% to 20% decline year over year if you did nothing. I published some of this data and some folks shared some other data points with me that kind of backed this up. So this was another example, obviously anonymized -13% a year on average. Another example that was -12% a year on average. So most of these backs up this data point that I’ve seen, like 10% to 20% decline.

And so that should be the starting point for your models. Of course, I’m not saying this is a bad thing for SEO. If you stop spending on paid search, it just turns off. But it should be factored into your model. Now, our game is all about SEO testing. And it’s also tempting to go, every winner we get just compounds up and to the right, hit the sky. I don’t think that’s fair either. Winning changes decay as well. Your competitors copy you, the market shifts, Google makes changes, whatever it might be. All those winning tests you get don’t last forever. So we can’t just go, yay, add 10% lift to 12% lift to 15% lift and we’re hitting the stratosphere. What you have to do is you have to say, Okay, they decay as well. And we built this model of what the winning tests add up to. And so you build a financial model that says, Okay, well we would have been down x% if we did nothing. We’re going to run these tests. They’re going to decay over time. And we add that up and here’s what we think we could achieve. And that’s your year over year number. That’s the number you’re shooting for. Sometimes you’re told that, right? Can you try and hit x% uplift? But your ROI is not that number. Your ROI is how much better you would do with the investment versus without the investment, not this year versus last year.

Crucial. If you take one thing away, start building your models like this, I think. So anyway, I’ve got to hammer through the end of it. So this is the model. I’m not going to talk about the proof of value. That’s what we do at SearchPilot. If you want to know more about check out at searchpilot[.]com how we run these tests. But we get all these data points. And the final part is getting the more budget to get more results. And I’m going to argue that this is basically about saying individual test results, if we could predict them, we wouldn’t need to run the tests. So the way to the results is about managing not the outcome, but the effort and the inputs, which are how many tests are we running, how good are our hypotheses, and how well are we measuring the results that come together. Put those things all intertwined, and we get the model that I was talking about. We get a performance channel where we have a model of how it works, we can prove its value, and we can put more in and get more out.

I’m at time. I’m going to tweet the link and so forth, but I expect you might want this thing again, so I’ll leave that up in the background while we wrap up. Thanks, everyone.