Mike King: With 15+ years of experience growing startups in global enterprises. She is a program leader at Northwestern University, mentoring future marketing leaders. And she’s presenting hybrid engine optimization, a crawler driven approach to maximizing search and AI visibility. Please welcome Jori Ford.

Jori Ford: Mostly because people are gonna wonder, why did I call it hybrid engine optimization? And I’m gonna be honest with all of you about why I did that instead of calling it GEO. I wanted all of you to be able to go home and say, went to the most epic conference ever, NYC, SEO, where I learned about SEO, AIO, LLMO, GEO, and then HEO, plus a little bit of EEAT sprinkled on top. That’s why I did it. But that’s not really why I did it. Jeff kind of set me up for success here because I think that we’ve looked at how we frame things as SEO and search and AI. And I don’t think that’s like one or the other kind of situation here. I think that we really have to look at it in the landscape of how does it impact our business. And from a landscape of how it impacts our business, many of us are still working in search. It’s not going away right now. So you guys remember when it was just SEO and it felt easy? Before like AI came in and started being disruptive to everything that we were doing? Our minds were blown. We’re like AI. And I’m like, don’t be afraid. It’s going to be okay. We’ve got new prospects on the horizon. And ultimately, hybrid just means it doesn’t have to be that hard. Like, we can figure out how to create balance. Because guess what? SEOs have been doing this for a very long time. I think Mike talked about the evolution of time. But I want you guys to think about this in your jobs and your careers. Right? There’s been a disruption across search every, what, three to five years. Have you lost your jobs? No. We’ve risen to the challenge and we’ve changed. We’ve adapted, we’ve grown, so we will continue to evolve. We test the crap out of everything and we evolve with it. So, I don’t know if you know who this guy is, but he’s been telling us that the whole time. Right? We’re going to change and evolve. It’s all going to move, and relevance is our new realm. I don’t know why, but he keeps saying that. So, we have to reframe our thinking on what works. I had that slide before he went through everything he did, you guys. I was like, oh, no. He’s going to tell.

So, things I want to go about today. I want to talk about the new landscape as it is because we’re going to talk a lot about the future later, and we’re going to talk about the things we can do. But, let’s talk about, like, right now where we’re at. We’ll talk about crawl logs because I think of myself as like the crawler queen. Every single time I talk, people are like, Oh, Jori, not the crawl logs again. But, I actually think they’re one of the most vital tools we have, so that’s what I’m going to talk about. Something that I hear a lot is, but what if you can’t get crawl logs? I’m going to tell you how to get around all the developers telling you no. And we’ll talk about building insights and dashboards that help evolve what we’re doing so that people understand how you’re changing and challenging the landscape and how you are adapting to AI right now in your roles. And then, we’ll talk about strategy. How do we implement these things? So, it’s a brave new world. The landscape is changing. AI is taking I’m just playing. So, really, here are the things that have changed. Right? We think of the journey as very linear or at least we did. I think a lot of us as marketers have been like, hey, you’d start here and it’s part of the funnel. And search has been part of that funnel, so we’ve been part of the solution for every problem across every step. But it’s changing, right?

We all know the journey is never ending. It’s cyclical. Right? Things keep evolving. People keep needing to learn. So we have to keep being involved. And now, we shove some AI in there. And so what does that do for us? Really, it just changes queries into questions. We go from queries in search to questions in search and agents because there’s more information for people to take in and distill. It shouldn’t be that bad. Right? Again, reframing how we think. So we know that search and discovery is going to change. Organic search is not just about position, it’s not about rankings anymore, it’s about presence. So, we have to adapt to our marketers within because for some reason, I don’t know why, but people don’t think SEOs are marketers. And I’m talking in the technical track and I still know I’m in marketing. So, it’s about presence in the AI-generated summary. And this is the piece I want you guys to take away. Right? If you’re not part of the journey, if you’re not part of the conversation, you’re not there. They exploded keyword rankings. So we run around and we’re like, what are we going to do? That’s what we that was our bread and butter. Right? Every tool that’s here right now has had to adapt to that. We can’t do keyword rankings. That’s okay. The surface changed a lot. That’s okay. We don’t have all of our eggs in one basket anymore, and that’s okay.

I’m not going to go too much into this, but when you think about how do you know you’re part of the journey, there’s lots of tools. People are going to talk about them all day, so I’m not going go into it. But, like, a little bit, we could just go into our analytics. We can do that. That’s easy. Look for referral traffic. We can go to Also Ask, dump some queries in, get a bunch of stuff out, and then if you want, you guys can steal my little AI mention tracker because I’m cheap, so I don’t pay for all the really fancy tools. No, I’m lying. I paid a little bit here because I use the GPT plugin and everything like that. But when you dump it in, what you’ll get is a quick search of whether or not you’re visible. That’s one of the questions you’re going to get asked as SEOs. Am I visible in the AI agents? And it doesn’t actually matter if it’s not good for your business, but if you want to answer them, use my tool. If you don’t want to use my tool and you want to pay a little money, not a lot of money, use something like Use Haul. If you want to be really sophisticated, there’s several tools. DemandSphere is one of them. 

But let’s get into the crawl logs because that’s that’s my bread and butter. Right? So, many of you have asked for crawl logs. What do you do if you get them? That’s like the weird part. Right? Because when do we ever get them? But, if we got them, well, you might have to parse through like 226 bots because in March, Cloudflare did this really awesome thing where they analyzed all the data that we give them and filter through them, and they identified 226 bots and crawlers. That’s a lot of work for you guys to have to filter through if you get into the crawl logs. But here’s the funny part. We shouldn’t care about that. 226 bots does not matter to you guys or your businesses. What might matter is the ones that are actually visiting your site. And the thing is is Cloudflare is not going to actually give you the user agents without you paying them. I don’t know if you guys heard them a little bit cheap, but the guys over at Momentic, Tyler, if you’re here, I know. I can’t look dead at you and say it. But amazingly good list because all the agents are listed here. So if you want to figure out who you should be following or what’s actually going on, go over to this resource and it distills it down into 24 and they actually keep it up to date. So you don’t have to do a lot of the busy work because the agents is what matters, not just how many bots are out there. 

So, why do I care so much about crawl logs? Why is it the only thing I constantly talk about? It’s because we, as SEOs, have overindulged in the use of tools that are hypothetical sometimes. Right? Keyword rankings are hypothetical. Can you validate all of them? No. Search is personalized. So you can look at your tool and you can say, I think this is what’s happening, but you don’t really know. Logs tell you the truth about everything that is happening on your website. That’s why I say that it’s your real qual equity because when you look at log files, it’s actually what the bots cared about. They’re only crawling what they care about and you can now see exactly what that is. So, if you happen to get really nice developers who are willing to assist you in this, we make the mistake of over asking without actually giving direction. I say this because I used to be an engineer and people used to ask me for things and I’m like, you don’t even know what you’re asking for. Don’t do that. So, if you want access to the logs, you can make it simple. Ask them for access logs. That’s very specific. And then, you’re going to get a developer that says, what about security? What about these things? Hold your horses, just ask for timestamp, the user agent, the URL path. Let them understand that it’s not secure information we’re worried about because we only want the get methods. We only want the things that the crawlers want. We don’t care about the rest. 

So how do you distill this down into information? What are the pieces that you actually need? So the IP address, that one’s an actual GPT bot. The reason I say that if you get crawl logs, you want the IP addresses, those 226 crawlers that Cloudflare found out, they found them out because they did IP mining and reverse lookups. Because here’s the beautiful new world of AI that you’re not going to be used to. AI lies. AI doesn’t have any standards. They just want to get what they need. So guess what? You have 226 unknown bots traversing your planet and you don’t know who they are, unless you have their IP address. Guess where you can get that? Your log files. They have to give it to you. Timestamping your request lets you know some frequency. HTTP method, you guys can read the slide, I don’t have to read it to you. You’re all smart. But what can you get with that? Every marketer asks these questions. These are normal day to day questions. But did you know that the log files can answer most of these day to days in a way that you don’t expect? Who? Which user agent? Which bots are visiting you? Because if your business tells you you need to prioritize bots, I don’t always say that you should listen to them. But if you decide to do that, find out if they’re actually visiting. Did you impact their visits? What pages were important? Are you optimizing the right things so that they can get there? Can you link from the pages that they actually are accessing? When? How frequently are you getting crawled? If you’re not getting crawled that much, you’re probably not that important to the agent. So, you’ve got to either figure out how to be more important or explain that you’re not that important, and that’s not a channel that you need to worry about. Where is my audience looking for me? I think that’s huge. Right? That’s part of the relevance, it’s part of the change because we don’t give these tidbits where we can walk into a room and say, I’m in SEO and I know where your brand actually is. I know what’s happening. And then, how am I doing and why am I falling behind my competition? If you know that your competition shows up in ChatGPT and you know that ChatGPT does not visit you frequently or index any of your pages, you know how your competition is beating you. They’re using ChatGPT. But what if they’re not? Then why are you worried about it? So that’s how you would look at these IP logs if you’re looking at raw log files.

But I’m not going to do that to you guys. I love you. So what I’m going to teach you is how to get around the developers. You’re not going to necessarily get the IP addresses, but from a bot perspective, you can track them on your own. You can find out if they’re important before you ask the developer for access or before you give the push back that you might need not to need to give. So I’m going to show you how to set up Google Tag Manager, connect it to your GA4, and merge in some of your Search Console stuff. So these are things we’re all familiar with. Right? But sometimes it can get a little inundating. But it’s really not that hard. Three simple steps. That’s how easy it is for you guys to all track search engine bots and AI bots. That’s what makes it hybrid. Right? You get to see it all in the same place. You set up a variable, you set up a trigger, you set up a tag. I think that sounds simple. So here’s what you really need to do. Now, again, cheating, lazy, and Ilias gave you guys toolkits. I’ll give you tools too. That’s why we’re part of the technical track. This script, it’s a resource that I’ll give to you. Just take it. It’s already in a Google collab. But you’re going to set up your variable and set up this custom script. And that’s going to let people like GA know which bot’s coming. Then you’re going to set up a trigger. So you don’t just want to know which bots are coming, you want to know when they come and on what pages. You set up a trigger, and now you’re isolating around that. That regular expression seems really complicated. It’s not. It’s just separating pipes on the same names that are in your JavaScript file. Then, we’re going to set up the GA4 event tag because without the tag, none of this fires and you don’t collect anything. Once you set this up to match the parameters, that’s going to get passed right through to your GA4. Guess what now you have? 

Alright. Yeah. Yeah. That’s it. You guys got it. Got it. Okay. So you have two ways to track, you have two ways to understand the agents, and now you have these really, really pretty reports. You get to see the bot name. You understand the events being handled. You can now customize this for your own view. You know how often and frequently they’re visiting you. And if you want to know how I got this into a GA account, I actually built a Google Collab script that runs this for demos. The reason I do that is because you don’t want to work on some of this stuff in a production environment because there are other tags set up. So I will give you this as well so you can set up your own demo accounts. You can now parse and sort it out by page frequency. Oh my goodness. Look at all of that. The not set’s my fault because I didn’t set it on the day that I ran it. Just ignore it, Okay? But in this hypothetical situation, you can see which pages are the most important to this crawler. So now you’ve got log files and you’ve got reports. But don’t you want to take that from there to the pretty stuff that I made? Okay. Well, it’s not that hard to do this. We all have access to Looker. Looker is free. Right? So in our free Looker account, we can set up these bot distribution reports. Also, I’ll give you this free template, free of charge. It’s because I’m nice. Don’t worry. But easy things that you can see really quickly. Overlapping pages where you see high impression and high event counts from the bot. That’s your existing opportunity to evolve. Right? So stuff you already own that you could just do things to. Go optimize it. In this opportunity, high bots, low quicks, missed opportunities. It’s rocket science I’m creating up here. Alright? So, you go to that landing page, now you know what else you can optimize. 

You have a plan. So, what do you do with the plan, Jori? Nobody’s going to ask that? Fine. Okay. So, what you do is you go get shit done. And this is when you get hybrid about it. Because if you know that your business needs to prioritize it, or you are getting traffic from AI agents, you are getting traffic from search, and you need to evolve, you need to expand your traffic set, get more conversions, do whatever it is you need to do. One, search is still important. I’m not going to say that it’s not. The reason search is still important right now, not in the future because we should prepare for it, but right now, most of your jobs hinder on search. Results in search, results in AIO because that’s what your teams know. And until you educate them out of that mind thinking, you still have to do your jobs. So today, Google still needs to be a priority. Why? Because Google actually still has Gemini. It that’s an AI too. So if you want to show up in their AI agent, you should probably work on, you know, what they use in their index for their rag model. They’re grounded in it, so that’s just me. Log and tag all the important bots. I say important because not every one of them is important to your business. 

This is where we become more strategic. Only track what you need to track. Other things are obsolete and noise. That’s one of the problems we have. We are analysts at heart. Right? You want to get lost in the noise, lost in the sauce. And if you do that, you kind of lose your audience, you lose the business. Train the LLMs on you. People wonder how that happens. You can actually talk to the LLMs about you until they just show up. And somebody actually called me out on this. They’re like, are you sure that that can happen? Yes, I’m sure. I’ve literally done it. I typed in Aspen Dental over and over and over because my buddy told me that I couldn’t do it, and Aspen Dental showed up in the LLMs where it was not. And I just kept asking it questions about if I needed dentures, should I get Invisalign, where could I get those things, and it happened. Create answer assets for retrieval on your pages. What does that kind of look like? Well, clearly isolate your brand on the pages, create product definitions. These are things we have access to. It’s just content. It’s not big wow’s. FAQs, we did that for a long time. You guys remember the ugly FAQs at the very top and bullets and people are like, these SEOs are screwing up our content. Why are they doing that? Now, can bring it back and go, guys, I know you hated that idea I had, but if we want to show up in AI, you have to let me do that really crappy thing again. I’ll show you a prettier way to do it though. Named entity references. Your brand matters. We’ve all talked about how we need to shift the dynamic towards brand. Reference your brand, reference your features, reference your categories. And guess what they’ll realize? SEOs care about branding. SEOs care about content and how the business operates and we get to be part of the business discussions now. Submit to Perplexity Pro, it’ll just put you in the knowledge base. It’s pretty easy. And then, if you’re not getting crawled, make sure you do the basics. XML sitemap and your robot text files, get them crawled. 

Create answer assets. Remember I said I’ll show you a better way to not do the ugly FAQs? It’s up there, over there on the right. I can’t look that far. I’m going to look down here. But once you do that, you can also add high quality explainer blocks. Does this page look ugly to you? Does it look like it’s not useful? It doesn’t to me. It looks broken up. It looks like something I can synthesize in chunks because just like people have to analyze, scan and absorb information because we do this every day. Right? The Internet is like this, constant information overload. But when you get to a page where things are simple, broken up, chunked out, I don’t know, I’ve heard a lot of people talking about chunks of information, elements that get indexed, something about AI and how they relate to information, set it up the same way and the LLMs love you. Use bulleted summaries. It’s pretty easy. Step by step guides, glossaries, linkable anchors. 

All of these things are things we have access to that make it really simple. Because I think we overcomplicate things sometimes and we don’t need to. Open graph and meta descriptions, they’re getting extracted, leverage them. And you want to know why you should do this and listen to what I said? Don’t listen to me because I’m cool. Aleyda said so too. So listen, if Aleyda says it, I’m right then. Just remember that. Okay? Thank you guys for your time.