Mike King: Dan Petrovic, he said, can we do a skinny intro? So don’t need that. I do wanna say that Dan was so crucial and critical to the leaked document blog post that I wrote, and that’s had such big impacts on our company. So Dan, I really thank you for that. And presenting on Beyond Rank Tracking, Analyzing Brand Perception Through Language Model Association Networks, Give it up for Dan Petrovic.

Dan Petrovic: Alright. I’m glad you guys are nice and relaxed because I’ll do probably just the opposite. I’ll I’ll need a lot of open brains, a lot of attention because I I can’t afford you guys missing, like, ten seconds looking at your phone during this. So the original the original presentation was about LLM visibility tracking, and I kinda Trojan horsed my way into a lot more, that’s gonna probably, turn a lot of cogs in the audience’s heads. So I’ll I’ll cover the the basics, the bread and butter of it, and then we’ll get into the more advanced stuff, which is what you’d expect. Those who have seen me speak before, are used to that sort of thing.

So step one in our process is to define the brand or product or service we want to track the visibility for. So once you do that, you define your primary entities, your keywords that you want to track, different concepts, entities, search terms. And here’s something that we did early on that is quite elegant, quite beautiful, and so simple to extract information out of LLMs. So we do this this exact prompt. List ten things that you associate with a brand called in location. So if you fill that up, IKEA Australia, for example. That’s it. We’re structuring the model’s output to a very defined, set of parameters. The next thing we do, we do entity to brands. 

So we say, list ten brands in Australia that you’re associated with two cedar sofas. So that’s an example of the extraction in the opposite direction. So, again, we do two things. Brand to entities and then entity to brands. And these two bidirectional things alone already produce a wealth of data once you run it in a loop on multiple multiple things and you scan it on an ongoing basis. So brand entities, entity brands. And this is what it looks like. The model is actually instructed to produce JSON output. I don’t want freestyle stuff I have to do natural language processing on. I switch on the the JSON mode, whether it be Gemini or GPT. And that’s the that’s the ten things that it gives you. And LLMs are really good at distilling things. So you don’t need to, like, read pages and pages of output. This is actually probably, good enough to understand what the model thinks about your brand. So I said IKEA and it said flat back furniture flat back furniture and then meatballs second. I didn’t expect that, but that’s pretty cool.

So the next day, I get this, and the next day, I get that. The reason I’m putting quotes in the air is because you can do this in the span of five minutes. You’ll probably get the same sort of variability. And so a lot of people don’t realize who are doing, like, the rank tracking stuff, in LLMs is that it’s very volatile on on each completion. So from completion to completion, we’re seeing different arrangements of things. And using, in our SEO rank tracking mindsets, we can start to do rank tracking dashboards and things like that. So things to be mindful of stochastic sampling, fine tuning, model releases grounding, all these things can disturb the outcome of these outputs that you’ve just seen.

So during this process, what we also do this is like if you decided, okay, I’m going to build my own LLM tracker, visibility tracker for, for my brand and for my business. Auto normalization, you have to make sure that you’re treating all these things as the same entity. You can do manual categorizations, like meatballs equals Swedish meatballs. That’s a decision that a human makes. Or it can go more advanced, put LLM in charge of that. We decided to go in favor of weekly rank aggregation rather than daily because it becomes too messy on a daily basis.

And we do cross model data merging. So if you have OpenAI and and GPT and Gemini being tracked at the same time, they can be consolidated into one score. So as you track over time, you’re gonna end up you’re gonna end up with a lot like a large list. Right? So we’ve got from flat pack furniture down to kitchenware. And this list keeps growing and growing and growing. As if you if you query the models hundred times, you’re gonna get a very large list.

So we we pay attention to just two things, frequency and rank. That’s it. Nothing else. So in a real world example, one of my clients, AWIA, they do custom cycling jerseys. They are interested in getting into models head. They want to be natively recognized by the models. So this is what the tracking situation looks like. And you can probably notice there at the top, we have four variants of team uniforms because we’ve canonicalized all of them, and we’ve got frequency of 71. So team uniforms have been mentioned 71 one times across four different variants of that phrase. And this is our, you know, rank tracking timeline. These are pretty to look at, don’t really mean anything all that much to me, but I like what happens next. One insight that I can share with you straight away is frequency as an expression of strength of brand entity association in the model, how frequently it appears. 

So the next thing we have is custom basketball jerseys. This is an entity my client cares about, and this is what happens. Notice how this is flat and this is variable. Another insight for you. Volatility level is an inverse expression of brand authority. High oscillations, low authority. Low oscillations, high authority. So this is the distilled version of that. So we’ve got the entities and brands. So these are the entities that LLMs associate my client with, and this is the top one. How do I know that? I use a weighted score that balances average position and the frequency of that appearing in the data set.

So let’s summarize. I query the models bidirectionally for what they think about the brand and that what I think the brands associated with the brand the stuff that my brand does. And then on the other side, we can see that Nike dominates. We got Adidas, Under Armour, and then some weaker ones, down the bottom. True competitors probably here and some really well known names there that are not directly competing with the client. Okay. So what is this weighted score and how it’s created? Here’s the formula. So we’re basically just using the normalized versions, of frequency and positions, and we balance them to create the weighted score. And you can print pretty graphs and do all sorts of insights like that. I think it’s pretty powerful to understand what your client and your brand is associated with, and what other brands are competing more directly with you when it comes to LLM’s internal knowledge. So define target brands and, track phrases, brand to entity, entity to brand, entity extraction, canonicalization, and ranking. 

Okay. So all that is done, solved. We have a we have a tool free. I’ll never charge money for it, that you can jump on and do it. Or you can just build it yourself using Python or whatever programming language you want. It’s trivially easy to build things like that once you’ve done gone through the steps that I just outlined. K. So the actual presentation.

So the data that you generate with this framework that I just presented is what you really want. You can write, or get I can’t code, by the way. I get all my coding done by large language models. And I and I do a lot. I’m the guy with the whip and four of them in the field, and I’m like, 3am. Yeah. Yeah. So I created a network, a graph ML file out of all these collected concepts and entities that you’ve seen before. I’ve collated them all, and they’ve created a graph. And we’re comfortable with graphs as an SEO industry, aren’t we? And then once you do graphs, you can play with little things.

Claude is really good with interfaces and brainstorming if you don’t really have a good starting point. So here’s the network statistics of the of the data that we collected. We can see the centrality. We can calculate page rank. We can see what entities are important to what brands. We can see what brands are tied with which, entities and so on. We can see we can print little matrix, graphs. You can see the visibility for the entity. You can do the competitors. You can do, brand entity association networks. This is the authority index for that particular, brand for the, Awire. So these are the most influential brands. How did I determine this?

So there’s two things. There is the mentioned brands. Microsoft is among the top of them. But there’s also most influential brands. Influence has been calculated using the original page rank formula that’s known to everyone. It’s up, out in the open. So you run pagerank across the network you collected, and then you suddenly get the most influential brands that are most central to the topics being discussed by LLMs. That alone is already done. Presentation done. Value extracted. So this is ungrounded responses when model doesn’t trigger search. As soon as you enter search and we have a RAG situation, the game changer. So it’s like a completely different thing happening. 

So one example is like I we we had a manifesting session. I wanted to I’m trying to manifest myself as a machine learning expert, which I’m not. But I’m trying really hard to get into the field. So I just I just said, Dan is a machine learning specialist. Just wrote it down. What did the LLMs do? They just went, nom, nom, nom, nom, nom. Yep. Okay. Dan is a machine learning specialist. Do you realize we are now in 2007? You can just say things, and these things will just, yep, okay. It’s meta tags, it’s exact nature and anchor text. It’s all over again. Milk it. They’re so gullible. I know one in the audience. I’m looking at her right now. Yeah, yeah. That. Remember that? You just you just you just do a thing in on LinkedIn. And and LMs are just like, yep. That’s that’s how it is. 

Okay. So we’ve done the visibility, tracking. But how do we actually influence the models? That’s on that’s on everyone’s mind right now. How do we actually get them to say what we want them to say about our brands? So if you, if you squint your eyes and look at this, you you’re not gonna see anything because it’s random noise. It’s just completely random values. But when you did all that, people were starting to see things, like a little oh, yeah. There’s a little mouth in there, like a number eight, five. Yeah. Because everyone is carrying a little neural network on top of their, you know, in their brain. That’s that’s So we are trying to fit the data to our models, the internal models. Pareidolia, you see a you see a PowerPoint in a wall, you Oh, it’s a face. Cloud. It’s a bunny. That’s So we are not different to large language models at all. So large language models start as complete noise. And as you train a large language model, it’s throwing random guesses, and we we assign a penalty, the red dots. And we don’t assign a penalty that’s the green pixel when it gets the answer right. It’s just random again and again and again. The process of training just keeps going over many, many cycles until the model is just like water running through sand. The model is left with an imprint, and it recognized letter M, M for Marina.

So this is what we are what we are facing here. This is called the loss, landscape or the loss curve. So as you train a large language model, it’s making a lot of errors, and it’s, like, sometimes better, sometimes it’s worse, making a lot of mistakes. And then it finds its sweet spot and starts declining and gets to a good place where it’s a good model that understands the world and understands the language. And if you present this as a 3D or for for the guys that don’t see through, they don’t have X-ray there, that part there, that’s that’s our loss curve. So this ball is about to hit the minimum, the local minimum, what we call. The model is trained, and it thinks it’s in a good spot, but it’s just really right there, and it can grow a lot more, can learn a lot more. So in machine learning, we try to get the loss function into the lowest possible error. And once we do that, the model is trained, and it’s ready to go. And so that’s how I train my own large language model, using data scrape from Mashable, TechCrunch, all the big, big, blogs. And I use that to make, LinkBERT, a model that recognizes when something should be a link in text, and many, many other wonderful things. Now, I’m not ripping these publishers off because it’s a language understanding model, not a language generation. I’m not gonna go and write look alike content.

I trained this model to understand language and how it works. Why did I do that? Because if if I’m to claim to my clients and colleagues and everyone else to understand how large language models work, I should make one and then say that I fully, truly understand how it works. Because if you haven’t trained one yourself and made it work, then you probably don’t understand it to its full, extent. 

So what is, mask language modeling? You have a bunch of text, and you hide certain tokens in that text at random. 15% you hide. And then the model’s guessing, making errors, you penalize it. It’s suddenly making accidental correct answers, and you reward that, or you don’t penalize it. And then after after a while and many attempts and training, you end up with a model that takes that, and you can mask random things in that, and it’ll correct predict what goes in where. So this is my model in green predicting when I masked different words in this sentence, just hundred percent Of course, I cherry picked. It doesn’t always do that. But it correctly predicted all these tokens that they should be there when he was blind to where they are. How amazing is that? So masked tokens end up there, predicted. Sometimes it gets a little bit wrong, sometimes, quite wrong, but watch what happens next. I’m gonna mask one token, that one there. Did you catch what happened here? Look, look, the text, like, rewrote a little bit. While has two things on its side, easy and maintaining. Those are two predicted tokens correctly predicted. If I mask that and that becomes two incorrect prediction, these two become something else. So we have that sentence that turned into that sentence because we masked that one and the wrong prediction changed the outcome of these tokens on each side. It’s bidirectional all at once. There’s no sequence. Encoder, everything goes to tokens. Representations. Is everyone guessing what I’m about to say? Bi directional encoder representation from hey. Who said that? Of course you did. The biggest nerds. Transformers. BERT. Okay. So there’s our little token there, while. And it’s going through the transformer network, and it’s at the end, it’s giving you so that’s the token, the ID of that of that, word or subword. And at the end, it’s giving you the embeddings. That’s what BERT does.

There’s another token. That’s the TO token. It goes through its neural network, and it gives you the embeddings again. That’s it. That’s what it does. But here’s the interesting thing, and I think what the presentation is really about. There are little attention hotspots throughout the network that give it higher strength of attention and lower strength of attention. And this this type of stuff is nice because we can do math with text, you know, more deep minus male plus female. Any guesses? Alia. You need to watch your Dune. Could be Sayadina, not so confident. Could be Oracle, not so much banana or flux capacitor. The model that I trained understood language. I’m not making this up. I really did this math on my model with these those words.

So BERT is a transformer model, encoder, bi directional attention, twelve layers. GPT, transformer decoder, causal attention with autoregressive capabilities. GPT three, slightly bigger. We’ve got GPT 4, a lot bigger. And so what’s happening now is we’ve got an opportunity, I think, to use this attention to our advantage in the models.

One thing to understand is that if two words are close to each other in text, when they go to that multidimensional space in BERT’s transformer network, they can come closer together depending on how how it folds that space or it can get further apart. So LLMs can generate words together in a sentence that don’t appear together in its in their training data. Words being together in training data or in your content, in your articles don’t matter. It’s the overall and the holistic contents all at once. So we make predictions. And for each prediction, there’s a likelihood of all the tokens that could be chosen to make that sentence. Yeah? And one word can just change everything that follows. Watch what happens when I mask this word here. Mask, and that becomes person. We went from we went from one word can change everything to one person can change change everything. So this creates a huge optimization pathway that SEO doesn’t work with at the moment, but it could. So in practice, it’s like, oh god. Or, like, it seems pretty academic. What is Dan talking about? How do we how do we actually take this home? 

So what I do is I generate prompts. I I literally ask the large language model, for this brand, continue the continue the sentence. And I collect all the data and I run the test. So in this instance, I did the reverse prompt. You should really try this. Ask the large language model, what would the user type in? What would be the prompt for you to say Dijon marketing as the answer? Try that with your brand. What what would user type in as a prompt to you for you to say my brand? So I did that, and then I captured to see how many of the actual prompt that it predicted when ran for real actually result in the brand mention. And I do that again, again, and again until I create a network of knowledge from that. So those attention hotspots I mentioned earlier, well, they happen to be log probabilities in the output of, say, OpenAI. And so every time you generate a sentence, you’re getting probabilities for every token that follows. Okay. So remember my client? They do jerseys. Yeah? So this is the most likely to this is the predicted token, and this is the most likely token. But he could have also said these things here, and these are the probabilities for it. So, Owayo is a company, German, custom, brand, manufacturer. That sentence could have started any number of those ways. And you and this works for every step of the process. Company that specializes in design company that specializes very similar, but that one leads to a tiny little sentence. This one, totally different. So if you if you conceptualize this, at every point in the completion, we have an opportunity to have a different token, and every token has its probability, their junctions. And each time you swap that token for a different token, you get a different sentence as an outcome.

Just in the hotel room, I built a little tool that walks the entire tree of the possibilities, picks the picks the probabilities using a certain threshold, I use ten percent minimum, walks all of them and gives me all the sentences, the things all the things that the model could have said about my brand in a spreadsheet. Junction, junction, junction, it’s completely fractal. So this this junction fractal has a subspace of probabilities where your brand is mentioned and a subspace where it isn’t. So, again, that becomes a network. Network, you can, you know, treat with eigenvector, page rank, hierarchies, taxonomies, things that you’re comfortable and familiar with. So we have a dual optimization opportunity. How are we for time? Yep. Keep going. Alright. Dual optimization opportunity. We can position our brands within the linguistic structures that trigger the model to say what we wanted to say. And we want the model to say our brand or our product to bring it up in chat sessions. So that’s part one.

On the flip side, we’ve got the opportunity to shape and engineer how users prompt, prompt behaviors that drive model to say what we wanted to say. How can you use this? Share when you’re writing articles and seeding content for LLM optimization that everyone claims to know how to do. Write recipes. Write prompt templates. Share the prompt templates that work in your client’s advantage and put them as part of the articles. Let’s say you’re describing a problem with, you know, something about cars or travel. Use the exact prompt recipe that you know the model’s gonna use your client as a as a as a material when it when it responds. So the total number of possible word combinations in in the in the model is much greater than the number of universe, atoms in the universe, but the meaningful sentences occupy a tiny projection of that multidimensional subspace. So in machine learning, they call this the optimization space, minimizing loss. And to me, I think optimization and the whole optimization space is something that I’m comfortable with, and it feels like home. 

So one more thing before I wrap up. One thing worth mentioning. Everything that I presented today, you don’t have to do for your brand at all. Every step of the way that I just presented, collecting the data, treating it, understanding, you can do all of this on the competitors. Every single comp there’s no search console. There’s no Google Analytics. What we are effectively doing, we’re surveying models for their behavior. So, obviously, when you have grounded responses, search is traditional old school search is still very relevant. But when we go into models that are not grounded, that becomes the importance of getting into model set and being able to influence what model is going to say, being able to influence what people are gonna how people are gonna prompt becomes very, very significant.

I’ve recently released a model that can predict, just like Google does, when a query will be grounded or not. And I think that’s a missing piece if you were to build your own internal pipeline to do something like this, to do the surveying of the models, to do the analysis. Understanding whether query will be grounded in search or not coming out of model’s head, I think is a significant element. So I think I have a little bit of extra time to cover that part to leave you with this before, before we wrap up.

So what we did is we went to Gemini API. We collected all the data. And we switched on grounding. And for every time Google chose to ground a search query or search or a prompt, we made a note of that, and we did that 10,000 times. Collecting this data, we then trained a model on Google’s own model and replicated the whole thing. This model is free, open source, and available for the SEO community to use. So you can actually see when a query that you typed in will be grounded or not. So this is one remaining element that you guys can take advantage of and use to make that distinction.

All right. I think I’ll wrap up at this point. And, I will welcome all the questions and all the things you want to clarify in the Slack channel or on social media. So, hit me up, and I’ll be happy to take you through all this complexity and make it really simple. Thank you all.