Garrett Sussman: In a world. No. Farrah Bostic’s coming up next. She is the founder and chief strategist of the difference engine helping brands like Google, JetBlue, and NBC turn audiences insight into action. She spent over 20 years shaping brand and product strategy across industries from fashion to finance to the future of play. As the host of Crosstabs and cohost of In the Demo, Farrah brings humor and humanity to research fandoms and the power of women’s voices presenting all the wrong tech for all the right reasons, Farrah Bostic.

Farrah Bostic: Alright. I had plans to wear heels, and then I walked 45 blocks yesterday. And, so I did not wear heels. And I’m happy about that up here. I think I would kill myself. So I’m Farrah Bostic. As, Garrett said, I run the Difference Engine. We’ve been around for about a dozen years. We do insight driven strategy work with a variety of clients across just about every category. And I guess in a way, I am bringing to you the voice of the CMO who has absolutely no bloody idea what you guys are talking about.

They are the people that when I sit down to first talk to them about what’s going on, why they’ve called us, what their problems are. We start talking about how, I don’t know, things were going great. We got acquired. We had all this extra resource. We started to notice the sales curve dip. We’re not real sure what’s going on. And so I pull up their ten ks and I say, oh, well, you screwed up. You stopped spending money on demand generation. How’s it going with SEO? How’s it going with social? Where are you spending your money online or above the line? And they go, oh, well, I can see the above the line budget and I know what we’re doing on our Instagram account, but I don’t know what’s going on in SEO. I don’t know what’s going on in the rest of our spend. I don’t even have access to that data. And so this is sort of a telegram from them. I’m saying telegram on purpose because they’re a little antiquated over there. But I want to also talk about another side of the equation that we’re experiencing in our practice, which is a lot of companies, particularly on the creative side, asking us what our AI strategy is, which as someone who for a living talks to humans is kind of an odd question.

I’m not going to be spending a ton of time talking to robots in order to do my work. It’s literally qualitative research talking to customers. But nevertheless, we’re being asked that question. And it has caused us to do a lot of thinking over the last few years about how we already use it, how we could use it better, what problem it’s really trying to solve, and why we’re even being asked that question. And so that’s what I wanted to talk to you about today. So I think there are a lot of really good motivations for using these tools, and yet they may not be the right tools to actually address the goals that are underlying the request.

So that’s where we’re going to start out. As has been discussed in some of the other talks, agents are in many respects now the new user experience, whether anybody on the brand side really likes that or not. This creates a really interesting challenge for thinking about what user experience ought to be and how to connect that to some of the brand goals and metrics that marketers are held accountable to. But this is the world that we are embarking upon increasingly, and so we do have to reckon with it.

One of the things, though, that I’ve been in the marketing world long enough to notice is that over the last 20-some odd years, we are living in a world that brands help build that seems to be actively trying to kill brands. We saw it in the earliest days with pop ups that piss people off, Craigslist gutting newspapers, search is sort of disentangling other marketing activities from commerce, social eating the world, and now we’re entering this agentic experience where the interaction between the customer and the brand is going to be even more mediated by something else that may or may not be fully in the control of the marketer. And I don’t know if anyone remembers the heady days right around the launch of Facebook and that kind of YouTube to Facebook transition where it was gonna be all about user generated content.

And then the CMOs freaked the F out because suddenly people were making ads for their brands that were not upholding their brand values, that were not talking about the brand in a positive way necessarily. There was a very famous I can’t remember which American SUV maker it was, but someone made this beautiful video talking about how gas guzzling those cars were. It looked great. It was part of the UGC campaign and it immediately made that car maker go, maybe not UGC. Maybe we don’t want to do this. 

So there’s a lot of nervousness at every wave of technology about letting some of the power go, some of the control go. And there’s a good reason for that because over time, this has been the path. And I think increasingly we have to ask what a brand even is. My working definition for twenty some years has been that a brand is a story we all kind of know. You may have heard the story a little differently than I heard it, but when you tell it back to me, I recognize the story. We’re all kind of on the same page. But increasingly, we have products and services that have all the aesthetic markers of a brand, but I don’t know anything about them and neither do you.

And so we’re trying to kind of figure out, as the last viewer was talking about, what establishes trust? When I see it enough times and I can recognize it the next time I see it, when I’m shown a clothing brand on Instagram and I go check on Nordstrom and, oh, it’s on Nordstrom. So Nordstrom thinks it’s a real company. So it’s okay. I can do business with them. Those kinds of things are getting harder and harder to do, and that is a real challenge as well. And the more that we put intermediaries between the customer and the brand itself, the harder it is to discern whether this really is a brand. What do we all actually know about it?

And I think this is going to be a real challenge for marketers as we continue to move into this agentic experience. And then I’ll take you down my, dark musings. I am a bit of a catastrophist by temperament, which is weird because I’m also super optimistic. So somehow I can do both. Not sure how I straddle that. But imagine that this becomes the only screen a consumer ever sees. They go to Google and they ask a question like, what’s the best pair of jeans for a tall guy? And you get an agentic response. And behind the scenes, a brand has paid in order to have their brand mentioned right there in the text. And, yeah, there are a couple of other links there just to give a little credibility to the whole thing, make it seem like you have a choice. But you decide that you’re going to click on that first little link icon next to the summary, and it shows you a completely generated by Sora or something else demo ad or unboxing ad or fake ad, just an ad that no one actually created on the brand side to showcase this particular product.

You get to the end of the little video. Click to buy. Okay. Yeah, let’s do that. Click to buy. It comes back and says, we found the best price on the internet for you. Do you want to go ahead and buy it now? Sure. Click to buy. My Google payment details are already there. The whole transaction took place on this page. Sorry. On this page, I never left the search bar effectively. I never visited the brand site. I have no idea what the brand queues are. They told me it was the best pair of jeans. That’s it. And this isn’t really new. This is something that other industries have already experienced.

Increasingly, what we’ve seen as we go through these cycles with technology is the technology platform yanks back the data. Right? We had open AIs. We had the ability to get in and look at the social graph or the interest graph or the taste graph. And then, oh, no. You gotta pay for that. Oh, and we’re not gonna give you everything. And so there’s no reason to believe that there won’t be rug pulls on the data in the agentic era either.

And the people who know this have been some of my clients in the past. Book publishers. Do you make a habit of buying Penguin books? Is your favorite book brand Harper Collins? You don’t think about it this way. Right? And they sell through Amazon and Bookshop and Barnes and Noble and various retailers around the country, and what they get back in information is extremely little. Amazon doesn’t really tell them the demographics of who bought abundance by Ezra Klein and Derek Thompson. All they do is say, yep. We sold through this many books. Barnes and Noble doesn’t tell them that either, Amazon’s worth. And so they set up pages for the books. The publishers do. And you go on there. It gives you a summary. Would you like to buy now? Okay. Great. Which retailer would you like to buy?

And the reason they do this is to get any feedback at all about who has any intent to purchase and which retailer they prefer. But even then, they don’t really know anything about you because you don’t have a Simon and Schuster login. Right? You are not known to them. And this isn’t the only situation that we’ve encountered this with. Years ago, we worked with Samsung. Now it’s a little bit better these days, but back in those days when we’d go do in store intercepts and talk to people about what kind of mobile phone do you have, people would either say an iPhone or they would generically say Android. But they would also say things like, oh, it’s a Verizon phone. Verizon wasn’t making phones. Oh, can I see your phone? Oh, yeah. It’s a Galaxy. And we sort of went, oh, it’s a Samsung phone. And they’d be like, Oh, yeah. Because that’s not who they did business with.

So Samsung knew how many units they sold into the channel. They knew how many units got returned to them. But they didn’t know anything about the person who bought from the Verizon store or ordered it off the Best Buy website or anything like that. They had no data about this at all. So they had to rely on, frankly, not very good quantitative research developing customer profiles for them. And that was all that they had. They did not have any kind of direct feedback or direct interaction with real customers. So this is not a new problem in the world. If you are not a direct to customer brand, you already have this kind of attenuated mediated through a retailer or a distributor kind of experience and getting to know who your customers are. So this begs a really important question if you are a brand manager, if you’re a product designer, if you are a CEO. If I don’t know anything really about who my customers are, what do I do? How do I plan my business? How do I know what products to launch? How do I know which color of things that they like best? How do I know what’s trending in the marketplace? What is the emergent behavior? Would people prefer to buy this way or that way? I don’t have any of that information at my fingertips.

And so I have good news. As near as I can tell, unless anyone wants to fess up, we are all still in meat space and you are all still wearing your meat suits, which is great because I talk to humans for a living. Humans still exist. There’s, like, billions of them. And, and yet people don’t seem to want to talk to them very often. And like, if you have a teenager, you know that they want to talk to them even less, than they used to. But, you know, I’ve heard all of the excuses about why you wouldn’t want to go talk to your customers. Right? It takes too long, costs too much. Steve Jobs didn’t do it, which is a lie, by the way. I used to work on Apple. We’ll just AB test our way to the answer. Analytics will tell us. We just don’t know how. And then someone will get cute and tell me that Henry Ford said something he didn’t say about faster horses. 

So but, heard all of them, but in the last two years, no excuse in town. We don’t even have to talk to humans. We can make up some humans and we can interview them. And so now we literally have products like synthetic users. Real thing. I get to program into synthetic users personas that I believe to be true and then I get to interview them. It’s amazing. Now, I made it all up. So they’re not real people. It it’s not useless, but it’s close. Waldo is the one that will do all of the work that your junior strategists do now. It’ll make the SWOT analysis that’s not that helpful and do the competitive audit that you could get just by googling your client’s name and seeing the other brands’ results on that left side or right side bar on Google. It’ll do that for you. It’ll save hours of an intern’s time. So that’s that’s cool. And then when you just look up AI market research, there’s a billion little AI tools that are like this. This is to say nothing of the fact that every piece of software we use from Canva to user interviews to a digital diary platform that we like called Recollective, all of them have AI features built in. So we are not really not using AI. Even if we wanted to not use AI, we’re sort of forced to. But this is the new thing, a new way to get out of actually just talking to people.

And this starts to collide with a real problem. Goldman Sachs’ report about AI concluded that in advanced economies, it would automate around 25% of labor tasks. But this begs a really important question. What are your labor tasks? Are 80% of your labor tasks the 25% that can be automated or only about 20% the 25% that can be automated? And that makes you start to wonder who in your workflow, organization can be replaced with this wonderful new world of agentic tools and AI. Is it going to be the researcher, the product manager, the strategist, the person who has the most interaction between the brand and the customer, the most experience and most expertise? Is it going to be the junior folks that you used to hire, the entry level, low skilled staff that you were hoping to train up and teach all of your wonderful tricks and tips and all of the things that you’ve fought so hard to learn over your career? Or are we just gonna ditch talking to customers? 

And certainly, there are people in my industry who think it should be this one, except that in a way, they’re the product. I mean, they’re the product in almost everything on the internet, but they’re definitely the product in my work as well. We are talking to them to find out from them what they think and feel and do and need. And if we’re not talking to them anymore, I’m not really sure why we have a market research industry.

But for me, as someone who’s been doing this for 20 years, it is a little bit like I call it Christmas with a catch. Like, all these things that I wanted to be able to do before because I’m a qualitative researcher structure unstructured data, query transcripts, find out what the themes are at a quick hit level so I can do the thing I always promise I’m going to do two days after we close field work and send the top line summary to the client. And instead, I’m like, actually, this is more complicated. I need three more days. All of that sounds great. Being able to write the first draft of a survey or a screener questionnaire where we know we’re going to ask 10 questions about demographics. Why do I have to rewrite them? We know we’re going to ask about, your brand preferences or familiarity with a set of brands. Why can’t that just be generated or templatized?

Those things are now all possible. We can start to use AI tools to identify people who are full of shit. We can start to determine whether or not people are, in fact, using AI to answer open ends on surveys, which is a thing that is happening all the time now. And we can do some asynchronous moderating. We do a lot of digital diary studies where we’re not actually talking to people in real time, but having follow-up prompts, encouraging comments, those kinds of things used to be a junior person’s job. And so now we can just do that kind of programmed into the diary. 

But on the flip side, it has been creating a lot more weird work for me. I have to check the work of this tool in a way that I don’t have to check the work of a human. I have to look for the non obvious trends. But the way my brain at least works in fact, I heard this, I think Carl Sagan may have said this. So I’m gonna I’m gonna try not to steal. But he talked about how, like, Carl Sagan fan in the house. But, like, he talked about how humans don’t have random access memory. It’s more like a tape. And so, like, you cue yourself up to the position in the tape, but you have to know what came before it. If you’ve ever seen a musician in a symphony perform, they’ve got all these cues about volume and expression and emotion, and they’ve played it a bunch of times and made decisions about how they want to express those notes. And their brain has stored all of that. And when they pick up from measure 75, their brain just really quickly spools through measures one through 75 and picks up right there because it knows everything that happened before that. If I skip the part about the basic themes from research and just try to figure out, well, what are the cool insights that no one’s going to notice but me, I still have to do all that fucking work. Like, I cannot skip to that page because I didn’t read the other pages yet. So it’s kind of creating weird duplicative effort for me that I didn’t have to do before.

We have to check that the AIs make sense, that they are doing skip logic in our surveys that make sense, that don’t break the experience for customers. We are in an arms race about inauthentic responding. They are using AI to answer our open ends. We are using AI to detect their AI generated open ends. And that just gets harder and harder. And if they, you know, only use a few words, we can’t detect them anyway.

And then with the asynchronous moderating, we don’t really want chat bots replacing all the human interaction because sometimes, literally, the person in the diary is saying, I can’t figure out how to plug in my computer, and the battery is dying. Right? Like, they’ve borrowed a laptop in order to finish the study. And the chatbot can’t help them do that. So someone has to get on and talk to them. For our junior level folks, it’s an absolute unmitigated disaster. We are no longer hiring note takers. We are not hiring transcriptionists. We use AI generated tools for that or AI tools for that.

Desk research, simple schemas, a SWOT analysis, a basic brand architecture, all of those things can be filled out by an AI and then just checked. So there’s just less work for interns and junior folks to do. And this is a real problem because we have already experienced kind of a 20-year downward slide of marketers having any skills at all, to be honest. Like, we have no offense. We have some CMOs that we work with that don’t know anything about marketing. Anything? It is like nothing?

But anyway and so if we’re not teaching our junior folks the basics, it’s going to be really hard for them to master these AI tools and use them to any kind of real insightful senior level. And then they will have the problem of, well, if they don’t have all those underlying concepts, they will never upscale into senior roles. We will just continue to have this problem of progressively down skilled marketing folks. It’s not how I would want to come up in this industry for sure. And then if you’re just an organization that’s never really done a lot of research before, my worry is all of these tools just present yet another reason for you to not ever go talk to your customers.  

So there’s a problem with this, which is that if you don’t go talk to your customers and find out what they think and feel and do, and you think you can trust an AI to tell you that, and if you think that you can trust the AI to do some analysis for you, then you find out from your so I have a friend who works, who runs a company that’s using mushrooms to kill pests. It’s a pesticide company, but they use mycelium to do it. It’s very cool stuff. He’s brilliant. He is an expert. He can use AI tools to basically write scientific papers for him because he knows when they’re right and when they’re wrong, when they’re hallucinating, and when they’re a little pessimistic or a little too optimistic. He can check all of that stuff. But the thing he’s not going to do is let perplexity decide which product they’re going to launch first.

Right? That’s not up to the AI tool to do. And this is a very old idea in computing. The 1979 IBM training manual says that computers can’t be held accountable, so they should never be responsible for making a management decision. And yet I am starting to see people who are extending the idea from quantitative research that the data will tell us what to do, that the AI will tell them what to do. And this is a very bad idea. 

So I have good news, though. It doesn’t have to be hard. It doesn’t have to be expensive. You don’t have to have a ton of expertise in what I do in order to talk to humans. I bet you’ve talked to some today. So you already have some basic, grounded skills for doing this. And I think that it is a perfectly good time today is as good a day as any to stop making excuses about why you can’t possibly talk to your customers and start using those conversations to make decisions. And so here is my recommendation. We call this quick strike research.

We can do it for you or you can do it yourself. I, you know, would prefer to take your money, but if you would like to keep it, you can do it. I’ll tell you how. A dozen one hour conversations, three to four hours observing or experiencing the product yourself. Go try to buy it. Every client we work on, I either get a demo or I try to buy the product. 

We worked with a company, kind of late in the pandemic who had had one of these experiences of like, oh, the sales curve went down. And, well, that’s because you turned off all of your demand gen and you shut down all but two of your retail stores. So good job. But I ordered a couple of products from them and they came. They were very nice. It took four more weeks for them to do any second touch with me, to ask me how my purchase was, if I’d like to make another purchase, would I like ten percent off my next purchase. Four weeks is a really long time to go. We were done with the project by the time I got the second touch from the brand.

So going through that process is something anybody can do. As I said, we call this Quick Strike Research and we orient it around five core questions that we think are central to creating any kind of brand dynamism. These questions are, what is your purpose? What is this brand on Earth to do? Who does it serve? What does it deliver? What is it up against? And what should they be thinking about next? And so these are the questions, the objectives that organize the questions that we ask in all of this. But I have great news for you. You don’t have to rack a field study. You don’t have to recruit anybody or pay any incentives. You can answer the question, what is your purpose, by talking to your stakeholders. Surely the founder, CEO, head of product knows what you guys are doing and why. Someone in the organization already knows the answer to this question. If they don’t, call me. You have bigger problems. Who do you serve? Do you know who knows really well your customers? It’s your sales team and your customer service support staff. They talk to them all the time. They see them on the day that they’re most excited about you, and they hear from them on the days where they are the most frustrated with you. Now there’s a whole universe in between there. That’s for sure. But if you can start with talking to those folks, then you can start to identify some customers that are worth talking to or some prospects that you might want to talk to. 

But you don’t even really have to leave the office or the Slack channel necessarily. Just get in the huddle. Just get in the huddle and ask some questions. What do you deliver? This is where you can start talking to your product and design teams. You can talk to the folks who are responsible for ops and delivery. You can talk to your distribution partners, your retail partners, any of your vendors, and start to experience that process and product yourself. And all of this will start to give you a firsthand view into what you deliver. In design research, we sometimes call these kinds of experiences service safaris, which is weird, as a phrase, but that is what it is called. What are you up against?

You probably know some friendly competitors people in your space or adjacent to your space who can tell you what they’re seeing. Not in any kind of granular detail necessarily, but they can talk to you about what’s going on with their supply chain, what they’re seeing as emergent customer trends, what they’re seeing as new challenges coming down the pike, what the what’s keeping them up at night, what’s getting them up in the morning. And you can talk, if you have them, to your government affairs and PR teams because they will know where you’re getting in trouble. They will know where you’re likely to get into trouble. They will know what kind of trouble you are desperately trying to avoid. And then, of course, there is social listening, which lately I feel like, in market research, AI is the new social listening. Like, everybody’s got a tool for it. None of them are very good.

What’s next is the final question. And this is where you do want to be tracking emergent technologies in your category, watching out for those changes in customer behavior, talking to extreme cases, people who use it a lot or hardly ever, people who are using it for a very niche use case that you didn’t anticipate when you launched your product. Those folks are really worth taking time to talk to. And then I say here, make time for collaborative imagining. And here, I really do just mean get your team in a room and ask what if until you’re bored of the question. Try not to get bored too fast.

But these are the kinds of things that you can do without ever hiring a researcher or a research company or having to think about how to design a research program. You can just start talking to folks. And the reason that this matters is if you are new to a business, new to a category, this is a way to begin to establish your expertise. And if you have already been in the business for a long time, this is a way to maintain and constantly level up your expertise because things are constantly changing around you. I think the theme of this conference has been exactly that.

So the thing about that is agents, though they are extremely confident in the way that they communicate with you, they sound like they are sure they are right, they are not experts in anything. They’re not human intelligences. LLMs are probably not the thing that’s getting us to AGI, by the way. AGI will be a separate work stream altogether. LLMs are stochastic parrots. They’re good at predicting what the next word in the sentence ought to be, and they’re getting better and better and better at that. But they’re not experts on anything. So you have to be the expert. And therefore, they should not be setting your strategy.

We do research of any level, the quick strike kind, the big multinational studies with quant and qual and ethnography and everything else, because we need to develop a strategy about what to do next. After, I don’t know, the first ten years in research, I realized no one actually wants a focus group. They just want to have a question answered. They want to know what they’re supposed to do. So that’s what a strategy ultimately is. It’s a plan for a desired result within a set of constraints. You need to identify what the constraints are. You need to know what you’re trying to get to. You need to come up with a plan. That’s what the research is there to inform.

So the most important thing you can be, again, assuming you’re human, is an expert on the thing that you’re an expert about. And so continuing to develop your expertise is really important. When we start to automate our understanding of other people, of behavior, of trends, of technologies, we actually lose the opportunity to know something nobody else knows. Anybody can ask ChatGPT that question. Anybody can go into perplexity and have a whole conversation with them about whatever. 

The other day, so confession, I went to law school. I, in fact, graduated. I’ve never practiced as a lawyer. You know, three years was enough. I felt like that was enough practice. And then I but I still love the law. And I have a friend who’s at one of these law firms that’s being attacked by the president of the United States. And he sent me a recent ruling from a district court judge sent sent to me and a couple other friends. And one of my very sweet friends takes that citation of the case, sends it to ChatGPT, and says, hey, can you summarize this case for me? And my friend plunks the response down into the chat. And then he says, okay. Now diagram the case for me. And I was like, oh, that would have been handy in law school, actually, someone diagramming the case for me. The diagram’s the case, drops it in. He’s like, alright. Now render the diagram in manga. And so I did that, and that was very cool too. And I said, you know, here’s the thing, though, buddy of mine. It’s a four and a half page opinion written in plain language, and yet the prose is incredibly moving. When I read it, I’m going to confess to this and you guys are going to think I’m a super nerd, I cried a little reading this opinion. It was lofty. It was democratic. It was full of high principle. And ChatGPT registered none of that. And so the experience of actually reading the opinion was just sucked out of the whole thing.

And, like, this is one of these things where if you had read the actual opinion, you would know how firmly and how fervently those judges felt about what they were saying. Not just the practical implications of what they were saying, but the fact that they were frankly appalled that they had to say it. And that is almost as important to the law as the outcome of the of the ruling. And so this is the thing.

Anybody can feed that citation into ChatGPT and get a summary of it. But if you don’t read the opinion, you will not know the thing that only the people who read the opinion will know. It’s a little example of it, and I know slightly orthogonal to this conference. But nevertheless, I think it’s really important to to keep that in mind.

Experts do more than know stuff. Experts should not be the people who are saying, well, we’ve never done it that way before. Experts should not be the people who are saying, well, we do it this way because tradition. Experts are people who know so much and have been through so much and are so creative that they can take those leaps. They can discern the difference between a good analysis and a bad analysis of something. They can identify opportunities, look around corners, invent new ways of doing things. And most importantly, experts are in a better position because of all of those other skills to make good decisions, not just to make an expedient one.

And I just want to kind of leave you with this because we are entering a period that is going to be very different from anything anybody in this room has probably ever experienced. We have been through recessions before. We have not been in global trade blockades before. We have not been in political isolation before. So this recession is going to be different. It just is. And that is going to require all of us who have expertise in whatever it is that we have expertise in to use that expertise to develop expertise in things you weren’t an expert in before.

I call that strategic resilience, and it is one of the things that I continue to get from constantly talking to other people. And I hope that you and the people that you advise in your roles will be able to take some of that on board and try it out for yourself. 

And that’s it. Thank you very much.