ÃÛ¶¹ÊÓƵ

Transcript

Hi, my name is Kelly Blackwell and I’m a Product Marketing Manager at ÃÛ¶¹ÊÓƵ. Today, I’ll be talking to you about ÃÛ¶¹ÊÓƵ MixModeller. MixModeller is ÃÛ¶¹ÊÓƵ’s AI-powered, incremental marketing measurement and planning tool. This marketing analytics product enables marketers to measure the ROI of their campaigns and optimize planning holistically across all channels, including paid, earned, and owned. It provides the incremental impact of all marketing activities on business and campaign outcomes through a holistic, end-to-end measurement application for both digital and offline marketing. As an ÃÛ¶¹ÊÓƵ Experience Platform application, MixModeller harnesses the power of the AI-as-a-Service framework, as well as proprietary ML technology to produce results very quickly and with added flexibility for users. Now, before we jump into how MixModeller works, let’s quickly set up the problem, the key challenges that impact our ability to measure marketing performance. This slide highlights the common challenges we see organizations have in measuring the effectiveness of their marketing. The first challenge is fragmented results across channels and methodologies. When you have two different marketing methodologies, such as Marketing MixModelling, or MMM, or Multi-touch Attribution, also known as MTA. In two different solutions, those solutions using different methodologies often give conflicting results on performance, which lowers the overall confidence and actioning on those insights. The second challenge is it’s often slow and time-consuming to get new insights. Organizations spend excessive time and resources spent on wrangling data. A lot of brands will outsource this work, but by the time they get results back, in-flight optimizations are no longer possible because the campaign has already finished. Thirdly, limited data capture erodes insight quality. Signal loss is a huge factor here. Now that third-party cookies have been turned off for 1% of Chrome users or 30 million users, companies are starting to really realize that the data depreciation that they’ve already experienced from walled gardens and other channels is only going to get worse. Marketers are trying to future-proof their strategy and reduce their reliance on an MTA-only approach since MTA measurement methods are going to be severely impacted by the signal loss, as they rely heavily on third-party cookies to connect customer touchpoints across channels. Finally, a lot of legacy measurement solutions do not easily drive actions. Reporting and planning tools are unintuitive to use, and they make it hard to optimize because they use static deliverables that are prone to error, like PowerPoint. Each of these challenges we discuss lead to cost and waste, including missed ROI opportunities and a general lack of alignment on marketing strategy. According to Gartner’s recent analysis, CMOs are even more likely in 2024 than they were in 2023 to report that their marketing org is viewed as a cost center rather than a profit center. On top of that, marketers are being asked to do more with less. eMarketer recently surveyed marketers, asking them what they needed to augment their strategy for the future. These marketers said they needed better and faster Media Mix Modeling, or MMM. They also said they needed a way to measure incrementality. ÃÛ¶¹ÊÓƵ Mix Modeler is the next evolution of marketing measurement and planning. Its AI-powered model uses a unified measurement methodology, which includes both MMM and MTA methodologies, to provide fast, comprehensive insights across paid, earned, and owned channels, which helps marketers make more confident optimizations. Additionally, MMM uses data tied to revenue and ROI, which empowers marketers to speak the same language as the finance teams within their organization, helping them clearly demonstrate the role of the marketing org as a profit center and growth driver. The ÃÛ¶¹ÊÓƵ Mix Modeler methodology. I’ll go from left to right as I talk through this. We’ll start with how we bring data into Mix Modeler, the inputs. We bring in data from mixed granularities, including aggregate-level data, or top-down-level data, which ultimately feeds into the marketing mix modeling engine. This includes data from channels like paid social, connected and linear TV, and offline channels like billboards. Mix Modeler also takes in event-level data, or bottoms-up data, that feeds into the MTA engine. Examples would include email, or owned websites, or mobile apps. Once your mixed granularity data is in, you have the flexibility to and how you configure your models. For example, you can utilize priors or learnings you have from prior modeling efforts to inform the output that this model is going to deliver. The middle piece of this slide represents our model’s intelligence. ÃÛ¶¹ÊÓƵ’s proprietary bi-directional learning technology allows Mix Modeler to ingest data at different granularities, conduct measurement on that data using two different approaches, MMM and MTA, and ultimately deliver consistent, unified outputs. This intelligence also shows the incremental impact of marketing’s effectiveness. I’ll briefly touch on the concept of incrementality. Incrementality really answers the question, how much credit can marketers claim that their marketing activities influenced a customer’s purchase decision? Put another way, if you were to stop running a campaign and sales are reduced by X dollars, that X dollars represents the incremental impact that your marketing had that would not have occurred without your marketing efforts. Brands are really starting to wake up to the fact that understanding incremental impact is by far a more accurate way of measuring the true performance and the true value of marketing. Marketing leaders are seeking incremental measurement because it adds credibility to the successes they claim their marketing organization had. As for the third and final stage of the slide, outputs, I am really going to dive into detail the outputs that Mix Modeler delivers during the demo, but it’s worth noting that Mix Modeler outputs are returned to you to action on in hours, not weeks or months or quarters. Now let’s pop into the interface to give you a taste of what this actually looks like. We typically start our Mix Modeler demos in ÃÛ¶¹ÊÓƵ Experience Platform, or AEP, because Mix Modeler ingests data using the same user-friendly data connectors as AEP. One thing to note, if you already have event-level data in AEP from a source like ÃÛ¶¹ÊÓƵ Analytics, you don’t need to bring this data in again. Instead, you can use what’s already in the AEP data lake. Now we’ll head to the Mix Modeler application itself. We’ll start on the Harmonize Datasets tab. Once data is ingested in Mix Modeler, you’re going to want to harmonize it, which has the goal of normalizing or standardizing the data so that it’s in a common taxonomy that can then be analyzed or read by Mix Modeler. There are four steps to this harmonization process. Fields, Dataset rules, marketing touchpoints, and conversions. In Fields, you start to create that common taxonomy. For example, if one data source calls revenue, total revenue, but another data source calls it revenue, you’ll want to make sure to standardize that naming convention so that Mix Modeler recognizes those terms as the same thing. There are several default fields preconfigured in AMM here, but you also have the ability to edit, delete these, or add a new field. Now once I’ve defined the fields, the next step in the harmonization process is to use Dataset rules to map the harmonized fields you’ve just created with the data that you’ve ingested in Mix Modeler through AEP. Each dataset you bring in is going to need a dataset rule that you’ll set up here. To create a dataset rule, go ahead and click this button. You’ll want to pull in the dataset that you’ve ingested through AEP. Within that dataset, you see fields available here, and then you have the ability to map them to the harmonized fields you created in the last step. Cancel out of this. In the banner here, you can also see that you have the ability to sync data on a cadence that you prefer. The next step, the third step in the harmonization process, is creating marketing touchpoints. So you have the ability to define what represents a marketing touchpoint within your data. Let’s say that you wanted to add a new marketing touchpoint called display. You’ll also want to add the touchpoint volume and the touchpoint spend that goes along with the touchpoint you’re adding. Note that each touchpoint is going to have a different touchpoint volume and touchpoint spend associated with it so that the model can calculate that touchpoint’s contribution to a conversion event. Let’s move to the fourth and final step of the harmonization process on the conversions tab. Similar to what you’ve seen so far, you have a lot of flexibility here to edit, delete, and add new ones. Know that every model in ÃÛ¶¹ÊÓƵ Mix Modeler is going to need a conversion event associated with it. So let’s say that you wish to track online sales, you also want to track in-store sales, and then you want to track total sales together. You can create that total sales conversion convention here. Now once you’ve ingested and harmonized your data, it will appear in ÃÛ¶¹ÊÓƵ Mix Modeler. But before you start building models, we recommend that you validate the data you’ve uploaded, ingested, and harmonized to ensure it’s all correct. So we’ll click over to the overview tab to do that. I have it open here as well. Oh, no, it’s loading here. I like to think of this overview tab as a data dashboard. It serves as a great visual way to validate the data that’s been ingested and harmonized. So you can see that there’s around $115 million total cost in the data that you have ingested. There’s ability to see that cost metric on a time series. You can also look at your data through the lens of impressions and clicks. If you want to filter down by campaign, you can do that on the left nav pane here. Now that you’ve validated your data, we’ve arrived to the fun part, building models and viewing their insights. I’m going to tab over to the models tab, which I also have loaded here. We’ll actually work backwards and start with model insights. And then later in the demo, I’ll show you how you can go about building or configuring a model that will result in the insights we’re looking at now. I’ve chosen this model to look at, Luma MMM online sales model with MTA. So at a high level, these model insights reflect the incremental impact of marketing on conversions, ROI, or other business priorities you might have. They’re available to view in this easy to review visualizations dashboard. But all of the data behind these visuals can also be downloaded to a CSV file, or it can be addressed directly from the AEP data lake in case you want to use this data in a BI tool of your choice. Now to speak to the visuals themselves, I’ll move from left to right, top to bottom. The first visual contribution by date and base media is a great visual expression of the concept of incrementality we discussed earlier. The light blue base showcases the contributions you would receive if you ran no marketing at all. The incremental lift or the true performance of your marketing is the darker blue colors on top of that categorized by spend and non-media spend. The next visual contribution by channel shows the contribution by channel. This is your biggest contribution, and then your incremental contribution is on top of that. Next, we’ll take a look at this third visual marketing performance summary. So this really represents the ROI on a channel by channel basis. In this model, we can see that brand being social has a large positive ROI, whereas video has a negative ROI. The fourth and final visual I’ll touch on, marginal response curves, shows the marginal break even point indicated by this star. This shows the point at which you start generating less than a dollar back in revenue for every dollar you spend. The marginal response curve is generated for all paid media, but you can also look at it on a channel by channel basis too. Let’s move to the next tab. So we’re shifting to the factors beta tab. This is where you can see what external and internal factors affected your sales or how they affected your sales and how much impact they had relative to one another. Here we can see that the S&P 500 had a positive impact on sales. The promo calendar, kind of hidden on this line, also had a positive impact on sales, though less so than the S&P 500. And inflation had a negative impact on sales. Now this tab is listed as beta because this is the first of many self-serve business related factors and features to come. Our team is in the process of seeking customer feedback to learn more about what kind of factors, visuals they would like to see here. Now let’s move to the attribution insights tab. If you have event level data added to your model, you can see where digital channels are driving the most incremental conversions on this tab. Note this attribution tab is still driven by ÃÛ¶¹ÊÓƵ Mix Modeler’s bi-directional transfer learning technology where MMM and MTA models are influencing and informing one another. Using this drop down at the top, you have the ability to compare attribution models. So the incremental model is the default and by itself is the default, but if you wanted to look at say first touch and last touch and compare those side by side, you can do that by unclicking this one and clicking those two. I won’t do that now just to save on time. And I won’t go through these models individually, but I did want to highlight one specific one that is one of my favorites, the breakdown by touchpoint position here. What this one shows, this one really illustrates if a channel is a better as a starter, a player or a closer so that you have a more informed position on where to flight that particular channel within one of your campaigns. I’d also like to note that all of this data here in Mix Modeler can be visualized in customer journey analytics for deeper analysis. Now we’re going to shift to the diagnostics tab. So the diagnostics tab underscores how truly transparent this technology is. You can see all the industry standard model fit metrics provided right here, R squared, MAPE, RMSE, which can empower the data scientists and marketers on your team to feel confident in making decisions based on these insights, based on a model fit target that your organization feels comfortable with. We’ve heard from our customers that they really appreciate having this level of insight into these kinds of metrics because it enables them to quickly understand the model quality and its fit and empowers them to make adjustments if they’re needed. So we just looked at model outputs. Let’s look at how to build a model. We’re going to open the model canvas to start a new model from scratch. We’ll call it test and click next. First off, every model needs a conversion. Let’s say we want to look at online orders. Models also need associated marketing touch points. So you want to decide what are all the relevant marketing touch points for this particular model to analyze relative to this conversion. I’ll click a handful here. You’ll define the eligible data population if needed, and you have the ability to add an external and internal factors data set. These are the visuals, by the way, that we looked at on the factors beta tab. Let’s say we want to look at the S&P 500. And for internal factors, let’s say that we want to look at an internal realm calendar. Lastly, on this page, you want to define a look back window. A look back window answers the question, how far into the future do we want to look back to influence what drives a conversion? Let’s say three weeks. Scroll up like next. Spend share, essentially, if you have any holes or sparsity in your data, the algorithm can fill those holes if you toggle spend share on. So we’ll go ahead and leave that on. MTA enabled this box. If you want to see the MTA attribution outputs like the ones we looked at before, you’ll want to toggle this on. And lastly, you have the ability to define prior knowledge as an input to your model. So, for example, let’s say that based on prior modeling efforts, your company has learned that display has historically contributed, let’s say 50% of your towards your contribution, and you’re highly confident in that we’re going to, we’re going to go up to 90% here. The model is going to incorporate this prior belief, which will make the mixed model or predictions more precise and tailored to your business. Now I won’t click next here, but the model would run within just a few hours so that you can act on those results quickly. I’m going to cancel and return us to the measurement homepage where we have a list of all of our models that we’ve previously built. And I want to chat to you just a bit about retraining and rescoring. So this was the model we were looking at, and here you’ll see the option to train or score this model. Now, what’s really special about retraining or rescoring with mixed modeler is that you have the ability to do so at no extra cost. For most of our competitors retraining and rescoring would be costly and manual to do on a regular basis, but due to the power of the platform and the inherent automation we built into mixed modeler, retraining and rescoring is quick and seamless, and again at no extra cost. Now for those who aren’t data scientists, you’re in good company, I’m not either. I’ll just provide a bit of context on what these two terms mean. So let’s look at retraining and rescoring in a different view. So retraining is essentially asking the model to take in new information and make changes to the model algorithm itself. It’s much more extensive than rescoring. Now, rescoring, on the other hand, is asking the model to take in new information, but not make changes to the model algorithm itself. You have the ability in mixed modeler to ask the model to rescore for an entire date range, or you can incrementally score, which means that you don’t want the model to mess with past data. You just want it to start scoring the new data moving forward. Now, why is this important and why should you care? Incremental scoring means that you have the ability to take in fresh data from an active campaign, and you can use that fresh data to make live, in-flight campaign optimizations. This is a huge differentiator for us, and I’ll speak to this capability more when we get to the performance to plan functionality later in the demo. So we’ve built our model and generated model insights. Now we’ve arrived at the point where we want to use mixed modeler’s AI to help us accurately predict future outcomes. You do this by creating a plan. I’ll follow a similar sequencing with plans as I did with models. First, we’ll start with the plan outputs. Then I’ll show you how you can compare two plans you’ve created side by side, and then I’ll show you how to build a plan afterwards. So let’s go to the plan insights page. This is what a plan insights page looks like. This page has a lot of great information useful to a media buyer or planner. You have the ability to see the plan’s ROI metrics by factor, the state it’s currently in, or by percent. The scatter plot illustrates the ROI and spend profiles of each touch point. You have some metrics around forecasted revenue and forecasted conversion at the channel level. And to get more granular guidance, you’re able to actually download this plan to Excel, where you’ll have data around the weekly spend per channel included for the duration of the campaign that you have chosen to build in the plan setup. So now you’ve seen how to use our AI model to create a plan for how to allocate your marketing budget in the future. But what if you receive a 10% budget increase? Or what if you want to negotiate for one and need some data-driven justification for that ask and increase spend? What happens to your bottom line metrics in that case? This is where the compare plans feature comes in handy. So let’s click on two plans we want to compare. This one has a 10% budget increase. Got to open in another tab here. So this plan on the left, it has a 10% budget increase. And with this, you are gaining $39 million more in expected revenue than the plan with the lesser budget. But at what cost, right? Well, your forecasted ROI here, it’s only taking a negative 8% hit. That’s relatively small. So if you wanted to optimize for revenue and this ROI decrease is not something that’s bothering you as much, this would give you quite the justification to ask for that 10% budget increase. It also, this compare plans feature shows you how you would go about spending that additional budget across channels. Combining measurement and planning within a single UI is a differentiator for Mix Modeler. And we’ve heard from our customers that there are operational efficiencies in using this planning capability as well, because they’re able to share Mix Modeler plans with the media agencies who buy on their behalf. Okay, so those are plan insights. Now let’s address how you would go about building a plan. So we start on the plans page and then click open plan canvas. Now the plan builder enables you to outline marketing spend over a given timeframe based on a conversion event. Let’s just call this Luma Q1. Each plan requires at least one model, a date range and a budget. For this model, we’ll choose this one. And for a date range, let’s pretend it’s a new calendar year campaign. So we’ll start on the first and we’ll have it run for three months. I’m going to input a budget of 25 million. You also have the ability to add different flights in here. So if I wanted to add another date range and additional budget, I could do that here. I’ll go ahead and click next. So I can leave it at just that. And if I click create plan now, Mix Modeler’s AI will generate a plan using the information we just gave it. But you can also edit channel budgets first. Let’s do that. So this is just one more way Mix Modeler provides flexibility to the user. We know that some brands have contractual obligations with certain agencies or publishers. There might be a campaign blackout period, for example, or they have a certain budget threshold that they need to hit. You can customize those constraints here. That’s it for how you build a plan. Now let’s talk about your ability to optimize in-flight performance. You’ve created a plan and executed on it. Once your campaign has been live for a few weeks, you’re going to want to see how the predicted ROI of the plan you chose stands up against the actual in-flight campaign data that you’re receiving. Thanks to Mix Modeler’s ability to regularly ingest campaign delivery data at both the aggregate and event level, you’re able to surface those insights. I have a tab open with those insights here. Using this data, you can see where your campaign is standing up and falling short against the chosen plan. So we have a high ROI and a high order count, but it does look like we’re under pacing in terms of our revenue target. So revenue to date is here, but we were expecting to be closer to the target. That said, it also looks like we’re under pacing in terms of budget. So we’ve spent a fair bit of budget, but we still have quite a bit of room to play here. These insights tell you that you have the opportunity to reinvest your budget in the highest performing channels. This performance to plan functionality we’re looking at is a very big deal to our customers. Instead of having to wait until after a campaign is live to view the actual versus predicted results, and then use those results to make decisions for the next subsequent campaign, with Mix Modeler, you can now understand how an active live campaign is performing against predictions that you made during the campaign itself. And you can make adjustments and optimizations from there. You’re essentially picking up dollars left on the table. So that’s it for our demo. There’s many, many more features I could show you, but the key takeaway is that ÃÛ¶¹ÊÓƵ Mix Modeler allows marketers to understand the incremental effectiveness of their marketing spin to help your organization make data-driven decisions with confidence. Thank you for your time.