"Just Quickly Spin Up an MMM," and Other Great Advice You May Want to Ignore

Our industry is full of great information. I’m constantly reminded of the generosity, expertise, and intelligence of folks involved in performance marketing, search engine optimization,

content marketing, social media, digital analytics, ecommerce, and data science. Of course, there is a growing pile of garbage floating around as well (and growing larger by the day with all the AI slop endlessly proliferating). That said, there is a genuine gold mine of really sound, helpful content created by established and up-and-coming people across digital disciplines.

The Jesuits have a set of pillars or disciplines they practice, and one of those spiritual disciplines is the practice of discernment. Though I have no horse in that race from a religious standpoint, the discipline of discernment is a practice we could all learn to do better: sometimes the best advice or tip is the one you decide not to take. “Why,” you ask? In spite of all the cortisol-fueled fear of missing out and shiny objects fueling decision-making (sometimes at the highest echelons of leadership), some really sound industry information and practices are simply not appropriate for your business, your objectives, or your commitments (or lack thereof).

Media Mix Modeling (MMM) and Marketing Mix Modeling (yes, there is a meaningful difference) are good examples of really sound, powerful statistics-based practices that are simply not relevant to many, if not most advertisers.. As the advertising world became more cookie and measurement (and addressability) challenged, there was a rising tide of advice and solutions for various kinds of aggregate statistical methods for assessing the impact of media dollars (and broader marketing, competitive, promotional, and econometric factors) on revenue outcomes. The ability to statistically model response curves, points of diminishing returns, ramp-up and decay, and to then be able to scenario plan ‘optimal budgets’ presents a seductive opportunity for marketing leadership. 

All of this sounds great in theory, and it truly is. The only problem is that the approaches used for media mix modeling and marketing mix modeling require a few baseline criteria to be met in order to be statistically informative, predictive, and useful for scenario planning:

MMM Viability Criteria

• Channel Diversity: if you only spend money on Google Ads every month, MMM is probably not the solution for you

• Variability: if your budget does not change by more than 5% from week to week with the exception of the US holiday peak, MMM is probably not the solution for you

• Volume of Data: if you don’t have at least 12 months of weekly data (preferably 24-36 months), MMM is probably not the solution for you

• Testing & Validation Frameworks: if you don’t have a framework, business commitment, and the team & skills to run structured tests based on the MMM outputs and scenario modeling . . . MMM is probably not the solution for you

Let’s say you do meet all of the viability criteria above, now comes the time to actually train up a model based on your media, organic, non-treatment, and revenue/sales data. You have some great open source options available, such as Robyn or Google’s Meridian offering. Technically speaking, even if you are starting completely from scratch, you could have the environment fully set up and configured, and a default instance of either model spun up with your own data within a handful of hours, but this is also where the value of MMM for most advertisers starts to show some vulnerability.

First and foremost, if you simply run any of these open source models with no parameter tuning of any kind, it’s a bit of a dereliction of your professional duty to combine your domain and business knowledge with technical execution, but maybe you don’t know how to adjust Adstock and Hill functions appropriately (or reasonably). Perhaps you don’t know if and how you should apply non-treatment control variables in your model. The customizations and tunings of the base model is how you make the model a closer approximation of your business, your industry, your own marketing programs, your media investments, and possibly even the overall quality and creativity of your advertising executions. Sound farfetched? Not at all.

The best and most creative advertising is not only more engaging than the worst, it achieves effective reach more efficiently, and it is impactful and memorable for a longer period of time. If you think cannot or should not be reflected mathematically in a media model, MMM is probably not a solution for you.

Are you running the model nationally? Do you have enough data to run the model with geographical segmentation? This will definitely impact how you can use the data.

Drive Business Value and Impact from Your Media Mix Model, Or Don’t Waste Your Effort

So you’ve got a model, and you’ve done the upfront due diligence through exploratory data analysis, feature engineering & variable selection, and you’ve fine tuned the parameters. You’re pleased with the informativeness of the outputs, but how do you keep these insights from sitting on the shelf among other interesting shiny objects that have not delivered the expected business impact?

When it comes to the scenario planning or budget optimizer aspects of media mix models, it makes more sense to think of these outputs as statistically informed hypotheses rather than deterministic facts. In other words, when the optimizer output suggests that with a given set of constraints (e.g. total budget) you should allocate 23% to CTV, 32% to Paid Search, 40.5% to Paid Social, and 4.5 to GPT Advertising, these should be considered the basis for testing and validation, not axiomatic truth.

Now is where the real work begins to do some structured media testing (or hopefully where you’ve already established a framework and infrastructure to support it).

Possible ways to approach these recommendations.

• If you built your model with geographical data, you might consider matched market tests, aka geographic split tests

• Holdout Testing

• Incrementality Testing

• Pre/Post Testing

My purpose here is not to review the pros and cons of these different approaches in depth but really to strongly suggest ways to avoid having your rigorously executed media mix model turn into an underwhelming data exercise with no business impact.

In my professional opinion, if you have no intention of testing and validating the statistically informed hypotheses generated through your MMM then you have no business investing your treasure, time, and energy in these methodologies in the first place (unless of course you are just curious and trying to learn something, in which case you can easily spin one of these up with demo data). The real impact of this set of approaches is when you triangulate the time bound media, marketing activity, promotional, and seasonal data with data from holdout tests (or other testing methods) in a loop that allows you to refine both model outputs and actual performance over time.