#36: Three Top Tips For 2023 / Tip #2
Custom algorithms... what, where, why.
Earlier this week, we kicked off 2023 with our first of three Top Tips. Tip #1 was on retail media attribution — a tricky problem, but solvable by experimenting with Shapley Values.
Tip #2 today is on Custom Algos. What are they? Where do they fit in? Why are they so important?
Tip #3 next week is on working media cost waterfalls. What could be more important than squeezing more out of working media when 2023 budgets hit the breaks?
Reading Time: 9 wonderful minutes
What is a DSP custom algo?
A custom algo is the difference between buying an off-the-shelf suit at Men’s Wearhouse and getting fitted by your friendly tailor.
In the non-custom, vanilla-flavored DSP world, all advertisers get the same thing. In the custom algo world, advertisers want to blend their own special-sauce data to figure out bid prices.
A regular DSP algo calculates how much to bid based on parameters set by a human user (usually a hands-on-keyboard media agency staffer) and uses the same vanilla algo on every campaign for every advertiser.
A custom algo is bespoke for each individual advertiser looking for a privileged strategy where optimization decisions are made by a machine instead of a human. In other words, the 3rd party machine sets its own parameters outside of a regular DSP’s decisioning process.
Humans and needles
Let’s make up a super bite-sized example. Let’s say you’re a handy-dandy programmatic trader with fast fingers on the keys. You power down seven double-shot espressos and chase it with your favorite chemical energy drink. You feel like Tony Montana.
Your boss told you to set up a campaign with five creative variations for ten ad sizes and four audience segments… and to run the campaign on ten SSPs across four browser types and two device types (desktop and mobile), but only in the Top 50 DMAs. She also told you to optimize (pull levers like a Vegas slot machine) to maximize viewable seconds.
WOW! Thank the lord you slammed those power drinks. You’re gonna need it! Your big, beautiful human brain has to deal with one million unique combos to find those awesome ten combos. Good luck finding that little needle in such a massive haystack!
Lucky you. After you come down from your caffeine high, you call up your custom algo friend to lend a hand. With the same constraints in play, the algo goes batshit crazy pulling every slot machine lever in the casino. It also keeps perfect track of which machines give the most.
Multi-Armed Bandit: Our slot machine is an example from machine learning called the multi-armed bandit problem. Optimizely — the website optimization company — does a nice job explaining using a newspaper site as an easy example anyone can understand.
And look at you now! You’re on the beach sipping a Tequila Sunrise while your custom algo does the heavy lift back home in the office!
Where does a custom algo fit in?
A consumer visits a publisher site like USA Today to take in some amazing journalism where there is zero chance of ad quality issues. 🤣🤣🤣
The consumer’s browser brings back the content along with an opportunity to serve an ad.
The browser redirects to the publisher’s ad server, which is basically a yield optimization selector looking to place the right ad in order to maximize publisher revenues (e.g., constantly find the combination of CPMs and inventory scarcity that maximizes ad revenue for the publisher).
If the publisher does not have a direct-sold guaranteed campaign to fill, then another redirect happens back to the consumer’s browser.
Hey, what about the environment? Programmatic auctions are basically a bunch of redirects from server to server to server. Lots of number crunching. If the ad turns out to be served to a bot instead of a human or to some crappy made-for-advertising (MFA) site, society (you and me) ends up paying the price with tons of obnoxious Co2.
Anywho, this is when the programmatic magic happens. The consumer (and the ad opportunity) is redirected to one or more SSPs used by the publisher.
The SSP sends a bid request to one or more DSPs. DSPs are used by advertisers (or usually a media agency hired by the advertiser). At some point in the past, a human being working at the agency set up various campaign parameters in the DSP, including an ad budget constraint, a goal (e.g., conversions or viewed ads), and various tactics to spend the money and maximize the chances of hitting the goal.
Normally, the DSP’s bidding engine (Men’s Wearhouse) will calculate a bid response and send it back to the SSP.
The SSP collects bid responses from several other DSPs, and the highest price wins — as far as anyone knows. 😇🤫
Generally speaking, the DSP collects two fees: 1) a tech fee, which is like rent paid by advertisers to use the DSP’s bidding technology; and 2) a fee for data used in the campaign set up for audience targeting.
When a custom algo comes into play, the bid response is disintermediated (margin-making opportunity) by a third-party company that supplies a private algo to advertisers.
In this case, human involvement is reduced to setting a few constraints: telling the machine what the ad budget is and giving it a goal to chase down. The machine figures out what levers to pull to boost outcomes (the perfectly tailored suit).
The custom algo calculates a bid response and passes it back to the DSP.
DSP Internal Auction (aka black-box bidding)
The DSP sends the bid value to the SSP and so does every other Tom, Dick, and Harry DSP.
If the bid wins the ad auction, the SSP lets the consumer’s browser know about this great news and tells it where to look for the actual ad creative (another redirect).
Now the consumer’s browser needs an ad to fill an empty box (ad placement) that comes along with content from USA Today, so it asks the DSP to send it over.
When a genuine human originally set up the programmatic campaign, it gave the DSP a creative ad tag (instructions) for the DSP to redirect back to the consumer’s browser, telling it to go look somewhere else for the ad creative.
That somewhere else is usually the advertiser’s ad server (another redirect).
Voilà! The ad server serves the ad in the consumer’s browser.
A few problems for adtech investors to think about
Now that you know where a custom algo fits into the programmatic market mechanism, we’re guessing you see at least two issues — one for media agencies and one for DSPs.
Problem #1
Media agencies live and die by billing out FTEs to clients. If these staffers only need to key in a budget and goal (low-value work that any offshore unit can handle), then what incentive is there to sell clients on custom algos as an alternative to optimization? All the billable workload is justified by having humans who claim to be awesome at repeatably finding needles in haystacks.
Then again… when mundane agency workloads like futile programmatic optimization evaporate, leaving behind found time to overserve clients with the kind of value-creating creativity that humans are really good at, agencies can invest in more upstream activities to find their way back to the client’s C-suite. That’s where it’s at! At least we think so!
Problem #2
If an advertiser is already paying its DSP a ~10% tech fee (rent) for a vanilla-colored Men’s Wearhouse algo and decides to put a privileged asset to work like a 3rd party custom algo, but has to pay another 10%, then what good is a DSP besides a “dump bidder pipe” to SSPs?
The rub
DSP fee disintermediation happens in two areas: One is functional and the other is financial:
Functional disintermediation happens when a 3rd party algo intercepts the bid response and does something (calculate a bid value) that the DSP did before.
Financial disintermediation happens when the DSP still charges 10% for tech fees (bidder rent) and now the 3rd party algo also charges 10%. That makes the supply chain longer by one player more = disintermediation.
DSPs can’t avoid heavy number-crunching costs, whether an outside custom algo is in play or not. That means an incremental 10% fee for your multi-armed bandit algo must drive more than 10% in outcomes to have a fair payoff.
The only way to know for sure is to test it out.
A good custom algo is programmed to solve “The Linda Problem”
The problem with people is we are not good at solving optimization problems when there are tons of levers to pull. Too many choices… not enough time in the day.
So what do humans at agencies do to optimize ad campaigns? They lean on a heuristic ( easily calculated, thumb-to-the-wind) procedure called representativeness.
Take this simple poll to see how representativeness screws up your decisions…
>>Scroll down to see the right answer.
The Point: A good custom algo won’t make mistakes with probabilities to find the right levers to pull. Besides the multi-armed bandit problem, human programmatic traders fall into this probability trap day in and day out.
Try an algo at home today!
You subscribe to Netflix, Apple TV, and Prime Video. You’re trying to figure out what to watch and feel anxiety coming on. You’ve been on a streaming frenzy lately so finding something new is a real needle-in-a-haystack problem.
Whataya do? You go to WaterCooler for some help. All you have to do is answer two questions and the algo inspire you.
What are you in the mood for? e.g. “find me fun”
What interests you most? e.g. “family friendly”
Boom! The algo fetches Wolfwalkers. You never would have found that one on your own. The kids loved it! You’re a hero!
Answer to Linda Problem: Linda is a consultant
Even if you think there is a low probability that Linda is a consultant like say 5%, and a high probability that she is an activist like 95%, then there is only a 4.75% chance that Linda is a bank teller AND also an activist (5% × 95%).
Since 5% > 4.75%, it is more probable that Linda is a consultant.
Yes, Option 2 feels more representative of Linda from the description of her but it is clearly mathematically less likely.Ask Us Anything (About Programmatic)
If you are confused about something, a bunch of other folks are probably confused about the same exact thing. So here’s a no-judgment way to learn more about the programmatic ad world. Ask us anything about the wide world of programmatic, and we’ll select a few questions to answer in our next newsletter.
Join Our Growing Quo Vadis Community
Was this email forwarded to you? Sign up for our monthly newsletter here.
Get Quo Vadis+
When you join our paid subscription, you get at least one new tool every month that will help you make better decisions about programmatic ad strategy.
Off-the-beaten-path models and analysis of publicly traded programmatic companies.
Frameworks to disentangle supply chain cost into radical transparency.
Practical campaign use cases for rapid testing and learning.
Thanks Justin. Yes, very true that more than one bid is sent back. I wanted to keep it simple as it does not change the black box nature of the internal auction.
Terrific explainer! For step 9 / DSP black boxes — many DSPs now send back 2 or even 3 winners in a multi-bid response. Though of course still true that the SSP selects one winning bid.