Morteza Nikoubazl/NurPhoto via Getty Images
- BNP Paribas CIB's AI chief officer says tokenmaxxing can be a "vanity metric."
- Instead, he said his team measures AI ROI against revenue and productivity.
- His comments come as some tech firms like Amazon and Uber question the ROI of AI spending.
For the past year, Silicon Valley has been obsessed with "tokenmaxxing" — the idea that maximizing AI usage (and therefore token consumption) should make workers more productive.
Charles Holive is measuring something else entirely.
The chief AI officer at BNP Paribas CIB, the investment banking arm of one of Europe's largest banks, told Business Insider that while his team tracks token consumption, it's far from the first thing he looks at.
"We try to go away from vanity metrics — billions of tokens per day," Holive told Business Insider at Mistral AI's summit in Paris last week.
"We try to make sure that what we track is an outcome, not a vanity metric," he added.
Instead of asking employees how many AI tokens they've used, Holive starts with a different set of questions.
"What did you do that you didn't do before? How much faster did you do it?" he said.
Outcomes over tokens
His comments come as some US companies are beginning to question whether rising AI costs are producing meaningful returns.
Amazon recently shut down an internal AI-use leaderboard after employees reportedly began performing tasks simply to climb the rankings.
Uber COO Andrew Macdonald has publicly questioned whether rising AI costs are translating into more useful products, while GitHub recently moved Copilot to usage-based pricing as AI bills continue to rise.
For the AI projects he oversees at BNP Paribas CIB, Holive said he begins with explicit assumptions about how much revenue or productivity they could generate.
His team then creates KPIs and monitors progress monthly or quarterly against those goals.
That doesn't mean his team ignores tokens entirely. Holive said the bank still has dedicated teams to monitor token consumption to track costs.
"Then we look at token consumption because I need to control my costs," he said.
Beyond the leaderboard
Holive's approach echoed comments from other executives at the summit.
Amit Kapur, chief AI and transformation officer at Tata Consultancy Services, told Business Insider that companies should focus on "business outcomes" and "business impacts" rather than "only token as one line item."
Antoine Pichot, director of innovation, digital, and data at La Banque Postale, told Business Insider that his bank evaluates AI projects based on factors such as efficiency gains, customer satisfaction, employee satisfaction, and financial impact.
Still, Holive said token usage can still be useful as a measure of adoption, especially in software engineering.
But he said his approach differs from companies that focus heavily on AI usage metrics.
"We didn't say, 'use the tool, do your best,'" he said. "We did the opposite of that," he said.
Read the original article on Business Insider