Micro Case Study: How American Express is underwriting AI agent error to unlock trust in $trillion-scale agentic commerce

The big question

If AI agents can execute payments, what guarantees that they execute the right payments?

The move

American Express has launched Agentic Commerce Experiences (ACE) Developer Kit, which formalizes and verifies user intent before any transaction takes place. The firm has also introduced what it calls an “industry-first” protection against AI agent error, agreeing to cover eligible transactions when an agent executes an authorized but unintended purchase.

For example, a user asks an AI agent to book a “quiet hotel room under $250”; the agent finds a deal and completes the booking, but if it’s next to a busy street, the transaction is valid yet misaligned with intent.

How it works

The new ACE Kit shifts payments from simple authorization to intent-driven execution:

  • User intent is captured as a structured, verifiable, and enforceable input.
  • That intent is authenticated and tied to tokenized credentials before any transaction is initiated.
  • Agents can transact on behalf of card members only within clearly defined, authenticated intent and control layers.
  • Amex extends purchase protection into agent-executed transactions.

Instead of resolving disputes after the transaction, the system aims to reduce ambiguity before execution.

“The model includes card member enrollment and authentication and gives card members the ability to manage controls directly in the Amex app – using structured intent, spend limits, merchant preferences, and tokenized credentials so the agent can only act within clearly defined boundaries set by the card member,” noted Luke Gebb, EVP and Head of Global Innovation at American Express. 

“The core of our approach is that intent is not treated as a loose instruction – it’s treated as a structured, verifiable representation of Card Member intent that the system can evaluate and enforce,” he added.

How is this different?

Traditional payment systems answer one question: Was this transaction authorized?

Agentic commerce introduces a harder one: Did this transaction reflect what the user actually meant?

That gap between execution and intent is emerging as one of the weakest links in AI-driven commerce.

McKinsey estimates that agentic and AI-driven commerce could generate trillions of dollars in economic impact by the end of the decade, but only if trust in automated execution scales alongside it.

Amex’s move directly targets that trust layer. Its closed-loop network provides end-to-end visibility across users, agents, credentials, and transactions, allowing it to link intent, execution, and liability within a single system.

Why it matters

By underwriting agent error, Amex is enabling AI-driven payments, but also pricing and absorbing a new category of risk.

That changes the equation for adoption. Agentic commerce won’t scale simply because agents can transact; it can scale when users trust that those transactions are executed correctly.

In the Chart: Amex’s take on securing the agentic commerce stack

Lili CTO Liran Zelkha on building AI that disappears

The fintech industry has spent the better part of the last two years racing to add AI to its products. Chatbots have been bolted onto banking dashboards. Summaries have been appended to transaction histories. Assistants have materialized inside apps that users open, at best, once a week. Liran Zelkha, co-founder and Chief Technology Officer of Lili, a financial platform built for small business owners, thinks most of this activity is pointed in the wrong direction. Zelkha has spent years thinking carefully about the relationship between technology and small business owners: customers who are skilled at their craft, pressed for time, and rarely interested in learning a new interface. This customer profile has shaped the architecture of Lili from the beginning, and it informs Zelkha’s view of where AI in finance could genuinely move the needle: The goal is not a better in-app AI experience. The goal is to make a company’s financial capabilities available through whatever AI the customer has already chosen to trust. “The business owner shouldn’t have to learn our app to get value from us,” Zelkha says. “They should be able to ask their AI companion about their cash flow and get a real answer, backed by Lili.” …

How agentic commerce is making execution, intent, and credit actionable inside payments

Agentic commerce is minimizing the distance between execution, intent, and credit within payments. Functions that once operated in separate layers of the stack are becoming more connected and responsive to one another.

And as that happens, the rest of the stack can’t stay passive. Authorization needs to interpret, not just validate. Credit has to be adjusted per transaction, and infrastructure has to carry context, beyond just credentials.

Across the stack, different players are enabling this shift in different ways, using agentic AI to reduce the friction between how decisions are formed and how they are carried out.

Here’s how it’s playing out.

Stripe and the push to make agents transactable

Stripe is addressing the execution layer of this shift, enabling AI systems to complete transactions once a decision is made.

Its product direction, including work around agentic commerce in addition to stablecoin infrastructure, is aimed at making AI systems economically native. The Agentic Commerce Protocol (ACP), developed with OpenAI, is an attempt to define a shared language between merchants and AI agents so transactions can happen without bespoke integrations for every system.

The firm is trying to normalize the idea that agents will increasingly initiate commerce flows, and the system has to treat that as a normal input.


How American Express is fixing the weak link in agentic commerce

Agentic commerce, where AI agents anticipate a user’s needs and act on their behalf, is starting to move from conceptual presentations into real-world pilots. These systems are expected to transform the entire transaction journey, from discovery and comparison to checkout and even post-purchase management. The economic upside of agentic commerce could be significant, with estimates from McKinsey & Company pointing to trillions of dollars in potential impact by the end of the decade.

But that future remains constrained by execution. That gap between decision and transaction is already shaping how these systems are designed.

It’s precisely this fault line that American Express (Amex) is targeting. With the launch of its Agentic Commerce Experiences (ACE) Developer Kit, the company is introducing a framework that enables AI agents to execute transactions on a user’s behalf but only within clearly defined, authenticated intent and control layers.

This balance between automation and constraint may become a key design principle of early agentic commerce.

Luke Gebb, EVP and Head of Global Innovation at American Express

“With the ACE Kit, the goal is to make purchases seamless without losing control or the trust, security, and service card members and Merchants expect from American Express,” says Luke Gebb, EVP and Head of Global Innovation at American Express.

“The model includes card member enrollment and authentication and gives card members the ability to manage controls directly in the Amex app – using structured intent, spend limits, merchant preferences, and tokenized credentials so the agent can only act within clearly defined boundaries set by the card member.”

At the same time, Amex is addressing the AI trust deficit more directly. Alongside ACE, it is rolling out Amex Agent Purchase Protection, extending safeguards to cover transactions carried out by AI agents, as an acknowledgment that before autonomy scales, accountability must be built in.

So, ACE is designed to do three things:

  1. Let AI agents transact using Amex cards
  2. Ensure those transactions reflect verified user intent
  3. Extend Amex’s protections into this new, agent-mediated layer

The real problem isn’t payment execution – it’s intent


Citizens’ CIO on the ethos that led the bank into the cloud and beyond

Not all banks are stuck in the mainframe era.

Today, we look at Citizens, which has been on an impressive modernization and innovation journey, speaking with the bank’s Chief Information Officer and Head of Technology Services, Michael Ruttledge, to understand how one of America’s oldest institutions shed the weight of legacy technology, moved entirely to the cloud, and built the organizational ethos to carry its progress forward.

The start of the road

Citizens was once owned by the Royal Bank of Scotland, which divested its stake in 2015. Ruttledge, who joined the firm in 2019, was coming into an organization already in the process of modernizing. “When I joined it was a good time because people started to move more applications to the cloud and more were moving into agile development,” he said.

Since his joining, the firm has been focused on its “Next Gen Technology” initiative that focuses on 5 main pillars and serves as the spine for its modernization efforts:

  1. Empowering the development cycle: Move into an agile environment through developing DevSecOps tools and test automation.
  2. Enhancing communication within the infrastructure: Leverage APIs to modernize the technology stack.
  3. Improving internal capabilities and talent: Upskilling the current workforce because the bank had previously skewed towards outsourcing and developed considerable technical debt within the team.
  4. Transitioning away from mainframes: Moving the infrastructure to the cloud and remote servers.
  5. Fortifying the core: Protecting its core banking software by enhancing stability and security against cyber threats.

For a firm that was established in 1828, and (in Ruttledge’s words) the “last company on the planet to be using IBM Big Insights,” the Next Gen Technology initiative has been able to realize big results: “We’re the only super regional bank that is completely in the cloud. All of our business apps are either in the Azure or AWS public clouds and we are now in the process of decommitting our data centers in North Carolina,” he shared.

  …

The ‘discovery’ problem in embedded finance – and how OMB Bank found the right fintech partner

For Missouri-based community bank OMB Bank, finding the right fintech partner used to be a slow, manual process. Executive Vice President and Chief of Staff Jessica Sims recalls working from static PDFs of the bank’s preferences, followed by endless back-and-forth emails whenever a fintech expressed interest. The process worked, but painfully slowly, and promising opportunities often slipped through the cracks.

That changed when OMB discovered Backpack, a university payments fintech on Treasury Prime’s AI Marketplace, whose priorities perfectly aligned with the bank’s focus areas. OMB moved quickly, creating a smooth collaboration that benefited both parties.

The ‘discovery’ problem in embedded finance

Embedded finance is a three-way intersection: banks provide the regulated foundation, fintechs handle the tech and integrations, and non-financial brands deliver the end service to users.

Despite all the innovation and growth in the space, one part of the process has seen little advancement: how banks and fintechs find each other in the first place. Behind the scenes, most teams still spend weeks sorting inbound interest, chasing warm introductions, and manually assessing early-stage fit, with many discussions that go nowhere – long before compliance or economics even enter the conversation.

Finding fintech partners that truly align with a bank’s strategy, risk appetite, and operating model remains slow, manual, and opaque – the very problem OMB faced before meeting Backpack.

Treasury Prime believes that discovery, not diligence, is the real bottleneck in embedded finance equations – and that AI is now mature enough to fix it.

In December 2025, Treasury Prime launched its new AI Marketplace to change that starting point. The platform automates partner discovery and accelerates collaboration between banks and fintechs.

“Discovery is where banks feel the most friction,” says Chris Dean, co-founder and CEO of Treasury Prime. “The biggest change is that banks are no longer starting from a blank slate with every new fintech conversation.” 

Chris Dean, co-founder and CEO of Treasury Prime

 

 

 

 


Tearsheet Pro

Trust Bridges Matter: When agentic systems meet payment reality

Agentic commerce, powered by AI agents that anticipate needs and act on a user’s behalf, is beginning to move from theory into practice. These agents promise to reshape commerce end-to-end, from discovery and negotiation through checkout and post-purchase workflows, potentially contributing trillions of dollars to global economic activity by the end of the decade, according to McKinsey. A key constraint, however, might already be at hand.

While AI systems are increasingly trusted to make decisions, they are not yet entirely trusted to share payment credentials directly. It may be early to talk definitively about agentic commerce systems, but this trust gap is already shaping how pilots are designed and how much autonomy is granted to AI in payments.

This article tracks those developments and the implications for commerce at large.

Orchestration moves faster than execution: Generative AI can generate content, surface recommendations, and simulate conversations. Yet the moment money changes hands, whether in checkout, authorization, or settlement, execution is constrained by consumer trust and the need for secure, regulated rails.

Recent PYMNTS data supports this trend: 33.5% of consumers prefer linking a digital wallet rather than allowing gen AI platforms direct access to card credentials or storing them directly. 

This means trust in AI’s decision-making does not automatically extend to moving money – a challenge that emerging agentic commerce


Tearsheet Pro

How Temenos is co-creating AI products with banks, not just for them

Nine months into her role as Chief Product and Technology Officer at Temenos, Barb Morgan is focused on a simple principle when it comes to product strategy: quality over quantity. “We want to build less, but build it better,” Morgan said during a conversation at the Temenos Regional Forum Americas 2025 held May 28-30 in Miami.

Temenos’ approach centers on co-creating meaningful solutions with bank customers rather than rushing to market with multiple products. Morgan emphasized that the company is “really focused on making sure that whatever we put out there is meaningful,” as the industry navigates what she calls the “AI hype curve.”

Morgan’s insights reveal why many banks struggle with AI adoption despite the technology’s promise. The real barriers aren’t about computing power or algorithms — they’re messier problems involving decades-old data systems that were never designed for AI and organizational cultures that haven’t caught up to the pace of technological change. 

Her conversation also detailed Temenos’ bet on bringing innovation closer to customers, such as through its new hub in Orlando designed for co-creation, and why the company is taking a strategic and deeply integrated approach to AI that enables banks to deploy AI-powered solutions faster and safer.

Listen to full podcast

A three-pronged AI strategy

Temenos has structured its AI approach around three core components: Gen AI embedded directly into its platform and products, agentic AI with a first solution for sanctions screening already live at one Tier-1 bank, and an AI studio for custom use cases. “We have a lot of customers coming to us with very unique use cases, and so we want to provide them a platform that’s pre-built with banking modules,” Morgan explained.

The company’s focus on embedded AI addresses a common industry challenge. “Having it embedded, versus our customers trying to figure out how to bolt it onto our product, is really important to us,” she said. This approach allows banks to access AI capabilities without investing too many resources into integration.

Banks are ready for AI – their data isn’t

One of the biggest obstacles to AI adoption isn’t fear of technology, but foundational data issues, shares Morgan. “A lot of banks over the past 10 to 15 years went through this huge digital transformation, but what they didn’t transform was the data in the back end,” Morgan noted. “In order to leverage the power of AI, you have to have your data clean.”

This reality has shifted many of Temenos’ client conversations toward data readiness rather than AI capabilities. “Our clients also want to leverage their own data systems. So how clean is your data? Is it really ready? Because for secure AI products, you have to have your data in order,” she said.

Cultural change is the harder challenge

Beyond technical hurdles, banks face significant organizational resistance to AI implementation. “I was talking with one of our US banks last week, and he said I underestimated the amount of cultural change that’s necessary, because so many people are afraid of AI,” Morgan shared.

The fear stems from job displacement concerns rather than technological limitations. “They think it’s going to take my job away, versus thinking of it as augmenting their job and being more of a side by side partner,” she explained. This cultural aspect has to become a major focus for banks that want to succeed with their AI implementations.

A gradual approach to AI deployment

Temenos’ strategy acknowledges these cultural and technical challenges by allowing banks to phase in AI adoption. Morgan described how one tier-one bank using their agentic AI product FCM AI Agent started with just 5% of its traffic, then gradually increased it to 20%. “It wasn’t because they didn’t trust the technology. It was because they were getting the rest of the organization comfortable,” she said.

This incremental approach extends to customer-facing applications as well. “A lot of people, it seems, have their favorite [Gen AI] tool on their phone,” Morgan observed. “I think maybe the banks have underestimated that the customers are actually ready to interact with AI.”

Bringing innovation closer to customers

Part of Temenos’ US expansion includes the opening of its Orlando Innovation Hub, designed specifically for co-creation with bank customers. “Instead of just expanding one of our existing offices, we’re actually going into a brand new building,” Morgan said. “It’s all about being able to do the design workshop, but then the space can transform to doing co-development together.”

The facility will include spaces that can replicate bank branch environments. “There’s a space where we can make it feel like you’re walking into the branch of the bank, and so we can actually recreate exactly what it’ll feel like for their customers,” she explained.

Market-centric over centralized delivery

The Orlando hub represents a broader shift in Temenos’ delivery model. “Over the past 30 years, we have had a pretty centralized delivery team, and this is about bringing it closer to our customers,” Morgan said. “Versus centralized delivery, it’s more about market-centric innovation.”

This approach is driving the firm’s hiring, with plans underway to recruit for 200 positions at its Orland Innovation Hub. “At a recent hiring event, every candidate who received an offer accepted,” Morgan noted. “They were really excited about the co-innovation and the ability to actually work how we want and bring our best selves.”

Building products that actually ship

Morgan has instituted a new discipline around product announcements, moving away from proof-of-concepts toward deliverable solutions. “We’re only going to announce things when they’re live and ready to use now,” she said.

The company has also allocated 25% of its development capacity specifically to customer-driven features. “We’ve actually allocated about 25% of our capacity to just listening to customers and putting their needs on top of what we would already have planned,” Morgan explained.

This customer-centric approach extends to the broader organizational transformation Morgan is leading. 

‘Trust me, I’m an algorithm’: How fintech is rebuilding customer confidence in the age of AI

The financial services industry has always been built on trust. Artificial intelligence is editing the rulebook on what that means. As banks and fintechs are pushing to deploy AI across everything from fraud detection to personalized recommendations, they’re discovering that customers’ definition of trustworthiness has evolved far beyond traditional metrics like security and reliability.

Today’s consumers want to know not just that their money is safe, but how algorithms are making decisions about their financial lives. They’re requesting transparency about data usage, explainability in AI-driven recommendations, and proof that these powerful new tools actually serve their interests, not just institutional bottom lines.

We asked industry leaders across financial services, fintech, and their supporting ecosystem how they’re navigating this new trust landscape. Their responses reveal both the complexity of the challenge and the emerging strategies that are actually working.

The new trust equation

The numbers tell a stark story about consumer sentiment. According to recent research from Accenture, while banks remain the most trusted entities for protecting customer data, 84% of customers are concerned about how that data gets used. Even more telling: only 26% are comfortable with extensive AI usage for data analysis, even when it promises better personalization.

“Today’s customers are no longer just evaluating institutions on performance — they’re scrutinizing how their data is used, how decisions are made, and whether emerging technologies like AI act in their best interests,” explains Monica Hovsepian, Global Senior Financial Services Industry Lead at OpenText. “This shift demands a new trust contract: one built not only on accuracy and speed, but on transparency, explainability, and ethical AI deployment.”

The message is clear: personalization must be transparent and demonstrably beneficial. Financial institutions can no longer assume that faster, smarter service automatically equates to better customer relationships.

Beyond the algorithm: Human-centered AI

For companies serving underbanked populations, this trust challenge carries additional weight. Kelly Uphoff, CTO at Tala, emphasizes that AI innovations must solve real customer problems while protecting dignity and identity. “Not all customers will be dazzled by AI unto itself,” she notes. “The technologists building these new solutions don’t often come from the communities we serve.”

Tala’s approach involves co-creating technology with customers from day one: showing early prototypes, listening to pain points, and incorporating feedback throughout development. They’ve also made hiring from the communities they serve a priority, creating a diverse workforce that better understands customer needs.

This human-centered approach echoes across different sectors of financial services. As Taran Lent, CTO at Transact + CBORD, puts it: “AI doesn’t replace the human relationships at the heart of meaningful engagement, it enhances them by making every touchpoint more relevant, timely, and personalized.”

The fraud fighter’s dilemma

Most likely, nowhere is the AI trust challenge more acute than in fraud prevention, where the technology serves as both weapon and shield. Parilee Wang, Chief Product Officer at Alloy, describes navigating AI from two sides: “It’s being used both as a tool for fraudsters and a tool for fraud fighters.”

While generative AI has enabled fraudsters to scale attacks like synthetic identity fraud, Wang argues that the real innovation lies in moving beyond detection to action. “An AI tool that alerts you to fraud without taking action is like a home alarm that goes off when someone breaks in. If it doesn’t call the police or lock the doors, what’s the point?”

Yinglian Xie, CEO and co-founder of DataVisor, sees AI transparency as critical to maintaining customer trust in fraud prevention. “The ability to explain and verify how AI systems work and the data that drives their decisions is of utmost importance,” she explains. The most effective approaches leverage AI to increase fraud detection while ensuring frictionless customer experiences, proving that security and convenience can be complementary rather than competing priorities.

Practical trust-building strategies

Many concrete trust-building strategies are emerging from early AI adopters in financial services:

i) Label and explain: Public’s approach involves clearly marking all AI-generated content and emphasizing the need for independent verification. “By clearly indicating that content is AI-generated and emphasizing the inherent risks associated with such outputs, we help our members understand what they’re using,” says Rachel Livingston, Director of Communications at Public.

ii) Value at every interaction: Scott Mills, President of William Mills Agency, advocates for using AI to provide consistent value: answering customer inquiries, explaining complex situations, and offering tailored solutions. The key is eliminating friction while adding genuine utility.

iii) Human oversight by design: Derek White, CEO of Galileo Financial Technologies, emphasizes that there’s no “set it and forget it” approach to AI in financial services. “AI applications are only as good as the data that goes into them, and the human oversight and strategy used to guide and deploy them.”

The content and communication challenge

As AI impacts how customers seek information, traditional marketing and communication strategies need updating. Anna Kragie, Account Director at The Fletcher Group, notes that with large language models changing how people look for answers, brands need “a smart AI content and PR strategy centered on content that builds trust with customers.”

This means pivoting toward more authentic, conversational content that directly answers buyer questions, while using media relations to establish authority on high-credibility news sites. In an environment where AI can generate massive volumes of low-quality content, human curation and authentic expertise become more valuable, not less.

Finding the balance

The self-driving car analogy keeps appearing in these conversations, and for good reason. As Brandon Spear, CEO of TreviPay, explains: “Just as autonomous vehicles require human oversight, AI-driven banking solutions must strike a balance between automation and necessary human intervention. The goal is not to replace human judgment but to enhance it with data-driven insights and improved efficiency.”

This balance requires what Transact + CBORD’s Lent calls “robust AI governance frameworks”, clear standards and best practices for both internal teams and vendors, combined with responsible piloting and focus on measurable outcomes over hype.

The trust dividend

Financial institutions that get this balance right stand to gain a significant competitive advantage. As Hovsepian notes, “In a digital-first world, where convenience is expected, trust has become the true differentiator, and the most valuable asset any financial institution can earn.”

The companies building trust in the age of AI are embedding security, privacy, and fairness into their AI models from the ground up, then communicating these efforts clearly to customers. They’re working to prove that AI can enhance rather than replace human relationships, and that transparency doesn’t have to come at the expense of innovation.

The financial services industry has always been in the trust business. AI isn’t changing that fundamental reality – it’s just raising the bar for what earning that trust requires.


This article features insights from members of Tearsheet’s monthly PR/Comms Working Group serving the best professionals in financial services and fintech. Contributions came from both in-house communications leaders and agency executives who represent major players in the financial services sector.

Become a member of Tearsheet’s monthly PR/Comms Working Group — reach us here.

The double-edged sword of Gen AI: Harms and risks for consumers and employees and why nobody talks about it

For our dedicated content series on Gen AI in financial services, we have had some of the biggest names in the industry speak to us about use cases that are unlocking pools of revenue and increased efficiency for these firms. These conversations have focused on Gen AI’s work in the back office at the biggest banks and fintechs in America, and how hundreds of teams across the industry are using the tech for tasks like software development, customer service, and summarization. 

But missing in these conversations is a deep and serious discussion on the risks and harms that can come with adopting Gen AI. In this article, I break down why the industry doesn’t like to talk about the potential harm from using Gen AI and what these risks even are. 

Why nobody talks about potential Gen AI harms 

Bad press: Gen AI adoption is allowing companies to position their brand as tech-forward and cutting-edge. External facing conversations on potential harms and risks do not make for good marketing, especially in a climate that is convinced of Gen AI’s capability to propel us into a new future. 

The financial industry is responsible for people’s money, and so these companies often have to prioritize an image of safety that bolsters people’s trust, and discussions that undermine this image are perceived as harmful to this marketing play. 

AI is complicated: Digital literacy is critical to understanding how AI works and its possible implications. While most AI practitioners are well aware of AI’s “black box” nature and the complex algorithmic overhead that goes into making AI algorithms explainable, consumers as well as non-tech bank employees may not have the same interest in understanding what AI is, how it works, where it can break, and how it impacts their lives. 

Products and features which are layered with user-friendly UX are much more approachable and demonstrate tangible value when used. Dedicating hours to understanding how the backend works is a harder goal to justify to board members, employees, and customers with likely no short term advantages other than building a more aware community. 

Gen AI is new: The novelty of Gen AI impedes the construction of sophisticated federal and state level regulations and sufficiently proactive company policies. This means financial leaders have no choice but to keep pace with competitors, adopt Gen AI, and watch their deployments closely for signs of harms and risks. The limited information in the market and vacuum of regulations on education, misuse, consumer and employee protection regarding AI does not encourage open conversations. 

Despite the reticence in the industry to openly discuss potential harms and risks, one can make a pretty good argument that such a conversation is absolutely critical to the Gen AI-fueled utopia the industry is dreaming to build: Organizations willing to lead real conversations have a chance to position themselves as thought leaders and, more importantly, may be able to coax the industry into coming out of its silos and collaborate to build industry-wide standards that can help mitigate potential lawsuits and harms faced by consumers and employees.

What are these potential harmful impacts I’m referring to? There are quite a few, but covering each one in one article is nearly impossible, so I’ll include the ones that have the closest ties to use cases already active in the industry. 

AI’s bot sized problem(s) for FIs

“Generative AI agents threaten to destabilize the financial system, sending it swinging from crisis to crisis,” writes the Roosevelt Institute. Gen AI tools are available to everyone, including bad actors that can use it to defraud customers, launch cyberattacks on FIs, and execute strategies to manipulate the market.

Moreover, an organization’s internal tools have the capability to subject customers to discriminatory behaviors, privacy breaches, and hallucinations, as well. Considering that many FIs are currently using Gen AI in the back office, similar adverse effects can be experienced by employees, too. 

The Gen AI powers that be: “The provision of AI agents may be an oligopolistic market, if not a natural monopoly” according to the Roosevelt Institute. This means that FIs that want to adopt Gen AI may face higher prices with the impetus for continued innovation being relatively low. It also means that bad actors can concentrate on these providers and exploit single points of failure that may expose an array of organizations and their customers to malicious activities. 

Conflicts of Interest: The market is obsessed with agentic AI. But it’s unclear whose interests these agents will act on behalf of if two negotiating parties are using the same agent. Moreover, if multiple Gen AI agents are drawing from the same data bank, they run the risk of reacting to market conditions in identical ways, opening up chances for algorithmic biases against certain products. They may also encourage large groups of customers to act in a similar manner, which can lead to bank runs or stock market crashes.

How consumers and employees maybe at risk due to Gen AI

Plain old vanilla AI has been reported to make decisions that can lead to discrimination in credit decisioning algorithms. In 2022, Lemonade wrote in its 10-Q that its “proprietary artificial intelligence algorithms may not operate properly or as we expect them to, which could cause us to write policies we should not write, price those policies inappropriately or overpay claims that are made by our customers. Moreover, our proprietary artificial intelligence algorithms may lead to unintentional bias and discrimination.”

This risk does not disappear with Gen AI. While a broad infusion of Gen AI in the credit decisioning process has yet to become commonplace, without stringent policies on what data Gen AI can or can not use, and how its decisions and outcomes will be governed, the industry has yet to build tools that will help prevent systemic discrimination against certain types of consumers barring them from accessing credit. 

“Nonbank firms like financial technology (fintech) companies, which are already subject to significantly more permissive regulations than banks, may be especially inclined to deploy AI in assessing customer worthiness for their products,” writes the Roosevelt Institute, a sentiment which is in line with industry behaviors, where fintechs have been much faster at adopting and launching Gen AI-facing features like chatbots and dedicated Gen AI to research stocks. 

It’s (not) a fact: Consumers and employees are also at the risk of being impacted by hallucinations. Although the biggest banks in the industry have yet to launch consumer-facing chatbots, most are now coming on record to talk about the productivity gains their employees are experiencing by using internal Gen AI chatbots. 

The most commonly cited use cases are customer service agents using Gen AI to quickly access answers to customer questions, technology teams using Gen AI tools for software development and code conversion, and team-agnostic tools that help employees access company policies regarding day to day questions about processes. 

The issue here is that it is unclear how these firms respond when employees take the wrong action based on the information they receive from Gen AI agents. 

The question we need to ask is this: Is it enough to say that “Gen AI can make mistakes, so please double check the answers to ensure accuracy” when a whole marketing engine is dedicated to positioning these tools as “time-savers” and their users lack the digital competency to understand the tools they are using? 

Sidebar: Gen AI in credit unions

We have extensively covered how the biggest banks are activating Gen AI use cases to benefit from efficiency and productivity gains. But smaller institutions are also hopping onto this train. We heard from two industry players

  1. Commonwealth Credit Union: Recently, the $2.5 billion, Kentucky-based CU decided to fill in this gap by integrating a tool by Zest AI called LuLu Pulse, which uses Gen AI to consolidate multiple data sources like NCUA Call Reports, HMDA, and economic data. This ultimately allows lenders to gain insight into how their products and services compare to their peers by querying the platform.
  2. Duke University Federal Credit Union (DUFCU): The firm is experimenting with how the new tech can enable it to expand reach and build a stronger marketing funnel. It recently integrated Vertice AI’s copywriting tool called COMPOSE. 

For DUFCU’s Director of Marketing Jennifer Sider, purpose-built tools focused on the financial services space offer her a significant advantage over free Gen AI tools available to the public. It’s also better than the manual alternative of managing the whole copywriting process alone.