In the race to implement AI solutions, many organizations skip the most critical step: defining the actual business problem. "It's very tempting to think that the answer is generative AI, no matter what the question is," cautions Keyvan Azami, Engineering Manager at Google.

"Start with the problem."

After eight years leading ML implementation for Google's internal operations, Azami has developed a methodical approach that cuts through the hype. His team's experiences offer valuable insights for any organization looking to effectively integrate AI into their workflows and build better products.

The Reality of ML Implementation: A Case Study

Google handles millions of IT support inquiries annually from its 180,000+ employees. Azami's team was tasked with automatically surfacing relevant support articles when employees raised tickets aiming to solve problems instantly rather than making users wait hours for human assistance.

This seemingly straightforward project reveals why ML implementation isn't linear. Unlike traditional software development, machine learning projects require constant iteration, failure tolerance, and comfort with uncertainty key principles for successful product development.

"ML is probabilistic," Azami explains. "If somebody says, 'I need 100 percent guarantee that you give me the right answer,' I can't do that project."

A Framework for ML Success

Azami's approach centers on five key phases, each with specific challenges:

1. Ideation and Prioritization

Before diving into technical solutions, Azami's team evaluates:

• The clear business problem (not the ML technique)
• Potential impact (in this case, thousands of hours saved)
• ML suitability (available data, risk tolerance, measurable success)

This user-centered design approach ensures that product strategy addresses actual user needs. The initial assessment showed promising potential. With millions of resolved tickets, approximately 10% linked to helpful articles representing tens of thousands of potential automation opportunities.

2. Data Acquisition

The team focused on validated responses tickets that were successfully resolved with article links. This crucial filtering step is often overlooked in ML projects but is essential for creating a solid foundation for product innovation.

3. Data Exploration

"We spend quite a bit of time here, so we don't jump into the solution," Azami emphasizes.

Using tools like Google Colab, they examined the relationship between ticket features and article recommendations. This interdisciplinary phase requires more than just engineering UX researchers and product teams provide crucial insights on capability identification. This cross-functional collaboration brings together different perspectives to enhance both user experience (UX) and developer experience (DX).

4. Model Prototyping

The initial approach used a dual encoder model to map tickets to articles. The results were disappointing: 40% precision (accuracy of recommendations) and only 15% recall (percentage of potential matches identified).
This moment represents the critical difference between ML and traditional development. Rather than pushing forward, the team stepped back and questioned:

• Is the problem with our features?
• Should we try different techniques?
• Is the problem itself framed correctly?

5. Iteration and Reframing

The team returned to ideation, shifting from finding any article to an 80/20 approach: solving the most common problems first. They also prioritized precision over recall ensuring recommendations were right, even if they couldn't catch every opportunity.

A crucial discovery came through stakeholder engagement: their training data contained incorrectly labeled tickets. The solution? A focused relabeling of 2,000 tickets by expert staff a prime example of lean product development prioritizing quality over quantity.

"Even with 2,000 tickets, you can make a dramatic improvement," Azami notes. The precision jumped to 80% a significant win in the ML world.

Beyond Model Building: Production Challenges

Azami emphasizes that ML code is just a small part of a production system. Additional considerations include:

• Infrastructure - Building deployment pipelines and monitoring systems
• Drift monitoring - Unlike traditional software, ML models degrade even without code changes as data distributions shift
• Continuous retraining - Regular update cycles to maintain performance

When COVID hit, for example, support questions shifted dramatically toward remote work issues, degrading model performance until retraining. This situation highlighted the importance of agile product management in responding to changing user needs.

Key Takeaways for Effective ML Integration

After years of implementing ML solutions, Azami offers several principles for teams:

• Structure your approach - Follow a methodical process rather than jumping straight to coding
• Embrace iteration - Be comfortable with failure and learn quickly
• Fail fast - Celebrate learning experiences and abandon unproductive paths early
• Use the right tools - ML requires different toolsets than traditional development
• Start simple - Begin with rules-based approaches before jumping to ML
• Involve designers early - UX designers should be involved "day one" to address how uncertainty is presented to users, creating the foundation for happier product teams

Human-Machine Collaboration, Not Replacement

Perhaps most importantly, Azami frames ML as assistive technology, not autonomous replacement. "It's people with machines, not people versus machines," he emphasizes.

This human-in-the-loop approach acknowledges the probabilistic nature of ML while maximizing its benefits. By establishing clear metrics (precision and recall), teams can communicate uncertainty appropriately and set realistic expectations a key aspect of product leadership.

Looking Forward

As ML models grow increasingly sophisticated, the approach Azami outlines becomes even more valuable. The fundamental challenges problem framing, data quality, and human integration remain consistent even as the underlying technology evolves.

By applying a structured approach, embracing uncertainty, and focusing relentlessly on the business problem, organizations can move beyond AI hype to achieve meaningful results. The key is starting not with the latest algorithm, but with the question: what problem are we really trying to solve? This focus on design thinking principles leads to customer-centric development.

Final Thoughts:Turning Data into Decisions

Google's ML journey is a masterclass in strategic AI integration. By focusing on real-world problems, iterating on solutions, and involving humans at every step, Azami's team transformed IT support efficiency through empowering teams and collaborative product teams.

But the lessons here aren't unique to Google. Whether you're building your first ML model or scaling an existing solution, the principles remain the same: start with the problem, iterate often, and always keep the end-user in mind.
Ready to take the first step? Watch Keyvan Azami's full talk on Strategic AI Integration: https://uxdx.com/session/strategic-ai-integration-in-engineering-teams1/

and download our report: https://uxdx.com/post-show-report/