OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil

Kevin Weil, CPO at OpenAI, explains that the AI models we use today are the 'worst' we'll ever use, constantly improving within months. This rapid advancement means that product development in AI requires a completely different mindset compared to traditional technology, where core technologies are more stable. He highlights that the boundaries of AI capabilities are always shifting, making it an exciting and challenging field.

Weil discusses his biggest career disappointment: the Libra cryptocurrency project at Facebook (later Novi). He explains that the goal was to enable free and instant money transfers via WhatsApp and Messenger, addressing the regressive fees in remittances. Despite his belief that the world would be better with Libra, the project faced challenges due to its ambitious scope (new blockchain, basket of currencies, integration into two major platforms) and Facebook's reputation at the time, ultimately leading to its failure to launch as intended. He notes that the underlying technology lives on in other blockchain companies.

Kevin shares a prompting tip: provide examples within your prompt to 'poor-man's fine-tune' the model, teaching it contextually what you expect. He also notes the effectiveness of persona-based prompting (e.g., 'You are Einstein...') to shift the model's mindset. He clarifies that while prompt engineering is currently useful, it shouldn't be a long-term requirement as models become more intuitive. He encourages users to provide feedback on OpenAI's products, emphasizing his active engagement on platforms like Twitter for continuous improvement.

Kevin recounts his swift recruitment process to OpenAI. He highlights the distinct working environment compared to his previous roles at Twitter, Facebook, and Instagram. The key differences are the unprecedented pace of AI advancements and the continuous evolution of the underlying technology. Unlike traditional tech where foundational elements like databases are relatively static, AI capabilities leap forward every few months, demanding constant adaptation and a flexible product development approach. This rapid change makes it challenging to plan long-term roadmaps, as the technology often outpaces initial expectations.

Weil emphasizes the growing importance of 'evals' (evaluations or tests) for product managers and builders in the AI era. He describes evals as quizzes for AI models, assessing their proficiency in various tasks, similar to unit tests in software development. Understanding an AI model's performance accuracy (e.g., 60% vs. 95% vs. 99.5%) is crucial for designing appropriate products. He illustrates this with OpenAI's 'deep research' product, where specific evals guided the fine-tuning of models to improve performance on complex research tasks. Effective evals allow developers to continuously improve models and define the boundaries of what AI can achieve in specific use cases.

Weil addresses concerns from AI founders about competing with foundational model companies like OpenAI. He explains that OpenAI's strategy focuses on building a robust API and enabling a vast developer ecosystem. Despite OpenAI's ambition, he believes that the sheer number of potential AI applications across diverse industries means they cannot build everything themselves. He emphasizes that significant opportunities exist for startups to fine-tune foundational models with industry-specific or proprietary data, creating tailored AI products that OpenAI or other large model providers won't pursue directly.

OpenAI prioritizes rapid iteration and a 'bottoms-up' approach to product development. While they set overall strategic directions, detailed roadmaps are kept lightweight due to the fast-changing AI landscape. Weil highlights the philosophy of 'iterative deployment,' where products are shipped early and often, learning and evolving in public with user feedback. This approach minimizes the need for extensive upfront scaffolding around model imperfections, as new, better models are constantly on the horizon. He also touches on the company's flexible review processes, ensuring that product launches are not bottlenecked by rigid approvals.

Weil challenges the common belief that chat interfaces for AI are a temporary solution. He argues that chat is a remarkably versatile and universal interface, mirroring how humans communicate. Just as we use spoken and written language to interact with varying levels of human intelligence, chat allows for flexible, unstructured communication with LLMs, accommodating their ability to understand complex human nuances. While specialized, more rigid interfaces might be better for high-volume, specific tasks, chat provides an essential baseline for open-ended, adaptable interaction with AI that leverages its core strengths.

Weil predicts that in the future, product teams will increasingly integrate researchers and machine learning engineers, even outside of foundational model companies. He explains that fine-tuning models for specific use cases will become a core workflow for building most products, necessitating specialized expertise within product teams. At OpenAI, there's a strong emphasis on collaboration between research and product, where they iteratively develop and fine-tune models based on user needs and evaluations. He also notes that successful PMs at OpenAI thrive in ambiguity, possess high agency, and lead through influence rather than strict hierarchy, guiding product development in a highly dynamic environment.

Weil discusses the profound changes AI is bringing to work, particularly for product teams. He feels that current AI integration into daily workflows is still nascent, despite widespread use of tools like ChatGPT for summarization and document generation. He advocates for more radical transformation, such as 'vibe coding'—using AI to rapidly generate prototypes and demos instead of traditional design tools. This approach allows for quick exploration of ideas and proof-of-concepts, accelerating product development and making even complex tasks accessible to non-engineers. He emphasizes that AI is making traditional skills more efficient rather than replacing them, pushing for more experimental and agile development practices.

Kevin expresses strong optimism regarding AI's long-term impact, viewing it as a driver of economic, geopolitical, and quality-of-life advancements, similar to historical technological progress. While acknowledging potential temporary disruptions and individual impacts, he believes society must proactively manage these transitions. He highlights AI's potential in personalized tutoring, a field he feels is still underdeveloped despite the models' capabilities and accessibility. He envisions AI democratizing creativity, citing examples like Sora enabling filmmakers to explore numerous creative options at low cost, leading to better final products rather than replacing human ingenuity. He asserts that the continuous improvement in AI models’ intelligence, speed, cost-efficiency, and safety means that 'the AI models you’re using today are the worst AI model you will ever use for the rest of your life.'

Kevin recommends "Co-intelligence" by Ethan Mullik (on AI and daily use), "The Accidental Superpower" by Peter Zeihan (on geopolitics), and "Cable Cowboy" about John Malone (for business insights). He mentions enjoying "Top Gun 2" for its American patriotism. For a favorite product, he highlights Waymo's self-driving cars for providing a consistent 'future' experience, and tools like Windsorf that enable 'vibe coding.' His life motto, inspired by Mark Zuckerberg, is "good work consistently over a long period of time," emphasizing consistent effort over seeking quick wins.