Build a DevOps Culture: Microsoft's Journey to adopt an Agile Mindset and DevOps culture - Gousset

Mickey Gousset, a Microsoft DevOps Architect, introduces the session on building a DevOps culture, focusing on Microsoft's journey, particularly the Azure DevOps engineering team's adoption of an Agile mindset and DevOps culture. He defines DevOps as the union of people, process, and products, with people and processes being the most critical elements, emphasizing the importance of culture.

Before Satya Nadella's leadership, Microsoft's organizational structure was fragmented, with different teams like Windows and Office operating independently, using varied tools and processes. This led to inefficiencies and difficulties in moving engineers between teams. Nadella introduced the 'growth mindset' and the 'One Engineering System' concept to standardize tools and processes, aiming for increased productivity across the company.

The Azure DevOps journey began with Team Foundation Server as a boxed product in 2005. Microsoft then ventured into software-as-a-service, starting development in 2010 with three-week sprints, leading to a preview in 2012. The company has continuously iterated, releasing updates every three weeks for a decade. Concurrently, Microsoft significantly increased its involvement in the open-source community, becoming a large contributor to GitHub and eventually acquiring it.

The first habit is being customer-obsessed. Microsoft focuses on listening to customers quantitatively and qualitatively through various channels like Developer Community, social media, and direct feedback on the Azure DevOps site. They track metrics such as acquisition, retention, NPS scores, deployment speed, and live site incident resolution. The 'build, measure, learn' approach is used for new features, acknowledging that failure is acceptable if lessons are learned, as exemplified by the 'Team Rooms' feature that was eventually removed due to lack of adoption.

A production-first mindset is crucial for a 24x7 service. A feature is considered 'done' only when it's live in production, collecting telemetry, and the data confirms its value. The team has evolved its handling of Live Site Incidents (LSIs), with Designated Responsible Individuals (DRIs) from feature teams being on call and having strict response times. All LSI activities are tracked in an internal incident management system to ensure issues are resolved and prevented from recurring. Transparency about problems, including providing an accurate service status page and post-mortems, helps build trust with customers. Partial automation in deployments is avoided, as it often leads to errors; all deployments are fully automated using version-controlled PowerShell scripts and Azure Pipelines, with consistent deployment methods across all environments.

Security is paramount and is integrated earlier in the development cycle, known as 'shifting left.' Microsoft utilizes red and blue teams for constant security testing, and all engineers are expected to contribute to security awareness. Efforts are made to prevent vulnerable code from being checked into version control, such as connection strings with credentials or reliance on vulnerable open-source packages. Tools like GitHub’s dependency management help identify and address issues before they are merged into the master branch.

This section emphasizes balancing team autonomy with enterprise alignment. Drawing a comparison to Wikipedia's success over Microsoft Encarta, the importance of 'autonomy, mastery, and purpose' for team motivation is highlighted. Microsoft's 41 feature teams operate with 'aligned autonomy,' adhering to common roles, cadences, and taxonomies, but having freedom in their specific planning and practices. Teams are self-managing, typically comprising 10-12 people, and own their features from inception to production. A unique 'sticky note exercise' allows employees to choose their teams every 12-18 months, promoting internal mobility and knowledge sharing. Planning is influenced by OKRs (Objectives and Key Results), which are transparent and define ambitious objectives with measurable key results. Communication is maintained through 'sprint mails' and regular OKR reviews to ensure alignment and progress.

To go fast without breaking things, Microsoft implemented a 'bug cap' for feature teams; exceeding this cap means the next sprint is dedicated solely to bug fixing, significantly improving quality. Branching strategy shifted to Git with a release flow and trunk-based development. Short-lived feature branches are continuously merged with master, and 'feature flags' are used to hide unfinished code. Pull requests are critical for quality assurance, requiring numerous unit tests and manual approvals before merging to the main line. The 'green means green, red means red' policy enforces strict test pass requirements. The testing strategy also evolved from slow end-to-end tests to extensive, fast-running unit tests (L0 and L1), with a commitment to writing unit tests for all new or touched code.

Recognizing that deployments won't always be perfect, the focus is on safe deployments and limiting 'blast radius.' Principles include consistent deployment methods across environments, maintaining quality signals, achieving zero downtime, and deploying during working hours. Feature flags are crucial for decoupling deployment from exposure, allowing features to be deployed silently and activated later, or quickly deactivated if issues arise. To mitigate risks, progressive exposure is used, deploying to multiple rings (canary, specific data centers) in stages. ChatOps tools like 'As DD' bot are used for real-time monitoring and management of deployments. The speaker also discusses the 'J-curve' of DevOps transformation, a period of initial quick wins followed by a 'valley of sorrow' as organizational culture and processes change, encouraging persistence through this difficult phase. The long-term benefits are evident in Microsoft's increased feature delivery velocity over the years.

Mickey addresses questions regarding the future of the on-premise Azure DevOps Server, confirming its continued existence with updates every three months, typically three months behind the cloud version. He clarifies that Azure DevOps and GitHub remain distinct products. He also discusses the dynamics of team changes and management structure at Microsoft, where developers can change teams, potentially leading to new managers. He acknowledges that some prioritization conflicts can arise between teams and management, which are typically resolved through internal discussions and justification.