Continuous Integration, Delivery, Deployment
I was challenged to ideate a Program with its Project processes for proving a Continuous Integration, Delivery, and Deployment model viable for a customer. Upon delving into the topic, I discovered many differing opinions, thoughts, acceptance, and resistance to this concept.
Most information was garnered through elicitation; conversations with specific purposes, collecting information that was not readily available nor wanted to be given freely in order to document the state of readiness for teams and their applications’ continuous readiness.
The Goals were many and simple to articulate, yet required a ‘new’ culture to develop and be empowered.
- Provide Continuous Value
- Continuous Planning and Intake
- Continuous Integration
- Continuous Delivery
- Continuous Deployment
- Continuous Customer Feedback
- Reduce Total Time for Deployments
- High reliance on tooling and automation
- Autonomous deployment; Less reliance on people
- Integrated processes adapting code to infrastructure ensuring predictable delivery
- Automate Change Control Review Board process (CCRB)
- Reduce Deployment Size to Reduce Risk
- Drive velocity of business features to Production delivery
- Drive throughput through smaller business deliverables to Production
- Simplify diagnosis of breakages and defects
- Put the Customer First
The Objectives were many and simple to articulate as well.
- Standardize Launch Orchestrations
- Simplify and standardize Launch Orchestration Documentation
- Simply and standardize Defect and Launch Feedback
- Simplify and standardize Build Deployments Patterns
- Remove manual steps from code and feature Deployments
- Automate Smoke Testing
- Automate Design Validation Testing (DVT)
- Automate Engineering Validation Testing (EVT)
- Streamline User Acceptance Testing (UAT)
- Streamline Production Validation Testing (PVT)
- Reduce Investment for all Deployments
- Leverage standardized launch deployments
- Canary Deployment
- Blue/Green Deployment
- Reduction of deployment time from days to minutes to multiple deployments per day
And the Benefits were as many and varied too; Customer oriented and Organization efficiency.
- Buyer / Seller
- Greater Customer Satisfaction that needs are met, continuously
- Business Outcomes
- Faster deployment of customer-specific business needs
- Greater customer satisfaction from increased usability
- Organization & Development Teams
- Detection of integration errors as quickly as possible
- Significantly reduced integration problems
- Development of cohesive software more rapidly
Upon a deep research and analysis, submittal of a supporting white paper and gaining approval, I was tasked with directing and managing the scope, design, architecture, and development-to-test-to-production, timeframe, and product owner interactions.
I, along with two (2) Operational Architects (one (1) Networking Architect and one (1) Systems and Applications Architect), started a POC to automate a single application. The bottom line result: reduction from a ‘standard’ 22+ day build-to-deployment to an average 22 minute build-to-deployment without defects.
The goal of a Canary Deployments is to push changes or a ‘Canary’ (a specific code or set of features) to a Production environment affecting a small number of (end) users ensuring that the changes are transparent and viable.
We chose to use the ‘Canary Deployment’ technique for its ability to ‘curve out’ groups of servers for new code deployment and to enable slow ‘ratcheting’ up of traffic on the ‘curved-out’ servers over a prescribe time frame.
The ‘Canary’ is incrementally increased (ratcheted and regulated) to the end-user population. Because the ‘Canary’ is initially pushed to a small number of users through controlled increments, its impact is considered relatively small. The section of users utilizing the new code, having not volunteered to test anything and therefore unaware of the new code other than possible UI changes, provide an early warning for potential impacts and defectives and the opportunity for quick rollback in case of previously undiscovered application defective(s).
Canary Deployment Patterns vary as to organizations. However, the deployment pattern is an incremental usability methodology.
- Dark Deployment
- Deploy to a small set of servers
- Limit user traffic to servers
- User unknowingly test the deployment
- Deployment increments to a large set of servers
- Light Deployment
- Unrestricted user traffic to servers
‘Blue-Green Deployment’ is a release technique that reduces downtime and risk of application deployment by running two identical Production environments (‘Blue’ and ‘Green’) with only one of the environments active (live) at a time with; the live environment serving all production traffic. If something unexpectedly occurs within a new release, the environments can be quickly rolled-back to the previous versioned environment.
After a new software release is prepared, deployed, and tested in the current non-live environment, the Production environments are switched (via router, load balancer, proxy, etc.) so that all incoming traffic (requests) move to the new environment (formerly the non-live Production environment).