IT organizations today are faced with the challenge of delivering new capabilities at higher velocity to enable digital transformation. While enterprise resource planning (ERP) systems provide a stable integrated foundation and emerging capabilities to meet business needs, their integration and deployment processes were built for a waterfall methodology.
As business agility is paramount to market success, Intel IT works to move to an agile and DevOps methodology to deliver new capabilities at high velocity while maintaining stability at optimal cost. This program built additional capabilities on top of SAP Solution Manager to enable self-service, continuous integration (CI) and continuous delivery (CD). The velocity of self-service has improved by 341 percent while decreasing deployment costs by 45 percent, with no change induced incidents. The quality of integration and deployments was improved, while the complexity of the integrated environment increased. The maturity level for DevOps adoption has been improved and 97 percent of our agile teams have L3 agile maturity with plans to reach L4 within a year.
The features designed to support DevOps included:
- Enabled a complex SAP landscape
- Enabled self-service by bundling needed dependent objects
- Provides adherence to the orchestration of change and release management steps
- Full audit traceability
- Auto rollback capability
- Architected with full DevOps CI/CD
The Agile and DevOps Program Transformation team did an industry scan and conducted detailed vendor evaluations to meet these needs. Most of those tools available introduce redundant change management functions currently provided by SAP Solution Manager. Due to the lack of feature-rich tools to meet the demands of the DevOps culture, Intel created the Enterprise Deployment Automation Tool (EDAT). EDAT solves the problem by providing self-service continuous delivery capabilities to development teams and enabling high velocity while leveraging SAP Solution Manager.
EDAT is a custom-built ABAP Web Dynpro program that integrates Solution Manager with self-service capabilities with workflow automation. The design was reviewed and approved by our internal audit, security, SOX, change management, enterprise and platform review teams. To meet these requirements, strong authentication and tight controls were needed to perform directed steps while restricting access. EDAT automation performs validations of code, approvals and other checks at increased speeds and improved quality.
One feature that makes EDAT unique is the ability to support step releases. Step releases allow transports across multiple changes and systems to deploy simultaneously while providing the ability to pause and resume later. It enables the ability to resume by showing the list of changes in the release and displays the progress completed, which easily allows for co-shared deployment across multiple regions and resources.
The picture above depicts the workflow automation implemented in EDAT. Once a user story is created and work is assigned to an iteration, development begins. Unit testing begins in development. Once tasks are released, automated testing is completed in each environment. Before the change can proceed to the next environment, the testing must result in a pass. If there is a failure, the developer is notified so a fix can be created which results in a repeat test until it passes. An automated backup of objects is completed prior to landing to the next environment, so they could be brought back to the previous state should user testing and validation show issues. An automated workflow is orchestrated within each action and code movement, validating and advancing Solution Manager, the service management ticketing system, agile planning tools and automated testing tools. Automated synchronization keeps the environments in sync.
Three methods of deployment are included; One-By-One, Set-Order, and Import-All. One-by-one best suits releases that require many manual steps between transports. Set-order allows for proper sequencing and was designed to allow imports to sequence in serial, while pausing on any landing error. This is often used in releases benefiting from a faster release with on-error stops. Import-all allows transports to land all at once, which best suits releases where the transports do not share objects.
Both standard and custom validations are enabled within EDAT. A structure was set up that allowed for standard validations, which include the various SOX and audit requirements common across all release types. Another structure was set up that allowed for custom validations, which include architecture-specific setup, rule logic and configuration requirements. This dual structure introduced the ability to deploy changes across SAP pipes from within a single screen and deployment rule logic.
EDAT enforces these validations:
- Rule logic based on system and objects.
- SOD/SOX compliance checks.
- Prevention of backdoor addition of unapproved transports.
- Change Manager SAP pipe ownership.
- Release date and the approval date to ensure change management process is adhered.
- Target release systems availability check.
- Lockout, for upgrades and support packs.
Release sequencing is a critical requirement for releases to maintain high quality and ensure a successful release, and is one of the top five reasons why additional resources were needed prior to EDAT. Allowing development teams to order transports and enable the system to import in this order alleviated resource constraints and helped achieve the self-service release capability necessary to increase velocity and improve quality at the same time.
Self-service ability to re-land transports is a key feature that is necessary. Some releases require re-landing a transport to ensure all objects are activated. The only caveat to this feature is to maintain a date check for production landing, to comply with change management.
Landing return codes are shown to the user automatically and full logs are also accessible to the development teams.
Plugin integration enables the ability to allow and support integration across various tools and provide these capabilities:
- Agile planning.
- Auto rollback.
- Pipeline architecture.
- Test automation.
- Live undo.
These capabilities allow EDAT to integrate with other technologies and other platforms through an open API.
EDAT automatically detects objects and platforms which require specific rule logic. Once detected, a summary is displayed to the user so there is an awareness of what will be completed prior to execution. In an upcoming iteration, predictive detection will validate inactive objects in target environments. This will detect issues in advance and initiate fixes automatically.
A single interface with integration across all platforms allows for increased quality, tool reduction and reduced confusion, and provides clear results. This benefit can be seen across multiple service groups and teams. With standard ERP release functionality, users had to use several tools and instances to complete releases. EDAT enabled synchronized and bundled releases from a single tool, where all platforms would import, whether SAP, ABAP or another platform. The ability to implement releases across all platforms is enabling our vision of integrated release across all platforms.
In an upcoming iteration, full test automation will be enabled, where tests will run and are automated. When these automated tests return with a pass, then EDAT will allow them to proceed to the next environment. When the automated tests return with a fail, then EDAT will not allow the code to proceed to the next environment. The development teams will be notified of the failure so a new set of tests can be run after needed fixes. These test results are updated in the SAP Solution Manager change screen for easy retrieval.
One key highlight, there were no change-related incidents and the cadence of releases significantly increased with continuous deployment compared to time-based sweeps. Self-service migrations from Q1 to Q3 2017 increased 341 percent over the same period in 2016.
EDAT made it possible to decrease migration management resources by nearly 45 percent. Migration and release managers are now able to move to higher-value activities. User feedback has been overwhelmingly positive. SAP has since performed a review of the tool and process and acknowledged the change control process has become very agile. The excitement and value realized are positioning the program to continue with improvements and advancing its DevOps’ capabilities.
This article was co-authored by Ramani Subramanian and Jeff Clancy
