DevSecOps CI-CD Reference for CSPs

8 min readMay 10, 2021

Background

CSPs have started adopting Agile and DevOps methods but they have a challenging task to manage their own platforms & capabilities, often consisting of a combination of on-premise infrastructure, public cloud and customer premises. That brings enormous complexity to maintain and manage the CI-CD Flow to enable agile delivery of the applications, systems, assets, services and networks. Leaders within these organizations recognize that minimizing deployment friction and decreasing the lead time for the delivery of applications, while simultaneously providing new features & services without compromising on security, can provide a competitive advantage.

In order to have a greater Delivery Experience and as well as Operations Experience it is inevitable that the Organizations have to build appropriate CI/CD Pipeline Flow Platforms to enable rapid deliveries and rapid experiences to customers. But building CI/CD Platforms, maintain and manage them is not a trivial task. It requires continuous investment of effort & time to continuously learn and evolve the platforms as per the changing technology landscape and as well as dynamic needs of the various core teams.

In this article, I have provided a reference architecture, framework and sample implementation based on my experience so far with DevSecOps in Telecom domain.

Introduction

The following picture illustrates in simple terms what is meant by DevSecOps in general and how the organization should approach DevSecOps implementation at high level.

As seen in the above picture the culture change in the organization is the primary step to be followed by the leadership. At first the traditional barriers of handovers, bureaucratic mandates, walled garden processes and political behavior have to be divorced by the teams and the leadership have to take up the necessary steps in order to break those barriers and increase the culture of sharing and trust among the teams and people in the organization including the vendors and partners.

Once the teams have started adopting the new age culture and practices for effective collaboration then they can start automating the things so that the time and effort will be saved for the teams. Eventually Organization can start using the Platform-as-a-Service model for the contextual capabilities such as CI-CD solution.

As per my experience there is no need to do complete reshuffle and merge all functions (such as development, test, security and operations) into one unit as that will not yield any major benefits rather it will introduce too much of confusion and chaos and also unwanted political behaviour among the teams and people for the threat of roles/positions. Instead leaders can introduce lean agile practices, overall alignment for effective collaboration among the teams, effective frameworks and means for culture change which will yield long term benefits and as well as immediate results to the organization to get mature in DevSecOps implementation.

Ultimately it requires overall alignment in the organization in order to achieve the intended benefits with DevSecOps implementation. Following picture illustrates the alignment paradigm across the teams, leadership and partners.

For eg. Many organizations will have experts created organically in Security function. Instead of merging that into single unit and losing the experts leadership can take steps to make those Security experts start creating the templates, codify the security practices and maintain Compliance as Code or Policies as Code so that it will be automated into CI-CD pipeline and avoid the routine manual gates process. That way the subject knowledge will retained and become useful effectively for the organization. Similarly other functional teams as well follow Everything as Code practices so that the automation can be effectively utilized to eradicate the manual gated processes and steps eventually.

Key Challenges for CSPs

Communication Service Providers (CSPs) are facing following key challenges which are becoming hindrance to build CI-CD platform and make Core teams & Business Agile.

Complexity of systems integration with many COTS systems and apps along with inhouse business and operational capabilities.
CSPs typically have existing “heritage” or “legacy” systems of previous generations of broadband and mobile which also need to be considered for automation alongside the new systems that are being built for new age networks
If the core teams are allocated to build the CI-CD platforms then CSPs will lose the competitive advantage due to missing the opportunities for releasing the new product features at right time for the end customers.
Leaders in CSP organizations are not distinguishing the budget allocations for building the Core and Contextual Capabilities. That is creating a conflict and many times leaders will take obvious choice to prioritize the most of the budget for core capabilities at the cost of contextual capabilities. CSPs fall victims to the traditional barriers and processes and will never be Agile.
CI-CD Platforms have to be continuously evolved, maintained and managed in order to take the advantage of new technology features, conduct frequent experiments & changes for maturity, licensing & Cost optimization, Open source adoption, etc.
Core teams will become overwhelmed and exhausted to continuously invest efforts and time for contextual capabilities. It will increase the burden of cognitive overload on core teams and eventually it will have side effect on delivering the core products and services.

Global-Local CI-CD Framework

A generic framework for CI-CD has been invented based on the experience obtained from various implementations across CSP clients. The framework can be easily customized to any CSP as per the needs of their core teams.

As a minimum following objectives have to be met by any CI/CD Platform solution and this generic framework complies with these inherently.

CI/CD platform should become an enabler for the Teams to be empowered
Enable Faster Innovation and Rapid Development for the teams
Support Global alignment as per organization strategy, value streams and roadmap
Maintain a right balance so that CI/CD will not become a bottleneck for the teams to achieve what they need
Collaboration & Sharing of the best practices, standards, etc. among the teams
CI-CD Platform should act as force multiplier for core teams, helping them to focus on core domain functionality through attention to the developer experience, operational experience, ease of use, simplicity of tooling, continuous evolution and richness of documentation. Treat Core teams as internal customers and build the platform as a Product or Service itself.
Build the CI-CD Platform with key principles such as API-first, Self-service, Declarative and most importantly build it with Empathy towards the core teams.

This CI-CD framework will have 2 layers (Global & Local Stacks) to empower the core teams and at the same time ensure the global alignment across the organization.

CSPs can adopt different options to leverage their IT vendor to build Global and Local CI-CD stacks as it is suitable for them.

For eg. Global stack can be developed, maintained and managed by the IT Vendor where IT Vendor will be accountable whereas the local stacks will be developed, maintained and managed by the respective application/stream aligned teams. IT Vendor can lay out the required templates, extensions and frameworks so that the application teams can follow the best practices and guidelines for consistency and alignment. There has to be continuous learning and sharing between the teams of global and local stack owners.

IT vendors can deliver the CI-CD platform capabilities as In-House capabilities or as a Service as suitable and desired by the Organizations. IT vendors should also cultivate the new age culture where their business models should encourage the teams to have greater trust and empathy towards the core teams and align to the organization goals. IT Vendors should not have traditional outsourcing business models for CI-CD Platform-as-a-Service rather they have to invent new methods in order to retain and grow the trust with the core teams and align to the organization culture for inclusive and collective growth.

This way cognitive load on core teams will be reduced and they can focus their efforts and time for core business goals while the IT vendors will be accountable for developing contextual capabilities such as CI-CD platform services.

Organization Aspects

Sample Operating Model

Following is an illustration of a sample organization model that will be suitable for effective DevOps practices.

Guiding Principles

Any organization should follow these fundamental principles which will be applicable for any type of organization that is on DevOps transformation journey.

Process Flow

Following picture depicts the various steps that are involved typically in a CI-CD flow of DevOps practice across the functions of Demand, Delivery, Deployment and Operations phases.

Best Practices for DevSecOps

Based on my experience so far these are the best practices that any organization can follow to implement DevSecOps.

CI-CD Reference for Telecom BSS/OSS/NFVs

Following picture provides a reference integration architecture for CI-CD implementation in any telecom organization for BSS, OSS and NFV domains.

Git Branching Strategy

It is always recommended to use Trunk based model for Git branching which will give lot of flexibility for developers and as well as release management and that helps a lot of smooth CI-CD management.

CI-CD Pipeline Stages— Sample

Following picture illustrates the various steps involved in creating the CI-CD pipeline. The tools mentioned in the picture are for sample and different tools can be used for respective implementations.

CI-CD Pipeline Flow — Sample

Following picture depicts a sample flow of the software from requirement to go-live using a CI-CD pipeline mechanism. The tools mentioned in the picture are for sample and different tools can be used for respective implementations.