“DevOps is a combination of best practices, culture, mindset, and software tools to deliver a high quality and reliable product faster”
DevOps agile thinking drives towards an iterated continuous development model with higher velocity, reduced variations and better global visualization of the product flow. These three “V's" are achieved with synchronizing the teams and implementing CI/CD pipelines that automate the SDLC repetitive and complex processes in terms of continuous integration of code, continuous testing, and continuous delivery of features to the production-like environment for a high-quality product with shorter release cycles and reduced cost. This ensures customer satisfaction and credibility.
A streamlined process in place with the help of best practices and DevOps tools reduce the overhead, and downtime thus giving more opportunity for innovation. As a matter of fact, DevOps way of defining every phase (coding, testing, infrastructure provisioning, deployment, and monitoring) as code also makes it easier to rollback a versioned code in case of disaster recovery and make the environment easily scalable, portable and secure.
“DevOps tools help you accomplish what you can already do but do not have time to do it.”
1. What are the tasks of a DevOps Engineer?
A Summary of day-to-day tasks carried out by a DevOps engineer -
- Design, build, test and deploy scalable, distributed systems from development through production
- Manage the code repository(such as Git, SVN, BitBucket, etc.) including code merging and integrating, branching and maintenance and remote repository management
- Manage, configure and maintain infrastructure system
- Design the database architecture and database objects and synchronize the various environments
- Design implement and support DevOps Continuous Integration and Continuous Delivery pipelines
- Research and implement new technologies and practices
- Document processes, systems, and workflows
- Creation and enhancement of dynamic monitoring and alerting solutions using industry-leading services
- Continuously analyse tasks that are performed manually and can be replaced by code
- Creation and enhancement of Continuous Deployment automation built on Docker and Kubernetes.
2. Who can become a DevOps Engineer?
DevOps is a vast environment that fits almost all technologies and processes into it. For instance, you could come from a coding or testing background or could be a system administrator, a database administrator, or Operations team there is a role for everyone to play in a DevOps approach.
You are ready to become a DevOps Engineer if you have the below knowledge and/expertise-
- You have a Bachelor’s or Master’s or BSC degree (preferably in Computer Science, IT, Engineering, Mathematics, or similar)
- Minimum 2 years of IT experience as a Software Developer with a good understanding of SDLC lifecycle with lean agile methodology (SCRUM)
- Strong background in Linux/Unix & Windows Administration
- System development in an Object-oriented or functional programming language such as Python / Ruby / Java / Perl / Shell scripting / Groovy or Go
- System-level understanding of Linux (RedHat, CentOS, Ubuntu, SUSE Linux), Unix (Solaris, Mac OS) and Windows Servers
- Shell scripting and automation of routines, remote execution of scripts
- Database management experience in Mongo/Oracle or MySQL database
- Strong SQL and PL/SQL scripting
- Experience working with source code version control management like Git, GitLab, GitHub or Subversion
- Experience with cloud architectures, particularly Amazon Web Services(AWS) or Google cloud platform or Microsoft Azure
- Good understanding of containerization using Dockers and/or Kubernetes
- Experience with CI/CD pipelines using Jenkins and GitLab
- Knowledge of data-centre management, systems management, and monitoring, networking & security
- Experience in Automation/configuration management using Ansible, and/or Puppet and/or Chef
- Know how to monitor your code using Configuration Monitoring tools such as Nagios or Prometheus
- Background in Infrastructure and Networking
- Extensive knowledge about RESTful APIs
- A solid understanding of networking and core Internet protocols (e.g. TCP/IP, DNS, SMTP, HTTP, and distributed networks)
- Excellent written and verbal English communication skills
- Self-learner, team layer, willingness to learn new technologies and ability to resolve issues independently and deliver results.
3. Roadmap to becoming a DevOps Engineer
3.1 Learn a programming language
A programming language enables a user to interact and manage the system resources such as the kernel, device drivers, memory devices, I/O devices; also to write software.
A well-written piece of code will be more versatile, portable, error-proof, scalable and optimized that will enhance your DevOps cycle letting you be more productive with a high-quality product.
As a DevOps Engineer, you will have to use many software and plugins for a CI/CD pipeline, and you will be at your best if you have a good grip on some of the popular programming languages:
1. Java: An object-oriented, general-purpose programming language. Goal – “Write once, run anywhere”, which is synonymous with the Dockers(or containerization) philosophy
2. C: Is a general-purpose procedural programming language, it supports structured programming
3. C#: A general-purpose, multi-paradigm object-oriented programming (OOP) language
4. Python: Python is an easy to learn, interpreted, high-level and powerful programming language with an object-oriented approach. Ideal for infrastructure programming and web development. It has a very clear syntax
5. Ruby: Is an open-source dynamic OOP programming language with an elegant and easy syntax. This implements multiple multi-paradigm approaches.
As you know, DevOps majorly emphasizes on automating the repetitive and error-prone tasks.
You ought to know any of the popular scripting languages:
6. Perl: Perl is a highly capable scripting programming language, with its syntax very similar to C
7. Bash shell script: Powerful set of instructions in a single shell script file to automate repetitive and complex commands
8. JavaScript: An interpreted scripting language to build websites
9. PowerShell for windows: A cross-platform automation and configuration framework or tool, that deals with structured data, REST APIs and object models. It has a command-line tool.
Good-to-know language:
10. Go: Go is an open-source programming language developed by Google. It is used to build simple, reliable and efficient software
3.2 Understand different OS concepts
As a Software developer, you must be able to write code that can interact with the machine resources and have a sound understanding of the underlying OS you are dealing with.Knowing the OS concepts will help you be more productive in your programming.
This gives you the ability to make your code faster, manage processes, interact with the input-output devices, communicate with the other OS, optimize the processing usage, memory usage and disk usage of your program.
As a DevOps engineer with infrastructure role, setting up and managing servers, controllers and switches becomes easier if you understand the resources, processes, and virtualization concepts very well.
To be able to administer the users and groups, file permissions and security you must know the filesystem architecture.
Essential OS concepts a DevOps engineer must know include:
I. Kernel management
Kernel is the core element of any OS. It connects the system hardware with the software. It is responsible for memory, storage, and process management
II. Memory Management
Memory management is the allocation/deallocation of system memory(RAM, cache, page) to various system resources and to optimize the performance of the system
III. Device drivers management
A device driver is a software program that controls the hardware device of the machine
IV. Resource management
The dynamic allocation/deallocation of system resources such as kernel, CPU, memory, disk and so on
V. I/O management
Communication between various input/output devices connected to the machine such as- keyboard, mouse, disk, USB, monitor, printers, etc
VI. Processes and process management
Every program that executes a certain task is called a process, each process utilizes a certain amount of computational resources. The technique of managing various processes to share the load of memory, disk and CPU(processing) usage also the inter-process communication is termed as process management.
VII. Threads and concurrency
Many programming languages support multi-threading and concurrency, i.e, the ability to run multiple tasks simultaneously.
VIII. Virtualization and containerization
Concept of simulating a single physical machine to multiple virtual machines/environments to optimize the use of resources and to reduce the time is taken and cost. Understand this well as you will often need to replicate the real-time environment.
Linux containers are a great concept to isolate and package an application along with its run-time environment as a single entity.
Run-time environment includes all its dependencies, binaries, configuration files and libraries. Dockers is a containerized command-line tool that makes it easier to create, run and deploy applications with containers.
Using both Virtual machines and dockers together can yield better results in virtualization
IX. Distributed file systems
A client machine can access data located on a Server machine. This is true in the case of a client/server-based application model.
X. Filesystem architecture
The architectural layout of how and in what hierarchy the data is organized on a disk, will make your task of managing data easier.
3.3 Learn about managing servers
As cloud deployments become more useful with DevOps approach, there is a need to manage a group of Servers (Application, Database, Web Server, Storage, Infrastructure, Networking Server and so on) rather than individual servers.
You should be dynamically scaled up/down the servers, without rewriting the configuration files.
Nginx: This is a web server that can also be used as a reverse proxy, load balancer, mail proxy, and HTTP cache.
This provides robust and customizable monitoring of your cloud instances and their status. Nginx offers more flexibility and configurability for better configuration and automation using DevOps tools like Puppet and Chef.
3.4 Networking and Security
In a highly connected network of computers, it becomes essential to understand the basic concepts of networking, how to enforce security and diagnose problems.
As a DevOps engineer, you would also be required to set up an environment to test networking functions. In addition, set up continuous integration, delivery and deployment pipelines for network functions.
Learn the basic networking concepts like Ip addresses, DNS, routing, firewalls and ports, basic utilities like ping, ssh, netstat, ncr and ip, load balancing and TLS encryption.
Understand the basic protocols(standard rules for networking) such as-
TCP/IP (Transfer Control Protocol/Internet Protocol), HTTP (Hypertext Transfer Protocol), SSL, SSH (Secure Shell), FTP (File Transfer Protocol), DNS (Domain Name Server).
Configuration management tools like Ansible and Jenkins can be used to configure and orchestrate network devices.
3.5 What is and how to set-up
As a DevOps methodology we often describe CI/CD pipeline, let us understand what is it?
Continuous Integration(CI) is a development practice wherein developers regularly merge or integrate their code changes into a commonly shared repository very frequently.
If I speak from a VCS (preferably Git’s) point of view -
Every minor code change done on various branches (from different contributors) is pushed and integrated with the main release branch several times a day, rather than waiting for the complete feature to be developed.
Every code check-in is then verified by an automated build and automated test cases. This approach helps to detect and fix the bugs early, resolve the conflicts that may arise, improve software quality, reduce the validation and feedback loop time; hence increasing the overall product quality and speedy product releases.
Continuous Delivery(CD) is a software practice where every code check-in is automatically built, tested and ready for a release(delivery) to production. Every code check-in should be release/deployment ready.
CD phase delivers the code to a production-like-environment such as dev, uat, preprod, etc and runs automated tests.
On successful implementation of continuous delivery in the prod-like environment, the code is ready to be deployed to the main production server.
It is best to learn the DevOps lifecycle of continuous development, continuous build, continuous testing, continuous integration, continuous deployment and continuous monitoring throughout the complete product lifecycle.
Based on the DevOps process setup use the right tools to facilitate the CI/CD pipeline.
3.6 Learn Infrastructure as code
Infrastructure as code (IaC) is to define(or declare) and manage the infrastructure resources programmatically by writing code as configuration files instead of managing each resource individually.
These infrastructure resources(hardware and software) may be set up on a physical server, a Virtual machine or cloud.
An IaC defines the desired state of the machine and generates the same environment every time it is compiled.
What does IaC do?
- Automation: Spinning up or scaling down many resources becomes easier, as just a configuration file needs to be compiled and run. This reduces the overhead and the time spent.
- Versioning: IaC is a text file which can be versioned controlled which means 3 things:
- Infrastructure changes such as scaling up/down the resources and or changing/updating the resources (filesystem or user management) can be tracked through the versioned history
- Configuration files are easily shareable and portable and are checked-in as source code
- An IaC text file can easily be scheduled to be run in a CI/CD pipeline for Server management and orchestration.
Manual errors eliminated: productivity increased
How to do it?
Use tools like Puppet, Ansible, Chef, Terraform
These tools aim at providing a stable environment for both development and operations tasks that results in smooth orchestration.
A. Puppet: Puppet is a Configuration Management Tool (CMT) to build, configure and manage infrastructure on physical or virtual machines
B. Ansible: is a Configuration management, Deployment and Orchestration tool
C. Chef: is a configuration management tool written in Ruby and Erlang to deploy, manage, update and repair server and application to any environment
D. Terraform: This automation tool builds, change, version and improve infrastructure and servers safely and efficiently.
How will IaC be applied in DevOps?
IaC configuration files are used to build CI/CD pipelines.
IaC definitions enable DevOps teams to test applications/software in production-like stable environments quickly and effortlessly.
These environments with IaC are repeatable and prevent runtime issues caused due to misconfiguration or missing dependencies.
---
3.7 Learn some Continuous Integration and Delivery (CI/CD) tools
In order to continuously develop, integrate, build, test, apply feedback, deliver our product features to the production environment or deploy to the customer site, we have to build an automated sequence of jobs(processes) to be executed using the appropriate tools.
CI/CD pipeline requires custom code and working with multiple software packages simultaneously.
As a DevOps Engineer, here are some widely used tools you must know-
a. Jenkins is an open-source automation server. Using Jenkins plugins CI/CD pipelines are built to automatically build, test and deploy the source code
Jenkins is a self-contained Java-based program and easy to configure, extensible and distributed
b. GitLab CI is a single tool for the complete DevOps cycle. Every code check-ins trigger builds, run tests, and deploy code in a virtual machine or docker container or any other server. Its has an excellent GUI interface. GitLab CI also has features for monitoring and security
c. CircleCI software is used to build, test, deploy and automate the development cycle. This is a secure and scalable tool with huge multi-platform support for IOS and MAC OS using MAC virtual machines along with Android and Linux environments
d. Microsoft VSTS(Visual Studio Team Services) is not only a CI/CD service but also provide unlimited cloud-hosted private code repositories
e. CodeShip tool empowers your DevOps CI/CD pipelines with easy, secure, fast and reliable builds with native docker support. It provides a GUI to easily configure the builds
f. Bamboo by Atlassian is a Continuous integration, deployment and delivery Server. Bamboo has built-in Jira Software and BitBucket Software Integration, also built-in git branching and workflows.
Jenkins is the most popular and widely used tool with numerous flexible plugins that integrate with almost any CI/CD toolchain. Also the ability of Jenkins to automate any project really distinguish this tool from others, thus it is highly recommended to get a good grip of this tool as a DevOps practitioner.
Note: Since this is also a key for enthusiasts to choose the right tool but should be short definitions
3.8 Know the tools to monitor software and infrastructure
It is crucial to continuously monitor the software and infrastructure upon setting up the continuous integration and continuous delivery pipeline (CI/CD) to understand how well your DevOps setup is performing. Also, it is vital to monitor system events and get alerts in real-time.
A hiccup in the pipeline such as an application dependency failure or a linking error, or say the database has a downtime must be immediately notable and taken care of.
This is where a DevOps Engineer must be familiar with monitoring tools such as -
1. Nagios: is an open-source software application that monitors systems, networks, and infrastructure(Servers) and generates logs and alerts
2. Prometheus: is an open-source real-time metrics-based event monitoring and alerting system.
3.9 Learn about Cloud Providers
As the computational need increases so do the demand of the infrastructure resources.Cloud computing is a higher level of virtualization, wherein the computing resources are outsourced on a “cloud” and available for use on a pay-as-you-go basis over the internet.Some of the leading cloud providers such as AWS, Google Cloud, Microsoft Azure to name a few provide varied cloud services like IaaS, PaaS, and SaaS.
Begin part of a DevOps practice, you will often find the need to access various cloud services say for infrastructure resources, production-like environment on the go for testing your product without having to provision it, get multiple replicas of the production environment, create a failover cluster, backup and recover your database over the cloud and various other tasks.
Some of the cloud providers and what they offer are listed below-
A. AWS (Amazon Web Services): provide tooling and infrastructure resources readily available for DevOps programs customized as per your requirement. You can easily build and deliver products, automate CI/CD process without having to worry about provisioning and configuring the environment
B. Microsoft Azure: Create a reliable CI/CD pipeline, practice Infrastructure as Code and continuous monitoring through Microsoft-managed data centres
C. Google Cloud Platform: Uses google-managed data centres to provide DevOps features like end-to-end CI/CD automation, Infrastructure as Code, configuration management, security management, and serverless computing.
AWS is the most versatile and recommended provider that you may wish to start learning.
4. What next after becoming a DevOps expert?
“Sky is the only limit for a DevOps person !!!”
Mastering the DevOps tools and practices opens up the door to new roles and challenges for you to learn and grow.
4.1 DevOps Evangelist
A technical Evangelist is a strong powerful and influential role that exhibits a strong thought process.
A DevOps evangelist is a DevOps leader who identifies and implements the DevOps features to solve a business problem or a process, and then shares and promotes the benefits that come from DevOps practice.
Also identifies the key roles and train the team in the same and is responsible for the success of entire DevOps processes and people.
4.2 Code Release Manager
A Code Release Manager measures the overall progress of the project in terms of metrics, he/she is aware of the entire Agile methodology. A Release Manager is more involved in the coordination among all the phases of DevOps flow to support continuous delivery.
4.3 Automation Architect
The key responsibility is to plan, analyze, and design a strategy to automate all manual tasks with the right tools and implement the processes for continuous deployment.
4.4 Experience Assurance
An experience Assurance person is responsible for the user experience and makes sure that the product being delivered meet the original business specifications.
This role is also termed as Quality Assurance but with extended responsibilities of user experience testing. This role plays a critical role in the DevOps cycle.
4.5 Software Developer/Tester
Under DevOps, the role and responsibilities of a Software Developer literally expand l, that the developers are no longer responsible for writing code, but also take ownership of unit testing, deployment and monitoring as well.
A Developer/Tester has to make sure that the code meets the original business requirement.
Henceforth; the role Developer/Tester or if the innovation extends further a Developer may also be referred to as DevTestOps.
4.6 Security Engineer
Security Engineer focuses on the Integrity of data by incorporating security into the product, and not at the end.
He/she supports project teams in using security tools in the CI/CD pipeline, as well as provide resolution of identified security flaws.
Conclusion
“If you define the problem correctly, you almost have the solution.” - Steve Jobs
In a nutshell, if you aspire to become a DevOps professional you ought to know -
- Programming language (C, Java, Perl, Python, Ruby, Bash shell, PowerShell)
- Operating System concepts (resource management)
- Source Control (like Git, Bitbucket, Svn, VSTS, etc)
- Continuous Integration and Continuous Delivery (Jenkins, GitLab CI, CircleCI)
- Infrastructure as Code (IaC) Automation (tools like Puppet, Chef, Ansible and/or Terraform)
- Managing Servers (application, storage, database, infrastructure, networking, web server etc)
- (Application, Database, Web Server, Storage, Infrastructure, Networking Server
- Networking and security
- Container Concepts (Docker)
- Continuous monitoring (Nagios and Prometheus)
- Cloud (like AWS, Azure, Google Cloud).
DevOps ways( The three ways of DevOps) open the door of opportunities to improve and excel in the process using the right tools and technologies.
“DevOps channels the entire process right from the idea on a whiteboard until the real product in the customer’s hands through automated pipelines(CI/CD).”
As a DevOps Engineer you must be a motivated team player, need to have a desire to learn and grow, optimize the process and find better solutions.
Since DevOps covers a vast area under its umbrella, it is best to focus on your key skills and learn the technologies and tools as needed.
Understand the problem/challenge then find a DevOps solution around the same.
