CloudOps with a touch of Automation

WHAT

Cloud Infrastructure has been well-knowned as the future of ICT. And by that, the need for experienced engineers on cloud aka Cloud Engineers is increasing day by day.

Their responsibility is all about managing infrastructure, monitoring, patching, and responding to incidents. They try their best to solve problems day by day. Some of it is challenging, most of it is simple and sometimes its just a PEBCAK. And dont get me wrong, they are very good at it. There are many roles that a Cloud Engineer can play, range from Operation to Development, Cost Optimisation or Dealing with consumers (aka other engineers).

Those terms about cloud might sound technical and complpicated to you, but it is financial benefit that make it different. Well-architected infrastructure brings business revenue, reduce operation cost, enhance user experience. And by that, proving Cloud Engineers are essentials part of the team.

When talking about automation, I used to think of labor optimizing and I dreamt my scripts can free me from the daily job, allow me to spend more time with my phone. But reality hits me hard when I actually work on it.

WHY

I am a Cloud Engineer myself, and all I care is how to keep my infrastructure and application in one piece. Yet, it is really boring when I have to work on the same thing every day. I worked on different phases of a software development lifecycle and I realized that thing tend to break every now and then. You cant stop it falling apart, it might be your teammates fault, your manager fault, your interns fault but it would never be your fault (just kidding, its everyone fault).

It is not the end of the world when your Product Manager wake you up at 3 AM asking what 503 Service Unavailable means and why users reporting delay loading web page. From my experience, the more you prepare for the fall, the quicker you can stand it up. And by that, come the holy word: "Automation"

Automation is not the shortest way to liberate myself from my daily tasks, the best way is doing nothing 🤷. What I want from automation is how to reduce my workload the smartest way possible.

HOW

Initially, I plan to write a series of article on building infrastructure, maintaining infrastructure and monitoring it, and introduce how to use automation for those steps. However, I realized that I was too lazy to write that much of tutorials, so this is what I do instead:

I will work on pulling my own setup & source code to the outside world. Then you can guarantee that these are the work has been proven in production workload.

No more joking, lets jump into the very first chapter.

Setting up the playground

Select a IaC to work with:

Terraform, AWS Cloudformation, AWS CDK your choice

>My choice: Terraform with my touch of automation scripts (👌 chef kiss). And might be AWS CDK because of my own interest not because of the trend 😉

You know what, I am gonna share my up-to-date IaC structure that I am using on my projects

• Disclaimer: use it at your own good, I am not taking any responsibility for unwanted AWS billing that cause by irresponsible usage

Copycopy code to clipboard
1terraform/
2├─ my-ground-breaking-application/
3│  ├─ *.tf
4│  ├─ *.tf
5│  ├─ tfvars
6│  │  ├─ backend.tfvars
7│  │  ├─ network.tfvars
8├─ tf_modules/
9│  ├─ my-modules
10├─ .gitignore
11├─ .terraform-version
12├─ README.md

Where to find it?

> I'm glad that you ask, check this out. It might or might not be in it well maintained state, depends on when you are reading this article.

Moreover, I would suggest to have your tfvars files stored separately or at least avoid committing any sensitive credentials such as database password, api token, ... And you can always further separate the structure into smaller pieces base on project/components/apps/... that fit your strategy of resource management. But in general, make sure that resources that shared the same characteristic and function should be stick together.

Terraform project config and authentication:

• Configuration file project_config/config.json define how terraform with interact with AWS
• For terraform to authenticate with AWS: you will need to update your access key and secret in ~/.aws/credentials to match aws_profile that defined in config.json mentioned above

Why should I trust you? - You might ask

> "No, you shoudnt"

Take a look at the below pros and cons and make your own choice

Now, enough of me talking, its time for your practicing

• Clone the repo: git@github.com:tienvu461/CloudOps-with-a-touch-of-automation.git
• Add your first resource (VPC & EC2)
• Run terraform to plan apply the resources of your choice

There are two most essential components when starting to build any cloud infrastructure:

• Network: where all of your applications and services communiaate to each other and communicate with the internet
• Servers: There are multiple type and definition of server that you might already knows, but in the end, it is where your applications and services logic are run and processed.

Optimization opportunities:

• If you are actually following this tutorial, you will soon realize that most of the time you spend is trying to install and setup the prerequisites to make it work. So if possible, automating the process of setup prerequesites is a must have feature
• Credential/Secret and privilege management is a pain for everyone, we should make it more simple and easier.
• I dont know, as long as it work, it is perfect (except from what I mentioned above)

Tips: Whenever I am starting a new project, the first thing I do is to setup my jumpbox/devbox. It is where I installed all my dependencies, scripts & tools so that I can comfortably run and build anything required by that specific project. For example, I can run plan & apply that specific project IaC from my devbox with an Instance Profile with high privilege dedicated for create/destroy resources which is only trusted and allowed to be assume by that devbox (of course with a proper access management, more on this later).One more benefit of this is that I can work from any laptop as long as I can SSH to my devbox from a trusted network with minimal setup time.

What's next

In the next article, let jump your first box, build it and use it like a professional. You will then have a chance to make use of the automation of IaC as well as understand why jumpbox is useful and recommended.