AWS Infrastructure is hard, and how we made it simple
TL;DR;
Visit http://octo.quadnix.com to learn more about Octo.
Have you ever hosted your websites and applications in AWS? Then you know how increasingly difficult and complicated it can get. The more you put in there, the more number of resources you will have to create and manage.
I remember writing my first infrastructure, and how annoyed it made me. With time, that feeling has only gotten worse, and its almost impossible to maintain a large project without a dedicated devops team.
Don't get me wrong - its so much better than bare bones servers, and I get why its complicated. AWS, and other cloud providers, does offer a lot more choices to developers, and for good reason. But it doesn't discount the everyday hassle of infrastructure. Try hosting a simple server in a standard ECS container and you will end up creating at least 10+ resources to support it. Try adding database and cache, and you will already feel the pain I describe.
If you are in this situation, the first step to bring back your sanity - would be to use an IaC (infrastructure-as-code) tool. What's that? Just a tool to pragmatically define your infrastructure as code, and then to run it again and again as you make changes which should keep updating your infrastructure as per your instructions.
This is far better than having to make changes manually.
There are a few IaC champions for you to consider - Terraform being the subjectively best one, and then Pulumi, AWS CDK, and a few more. Terraform has its own syntax for writing infra code, unlike normal programming languages - its relatively easy. Pulumi and AWS CDK leverage existing languages like TypeScript, Java, and Go to define infra resources.
But I had my own personal challenges with them - mainly, the fact that they offer a one-to-one resource object for every AWS resource. E.g. a s3 bucket in AWS is represented by a TF module, a Pulumi module, and AWS CDK construct. Same for VPC, security groups, target groups, and so on.
When I wrote code in these frameworks, I often found the complexity to still exist, just in a more manageable and programatic way. I was looking for something more easy, and more abstract.
That's when I started Octo - a framework to not only abstract away infrastructure, but to offer community built use cases that I can just import and use. No re-inventing the same wheel again and again. Just plug and play!
Here's what Octo can do for you,
- Help you abstract away AWS resources in Typescript classes. This is similar to what other IaCs provide.
- Then allows you to define models on top of resources to abstract and categorize them behind familiar terms that you understand and care for. A s3 website becomes a service, a vpc becomes a region, a load balancer becomes a service, and so on. This categorization helps you visualize your resources as high level abstracted components that you are working with.
- You can then define modules - a complete end-to-end functionality that you can reuse, and even share with others, or import from others. These modules take care of one single piece of infrastructure work for you. E.g. a module to create a region, to create a server, to create a load balancer, and so on.
- Finally you can then create templates that defines full blown infrastructure that you can just plug your specific values in and enjoy a fully setup infrastructure that you didn't have to create manually or think about. E.g. a template that hosts your static websites using S3 buckets, or a template to host a server behind ECS/EKS. These templates can do large chunks of work, from setting databases and caches, to setting up load balancers and HTTPS, DNS, and anything else you expect in between to get your infrastructure up and running.
Why I feel this is the right pattern?
- Reusability: to be honest, most infrastructure are the same pattern I see over and over again. At least all my services are hosted pretty much the same way. Having a template allows me to reuse the same concepts and setup for different use cases.
- Security & Compliance: the same setup allows me to be consistent across all my services. Fixing a security flaw, or adding PII and GDPR compliance is now a matter of updating my packages and tweaking the options to my liking.
- Visually Stunning: Octo is built on graphs - and so it makes it possible to represent my infra in graphs and can show me all my components and resources and models in graphical forms, including their dependencies. It gives me the visual reference of how different parts affect each other. This is a huge advantage, since a picture is really worth a thousand words :)
If you are curious, I would really implore you to checkout the Octo project - http://octo.quadnix.com
Its open-sourced and currently supports a few AWS modules.
The best part - there is nothing stopping people from writing modules for other cloud providers as well, so I definitely see a future where you can even mix and match infra component pieces from different cloud providers together. And all of this neatly bundled in templates - so you can just plug and play!