Automating infrastructure provisioning for a maritime research firm leading to a significantly reduced cloud cost

Daffodil Software’s team of seasoned DevOps engineers undertook an in-depth evaluation of Drewry’s existing resource provisioning framework. With a clear understanding of Drewry’s need for agility and efficiency, Team Daffodil set out to create a responsive, on-demand infrastructure that could seamlessly adapt to Drewry’s unique operational needs, ensuring optimal performance and cost-effectiveness for their data processing activities.

Team Daffodil implemented a comprehensive suite of solutions that involved:

Automated Infrastructure Provisioning with AWS CloudFormation
The DevOps team at Daffodil ensured that infrastructure is created on demand whenever there is a need to run processes with high resource utilization. To automate infrastructure provisioning, AWS CloudFormation was utilized. AWS CloudFormation is an Infrastructure as Code (IaC) service that allows automating and managing resources across all AWS accounts & regions through a single operation.

Comprehensive CloudFormation Templates
Daffodil crafted CloudFormation templates that were capable of launching EKS node groups in various sizes to meet specific requirements. These templates were meticulously stored in Amazon S3, providing version control and the ability to download at the time of stack creation, thus ensuring consistency and reliability in infrastructure deployment.

Integration with Airflow and Boto3 for Streamlined Operations
The AWS SDK for Python (Boto3) was utilized to integrate these CloudFormation templates with Apache Airflow DAGs. Team Daffodil designed two types of DAGs: one for creating and another for deleting the CloudFormation stacks, which included the necessary EKS Node Groups for running ETL pipelines. This allowed for precise control over the infrastructure during periods of high demand.

Container Orchestration with Amazon EKS
Amazon Elastic Container Service (EKS) was utilized to deploy Apache Airflow [an open-source platform used for orchestrating, scheduling, and monitoring complex workflows or data pipelines] and its components, ensuring the portability and consistent management of containerization tasks. This strategic deployment on Amazon EKS enhanced the deployment process, contributing to the high availability and scalability of the infrastructure.

CI/CD Pipelines for Efficient and Automated Software Deployment
Daffodil streamlined the software development and deployment processes through AWS CodeCommit and Jenkins, an open-source CI/CD tool hosted on AWS EC2. CodeCommit securely and efficiently stores application’s source code, while Jenkins efficiently handles code building, testing, and deployment. This approach has significantly expedited software delivery, making it both fast and reliable.

Cost Optimization and Processing Efficiency
In a quest to reduce processing time, the Daffodil team adopted a strategy of breaking down workflow pipelines into multiple parallel executions. While this approach greatly enhances efficiency, it necessitates the use of high CPU and memory-intensive servers (EKS Worker Nodes). However, this could lead to increased infrastructure costs.

To address this challenge and optimize costs CloudFormation was utilized to help in automating the infrastructure provisioning. It provided the necessary control to the product admin to create infrastructure as per their need when they wished to run a highly intensive process once or twice in the month & delete it when the process completes. Thus aiding in cost optimization as it would reduce the need to run servers round the clock. The result was improvement in operational efficiency and resource management.