[AWS Fargate] Platform Version 1.4.0. Updates

AWS Fargate Platform Version 1.4.0 introduces some new AWS Fargate capabilities and this entry describes some of them.

Fargate tasks now support Elastic File System (EFS) endpoints

Dating back in 2018, conversation on this github roadmap issue started. Almost all of the developers that added comments wanted the feature to be added onto the Fargate system for the obvious reason that is to have persistent storage in the duration of their Fargate tasks intermittent execution.

Last week, AWS announced that the said feature is now generally supported. From their blog post, it is stated that ECS task definitions for both EC2 and Fargate now support the new EFSVolumeConfiguration parameter which leads to:

• ECS customers using the EC2 launch type no longer need to take care of the cumbersome configuration and automation of storage on EC2 container instances.
• Fargate customers, on the other hand, can now start running stateful workloads inside Fargate tasks, something that they couldn't do before.

Test

To test this new feature, this Qiita blog entry was generally followed as it suggests an easier and a quicker way to explore the said new feature. General flow is as follows:

Create an ECS Cluster. Use the default VPC, Subnet(s), and Security Group for simplicity. You may also use the default ecsTaskRoleExecution IAM role.

Create an EFS File System. Create a volume and take note of the File System ID. Take note that this needs to be in the same Availability zone, VPC, and subnet of your choosing. You may leave the EFS IAM authorization section, again, for simplicity.

Configure Task Definition. You may use the default values. In the Volumes section, add your newly created EFS.

Task Definition Volumes Section

Configure Container Definition. Input the pre-configured ECR Image. In Storage and Logging section, select your source volume and specify /usr/local/apache2/htdocs in the container path.

Storage and Logging section of Container Definition

Mount EFS. Create an EC2 instance in the same VPC, Availability Zone, and Subnet. ssh to that ECS instance and run sudo yum install amazon-efs-utils then mount using sudo mount -t efs <file system id>:/ /opt. 

ssh and amazon-efs-utils installation

Mounting the EFS. Overwriting /opt/index/html

Prior to mounting the EFS volume and re-configuring container definition, a simple HTTPD application was run on platform version 1.3.0:

Platform 1.3.0 Task

Pre-EFS configuration HTTPD sample app

df -hT /usr/local/apache2/htdocs/ was also executed to verify if the EFS is already mounted.

No EFS is seen mounted.

Now, if you create a new revision of the same task definition, explicitly run it on platform version 1.4.0. You should get the updated HTTPD web app:

Platform 1.4.0 Task

Overwritten HTTPD app

EFS is seen mounted

It is as simple as it can get. The gist is that you should have an EFS prior to configuring your Task and Container definitions. Only then will you be able to mount your persistent storage.

Some interesting use cases may include installing a SQL client, shared storage across different Fargate tasks, and leveraging EFS hierarchy for some applications like PHP, Jenkins, etc.

Currently CloudFormation support is on the way.

Fargate tasks now have a consolidated 20GB ephemeral volume

As the title suggests, Fargate now has a bigger on-the-fly storage.

Test

This was simply tested. The idea is run df -bg command in the container in both platform versions 1.3.0 and 1.4.0.

Platform version 1.3.0. df results

Platform Version 1.4.0 df results

Notice the difference in values provided by running the df command.

This enables the customers to run slightly larger data in their Fargate tasks.

Network performance metrics are available in CloudWatch Container Insights

This is merely a fix for the pre-existing bug in AWS. From their blog, Up until platform version 1.3.0, Fargate tasks could not report network performance metrics back to Container Insights due to a limitation that exists at the intersection of the awsvpc networking mode and the ECS agent.

Test

When you create an ECS cluster, just tick the Enable Container Insights box. You may check the metrics logs in the Metrics tab of your cluster. Click the View Container Insights button.

Container Insights

Notice the Network section which AWS reported to be unresponsive before now receiving data without a problem.

Similarly, AWS also announced that these Network stats are now available in Fargate via the task metadata v4. This task metadata now also includes the Availability Zone attribute.

Other Updates

Task elastic network interface (ENI) now runs additional traffic flows

The takeaway is that customers may now be able to see the traffics when ECR login credentials, sourcing secrets from AWS Secrets Manager, and sourcing secrets from AWS Systems Manager. Previously, those traffics are sent through Fargate ENI which is not visible to the customers.

Fargate tasks now support the CAP_SYS_PTRACE Linux capability

AWS previously decided to disable Linux capabilities. Since receiving feedback from customers as well as AWS partners, they have decided to add the CAP_SYS_PTRACE capability in Fargate task.

Containerd is replacing Docker as the container runtime

From AWS, this will allow Fargate to innovate faster. Docker engine builds on top of Containerd. This means that Docker containers may still run without a problem. This is not a customer consumable feature and is handled by AWS internally. You may read more on this blog post.

The Fargate agent is replacing the ECS Agent

Fargate agent, as the name suggests, is purpose-built for the Fargate environment and will allow AWS to innovate faster for the customers. Combination of it with Containerd will propel the new Firecracker-based architecture.

Now, due to the changes in container runtime and the agent running the VM, it follows that there is a need for a change of the relevant error messages. For example, DockerTimeoutError does not make sense any longer since container runtime is not Docker anymore. ContainerRuntimeTimeoutError makes more sense now. Refer to this documentation for the complete list of Fargate error messages.

Task ENIs support jumbo frames for improved networking efficiency

Starting with this new version, Task ENIs support jumbo frames like all other VPC ENIs. AWS says that this will increase the efficiency and reduces compute overhead whenever the network path between source and destination supports jumbo frames.

Conclusion

In this blog post, the updates included with the new platform version 1.4.0 were summarized. Some tests we presented to show how these new updates are utilized.

References

• https://aws.amazon.com/jp/blogs/containers/aws-fargate-launches-platform-version-1-4/
• https://aws.amazon.com/blogs/containers/aws-fargate-platform-versions-primer/
• https://github.com/aws/containers-roadmap/issues/53
• http://man7.org/linux/man-pages/man7/capabilities.7.html
• https://qiita.com/cumet04/items/4b5cbdb29d76c72eb26c