Each job type requires an Amazon Elastic Container Services (Amazon ECS) instance (desiredCount = 0), Amazon Simple Queue Service (Amazon SQS) queue, and associated Amazon CloudWatch alarms. See step 4. Amazon API Gateway provides an interface to add, query and cancel jobs. The “add job” Lambda function creates the job state item in DynamoDB, and enqueues the job ID on SQS. The SQS job queue contains messages with the IDs of the jobs to be processed. Amazon CloudWatch monitors the SQS queue depth. Alarms are triggered when the depth exceeds a configurable value, and when the queue is empty. Amazon ECS responds to the CloudWatch alarms, changing the desired count of ECS and scaling the number of parallel jobs. AWS Fargate runs job processing containers from Amazon Elastic Container Registry (Amazon ECR). The containers poll SQS for job IDs, read the job item from DynamoDB, and perform processing, updating the job status in DynamoDB and publishing to an Amazon Simple Notification Service (Amazon SNS) topic on a status change. A DynamoDB table stores the state, parameters and results of the jobs, by job ID. Amazon ECR stores the job processing container images. An SNS topic publishes job items on state change notifications to subscribers. An SNS topic publishes job items on state change notifications to subscribers.
