|
Today, I am pleased to announce the availability of AWS Cloud Control API a set of common application programming interfaces (APIs) that are designed to make it easy for developers to handle their AWS and third-party services.
AWS delivers the broadest and deepest portfolio of cloud services. Builders leverage these to construct any type of cloud infrastructure. It initiated with Amazon Simple Storage Service (Amazon S3) 15 years ago and grew over 200+ services. Each AWS service has a specific API with its own vocabulary, enter parameters, and error reporting. For example, you use the S3 CreateBucket API to create an Amazon Simple Storage Service (Amazon S3) bucket and the Amazon Elastic Compute Cloud (Amazon EC2) RunInstances API to create an EC2 instances.
Some of you use AWS APIs to construct infrastructure-as-code, some to inspect and routinely improve your security posture, some others for configuration management, or to provision and to configure high performance compute clusters. The use cases are countless.
As applications and infrastructures become increasingly sophisticated and you work across more AWS services, it becomes increasingly difficult to learn and handle distinct APIs. This challenge is exacerbated when you also use third-party services in your infrastructure, since you have to construct and maintain custom code to handle both the AWS and third-party services together.
Cloud Control API is a standard set of APIs to Create, Read, Update, Delete, and List (CRUDL) sources across hundreds of AWS Services (more being added) and dozens of third-party services (and growing).
It exposes 5 common verbs (CreateResource, GetResource, UpdateResource, DeleteResource, ListResource) to handle the lifecycle of services. For example, to create an Amazon Elastic Container Service (Amazon ECS) cluster or an AWS Lambda function, you call the same CreateResource API, passing as parameters the type and attributes of the resource you want to create: an Amazon ECS cluster or an Lambda function. The enter parameters are defined by an unified resource model using JSON. Similarly, the return types and error messages are uniform across all verbs and all sources.
Cloud Control API provides support for hundreds of AWS sources today, and we will proceed to add support for existing AWS sources across services such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3) in the coming months. It will support new AWS sources typically on the day of launch.
Until today, when I want to get the details about an Lambda function or a Amazon Kinesis stream, I use the get-function API to call Lambda and the portray-stream API to call Kinesis. Notice in the example below how different these 2 API calls are: they have different names, different naming conventions, different JSON outputs, etc.
aws lambda get-function --function-name TictactoeDatabaseCdkStack
{
"Configuration": {
"FunctionName": "TictactoeDatabaseCdkStack",
"FunctionArn": "arn:aws:lambda:us-west-2:0123456789:function:TictactoeDatabaseCdkStack",
"Runtime": "nodejs14.x",
"Role": "arn:aws:iam::0123456789:position/TictactoeDatabaseCdkStack",
"Handler": "framework.onEvent",
"CodeSize": 21539,
"Timeout": 900,
"MemorySize": 128,
"LastModified": "2021-06-07T11:26:39.767+0000",
...
aws kinesis portray-stream --stream-name AWSNewsBlog
{
"StreamDescription": {
"Shards": [
{
"ShardId": "shardId-000000000000",
"HashKeyRange": {
"StartingHashKey": "0",
"EndingHashKey": "340282366920938463463374607431768211455"
},
"SequenceNumberRange": {
"StartingSequenceNumber": "49622132796672989268327879810972713309953024040638611458"
}
}
],
"StreamARN": "arn:aws:kinesis:us-west-2:012345678901:stream/AWSNewsBlog",
"StreamName": "AWSNewsBlog",
"StreamStatus": "ACTIVE",
"RetentionPeriodHours": 24,
"EncryptionType": "NONE",
"KeyId": null,
"StreamCreationTimestamp": "2021-09-17T14:58:20+02:00"
}
}
In contrast, when using the Cloud Control API, I use a single API name get-resource, and I receive a constant output.
aws cloudcontrol get-resource
--type-name AWS::Kinesis::Stream
--identifier NewsBlogDemo
{
"TypeName": "AWS::Kinesis::Stream",
"ResourceDescription": {
"Identifier": "NewsBlogDemo",
"Properties": "{"Arn":"arn:aws:kinesis:us-west-2:486652066693:stream/NewsBlogDemo","RetentionPeriodHours":168,"Name":"NewsBlogDemo","ShardCount":3}"
}
}
Similary, to create the resource above I used the create-resource API.
aws cloudcontrol create-resource
--type-name AWS::Kinesis::Stream
--desired-state "{"Name": "NewsBlogDemo","RetentionPeriodHours":168, "ShardCount":3}"
In my opinion, there are 3 types of builders that are going to adopt Cloud Control API:
Builders
The first community is builders using AWS Services APIs to handle their infrastructure or their customer’s infrastructure. The ones requiring utilization of low-level AWS Services APIs rather than higher level tools. For example, I know companies that manages AWS infrastructures on behalf of their clients. Many developed solutions to list and portray all sources deployed in their client’s AWS Accounts, for management and billing purposes. Often, they constructed specific tools to address their requirements, but find it hard to hold up with new AWS Services and features. Cloud Control API simplifies this type of tools by offering a constant, resource-centric approach. It makes simpler to hold up with new AWS Services and features.
Another example: Stedi is a developer-focused platform for building automated Electronic Data Interchange (EDI) solutions that combine with any business system. “We have a strong focus on infrastructure as code (IaC) inside Stedi and have been looking for a programmatic way to discover and delete legacy cloud sources that are no longer managed through CloudFormation – helping us reduce complexity and handle cost,” said Olaf Conjin, Serverless Engineer at Stedi, Inc. “With AWS Cloud Control API, our teams can easily list each of these legacy sources, cross-reference them against CloudFormation managed sources, apply additional logic and delete the legacy sources. By deleting these unused legacy sources using Cloud Control API, we can handle our cloud spend in a simpler and faster manner. Cloud Control API allows us to remove the need to author and maintain custom code to discover and delete each type of resource, helping us improve our developer velocity”.
APN Partners
The second community that benefits from Cloud Control API is APN Partners, such as HashiCorp (maker of Terraform) and Pulumi, and other APN Partners offering solutions that depends on AWS Services APIs. When AWS releases a new service or feature, our partner’s engineering teams need to learn, combine, and test a new set of AWS Service APIs to expose it in their choices. This is a time consuming process and often leads to a lag between the AWS release and the availability of the service or feature in their resolution. With the new Cloud Control API, companions are now able to construct a unique REST API code base, using unified API verbs, common enter parameters, and common error types. They just have to merge the standardized pre-defined uniform resource model to interact with new AWS Services uncovered as REST sources.
Launch Partners
HashiCorp and Pulumi are our launch companions, both solutions are integrated with Cloud Control API today.
HashiCorp provides cloud infrastructure automation software that enables organizations to provision, secure, connect, and run any infrastructure for any application. “AWS Cloud Control API makes it simpler for our teams to construct solutions to combine with new and existing AWS services,” said James Bayer – EVP Product, HashiCorp. “Integrating HashiCorp Terraform with AWS Cloud Control API means developers are able to use the newly launched AWS features and services, typically on the day of launch.”
Pulumi’s new AWS Native Provider, powered by the AWS Cloud Control API, “gives Pulumi’s users faster entry to the latest AWS innovations, typically the day they launch, without any need for us to manually implement support,” said Joe Duffy, CEO at Pulumi. “The full surface area of AWS sources provided by AWS Cloud Control API can now be automated from acquainted languages like Python, TypeScript, .NET, and Go, with standard IDEs, package managers, and test frameworks, with high fidelity and worthy quality. Using this new provider, developers and infrastructure teams can develop and ship modern AWS applications and infrastructure faster and with more confidence than ever before.”
To learn more about HashiCorp and Pulumi’s integration with Cloud Control API, refer to their blog post and bulletins.
AWS Customers
The third type of builders that will advantage from Cloud Control API is AWS clients using resolution such as Terraform or Pulumi. You can advantage from Cloud Control API too. For example, when using the new Terraform AWS Cloud Control provider or Pulumi’s AWS Native Provider, you can advantage from availability of new AWS Services and features typically on the day of launch.
Now that you understand the benefits, let’s see Cloud Control API in action.
How It Works?
To start using Cloud Control API, I first make sure I use the latest AWS Command Line Interface (CLI) version. Depending on how the CLI was installed, there are different methods to update the CLI. Cloud Control API is available from our AWS SDKs as well.
To create an AWS Lambda function, I first create an index.py handler, I zip it, and upload the zip file to 1 of my private bucket. I pay consideration that the S3 bucket is in the same AWS Region where I will create the Lambda function:
cat << EOF > index.py
import json
def lambda_handler(event, context):
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
EOF
zip index.zip index.py
aws s3 cp index.zip s3://private-bucket-seb/index.zipThen, I call the create-resource API, passing the same set of arguments as required by the corresponding CloudFormation resource. In this example, the Code, Role, Runtime, and Handler arguments are mandatory, as per the CloudFormation AWS::Lambda::Function documentation.
aws cloudcontrol create-resource
--type-name AWS::Lambda::Function
--desired-state '{"Code":{"S3Bucket":"private-bucket-seb","S3Key":"index.zip"},"Role":"arn:aws:iam::0123456789:position/lambda_basic_execution","Runtime":"python3.9","Handler":"index.lambda_handler"}'
--client-token 123
{
"ProgressEvent": {
"TypeName": "AWS::Lambda::Function",
"RequestToken": "56a0782b-2b26-491c-b082-18f63d571bbd",
"Operation": "CREATE",
"OperationStatus": "IN_PROGRESS",
"EventTime": "2021-09-26T12:05:42.210000+02:00"
}
}I may call the same command again to get the status or to learn about an eventual error:
aws cloudcontrol create-resource
--type-name AWS::Lambda::Function
--desired-state '{"Code":{"S3Bucket":"private-bucket-seb","S3Key":"index.zip"},"Role":"arn:aws:iam::0123456789:position/lambda_basic_execution","Runtime":"python3.9","Handler":"index.lambda_handler"}'
--client-token 123
{
"ProgressEvent": {
"TypeName": "AWS::Lambda::Function",
"Identifier": "ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9",
"RequestToken": "f634b21d-22ed-41bb-9612-8740297d20a3",
"Operation": "CREATE",
"OperationStatus": "SUCCESS",
"EventTime": "2021-09-26T19:46:46.643000+02:00"
}
}
Here, the OperationStatus is SUCCESS and the function name is ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 (I can pass my own name if I want something more descriptive 🙂 )
I then invoke the Lambda function to ensure it works as anticipated:
aws lambda invoke
--function-name ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9
out.txt && cat out.txt && rm out.txt
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
{"statusCode": 200, "body": ""Hello from Lambda!""}When completed, I delete the Lambda function using Cloud Control API:
aws cloudcontrol delete-resource
--type-name AWS::Lambda::Function
--identifier ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9
{
"ProgressEvent": {
"TypeName": "AWS::Lambda::Function",
"Identifier": "ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9",
"RequestToken": "8923991d-72b3-4981-8160-4d9a585965a3",
"Operation": "DELETE",
"OperationStatus": "IN_PROGRESS",
"EventTime": "2021-09-26T20:06:22.013000+02:00"
}
}Idempotency
You might have noticed the client-token parameter I passed to the create-resource API call. Create, Update, and Delete requests all accept a ClientToken, which is used to ensure idempotency of the request.
- We recommend always passing a client token. This will disambiguate requests in case a retry is desired. Otherwise, you may encounter unexpected errors like
ConcurrentOperationExceptionorAlreadyExists. - We recommend that client tokens always be unique for every single request, such as by passing a UUID.
One More Thing
At the heart of AWS Cloud Control API source of data, there is the CloudFormation Public Registry, which my colleague Steve announced last June in this blog post. It allows anyone to expose a set of AWS sources through CloudFormation and AWS CDK. This is the mechanism AWS Service teams are now using to release their services and features as CloudFormation and AWS CDK sources. Multiple third-party distributors are also publishing their solutions in the CloudFormation Public Registry. All sources revealed are modelled with a standard schema that defines the resource, its properties, and their attributes in a uniform way.
AWS Cloud Control API is a CRUDL API layer on top of sources revealed in the CloudFormation Public Registry. Any resource revealed in the registry exposes its attributes with standard JSON schemas. The resource can then be created, updated, deleted, or listed using Cloud Control API with no additional work.
For example, imagine I choose to expose a public CloudFormation stack to let any AWS customer create VPN servers, based on EC2 instances. I model the VPNServer resource type and publish it in the CloudFormation Public Registry. With no additional work on my side, my custom resource “VPNServer” is now available to all AWS clients through the Cloud Control API REST API. Not only, it is also routinely available through solutions like Hashicorp’s Terraform and Pulumi, and probably others who adopt Cloud Control API in the prospective.
It is worth mentioning Cloud Control API is not aimed at replacing the traditional AWS service-level APIs. They are nonetheless there and will always be there, but we think that Cloud Control API is simpler and more constant to use and you should use it for new apps.
Availability and Pricing
Cloud Control API is available in all AWS Regions, except China.
You will only pay for the utilization of underlying AWS sources, such as a CloudWatch logs or Lambda functions invocations, or pay for the number of handler operations and handler operation period associated with using third-party sources (such as Datadog monitors or MongoDB Atlas clusters). There are no minimal fees and no required upfront commitments.
I can’t wait to discover what you are going to construct on top of this new Cloud Control API. Go construct!
