Dynamodb OnDemandThroughput max write/read requests units setup problems

[ferran-juan]

Describe what happened

While adding the recent functionality for dynamodb tables for max_read_request_units and max_wirte_request_units I found some problems that forces me to not use this option.
We have a library where engineers can use to deploy pulumi resources by passing some inputs. For our dynamodb resources, we added the TableOnDemandThroughputArgs set to -1 by default which means no limit, this works, but it always cause drift in the pulumi execution because in AWS this value is none and the sdk is going to change it to -1 to no limit, fair enough.
The problem is for createTable operations, this value -1 is throwing an error saying the value is out of range.

SDK error:

operation error DynamoDB: CreateTable, https response error StatusCode: 400, RequestID: G61AFPHHV0POGANSNRJ4DEBEIJVV4KQNSO5AEMVJF66Q9ASUAAJG, api error ValidationException: One or more parameter values were invalid: Requested MaxReadRequestUnits for OnDemandThroughput for table is outside of valid range: [email protected]

So, why not set this value to none by default? For create table operations this works well, but in any moment the dynamodb table have some value for this args, if you want to remove it, you cannot simply set again to none the TableOnDemandThroughputArgs because the sdk complains saying this value is mandatory, so we need to set it to -1 by default, causing drift every time Pulumi runs.

SDK error:

 operation error DynamoDB: UpdateTable, https response error StatusCode: 400, RequestID: 5V4CESNVI0Q9NAMRF6ANUV5GBBVV4KQNSO5AEMVJF66Q9ASUAAJG, api error ValidationException: At least one of ProvisionedThroughput, BillingMode, UpdateStreamEnabled, GlobalSecondaryIndexUpdates, SSESpecification, ReplicaUpdates, DeletionProtectionEnabled, OnDemandThroughput, WarmThroughput or TableClass is required: [email protected]

To be able to use this settings, we have to find some workaround to identify if the table actually exists to check the real infrastructure values and potential ignore changes for this in case the value have to be none while the code is -1 by default and, if the table do not exist, set the TableOnDemandThroughputArgs to none. This is not ideal because it adds complexity in our code so we decided to not use this functionality.

Sample program

This is an example of both dynamoDBTableArgs and the actual dynamodb Table creation inside our code. It is not functional alone, but you can update it to have specific values and pass the args to the table.

self.dynamoDBTableArgs = aws.dynamodb.TableArgs(
            name=name,
            billing_mode="PAY_PER_REQUEST" if billing_mode is None else billing_mode,
            attributes=attributes,
            # This should only be set when the table is ready to delete, which will turn delete protection off
            deletion_protection_enabled=(not ready_to_delete) if ready_to_delete is not None else True,
            hash_key=hash_key,
            range_key=range_key,
            global_secondary_indexes=global_secondary_indexes,
            local_secondary_indexes=local_secondary_indexes,
            read_capacity=read_capacity,
            write_capacity=write_capacity,
            on_demand_throughput=TableOnDemandThroughputArgs(
                # Setting -1 indicates no limit. This is useful when the value is removed from the yaml file.
                # This ensures that the value is updated correctly instead of retaining a previous value.
                max_read_request_units=-1 if max_read_request_units is None else max_read_request_units,
                max_write_request_units=-1 if max_write_request_units is None else max_write_request_units,
            )
            if max_read_request_units is not None or max_write_request_units is not None  # <-- This do not work in update operations if the values are removed
            else None,
            ttl=ttl,
            tags=tags,
        )


dynamo_db_table = aws.dynamodb.Table(
            resource_name=self._name,
            name=self._name,
            billing_mode=self.args.dynamoDBTableArgs.billing_mode,
            attributes=get_table_attributes(self.args.dynamoDBTableArgs.attributes),
            hash_key=self.args.dynamoDBTableArgs.hash_key,
            range_key=self.args.dynamoDBTableArgs.range_key,
            global_secondary_indexes=self.args.dynamoDBTableArgs.global_secondary_indexes,
            local_secondary_indexes=self.args.dynamoDBTableArgs.local_secondary_indexes,
            read_capacity=self.args.dynamoDBTableArgs.read_capacity,
            write_capacity=self.args.dynamoDBTableArgs.write_capacity,
            on_demand_throughput=self.args.dynamoDBTableArgs.on_demand_throughput
            if self.args.dynamoDBTableArgs.billing_mode == "PAY_PER_REQUEST"
            else None,
            ttl=self.args.dynamoDBTableArgs.ttl,
            point_in_time_recovery=TablePointInTimeRecoveryArgs(enabled=True),
            deletion_protection_enabled=self.args.dynamoDBTableArgs.deletion_protection_enabled,
            tags=self.args.dynamoDBTableArgs.tags,
            opts=ResourceOptions(parent=self),
        )

Log output

operation error DynamoDB: CreateTable, https response error StatusCode: 400, RequestID: G61AFPHHV0POGANSNRJ4DEBEIJVV4KQNSO5AEMVJF66Q9ASUAAJG, api error ValidationException: One or more parameter values were invalid: Requested MaxReadRequestUnits for OnDemandThroughput for table is outside of valid range: [email protected]

operation error DynamoDB: UpdateTable, https response error StatusCode: 400, RequestID: 5V4CESNVI0Q9NAMRF6ANUV5GBBVV4KQNSO5AEMVJF66Q9ASUAAJG, api error ValidationException: At least one of ProvisionedThroughput, BillingMode, UpdateStreamEnabled, GlobalSecondaryIndexUpdates, SSESpecification, ReplicaUpdates, DeletionProtectionEnabled, OnDemandThroughput, WarmThroughput or TableClass is required: [email protected]

Affected Resource(s)

https://www.pulumi.com/registry/packages/aws/api-docs/dynamodb/table/#aws-dynamodb-table

Output of pulumi about

CLI          
Version      3.109.0
Go Version   go1.22.0
Go Compiler  gc

Plugins
NAME             VERSION
aws              6.66.2
datadog          4.25.1
local            1.2.1
pulumi_aws_tags  1.2.0
python           unknown

Host     
OS       darwin
Version  14.7.1
Arch     arm64

This project is written in python: executable='/Users/xxx.xxxx./.pyenv/shims/python3' version='3.11.6'

Backend        
Name           
URL            s3://
User           xxx.xxxx
Organizations  
Token type     personal

Dependencies:
NAME                      VERSION
aws-lambda-typing         2.20.0
awscli-local              0.21.1
backend-common            1.91.0
backend-common-dev        2.11.0
boto3-stubs               1.34.97
chardet                   4.0.0
elevated-access-common    0.58.1
email-validator           2.0.0.post2
exceptiongroup            1.1.3
fastapi                   0.109.2
flake8                    3.8.4
httpx                     0.27.0
importlib-resources       5.12.0
install                   1.3.5
isort                     5.13.2
jsonschema                4.17.3
localstack                3.0.1
mock                      5.1.0
mypy                      1.11.0
mypy-boto3-dynamodb       1.34.97
pep517                    0.13.1
pre-commit                3.6.2
pulumi-local              1.2.1
pydantic-settings         2.3.4
pytest-cov                4.1.0
pytest-env                0.8.2
tomli                     2.0.1
uvicorn                   0.27.1

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[corymhall]

@ferran-juan thanks for raising this enhancement request. We have a separate issue in pulumi/pulumi which I think is related to this which I have linked. Unfortunately I don't know of a good workaround right now that will enable you to create a generic solution. You may have to expose these properties themselves to the user so they can make the determination of which value to provide.

[ferran-juan]

Thanks @corymhall for your response! The possibilities mentioned in that issue make sense to me and could provide a very useful setup, helping to address this issue.

Perhaps the SDK behaviour might be out of scope here, as it falls under AWS's domain? If, by their documentation, -1 represents no limit, passing -1 during creation should be correct and also it shouldn't cause drift when updating a table that already has a no-limit setting which are, in essence, the main problems.