From 59db6d158016fc4faed0010ec555c4f58b393e56 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Tue, 21 Nov 2023 23:28:13 -0600 Subject: [PATCH 1/9] Adopt AEP-155: Idempotency --- aep/general/0155/aep.md | 106 ++++++++++++++++++++++++++++++++++++++ aep/general/0155/aep.yaml | 9 ++++ 2 files changed, 115 insertions(+) create mode 100644 aep/general/0155/aep.md create mode 100644 aep/general/0155/aep.yaml diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md new file mode 100644 index 00000000..5835e4ae --- /dev/null +++ b/aep/general/0155/aep.md @@ -0,0 +1,106 @@ +# Idempotency + +It is sometimes useful for an API to have a unique, customer-provided +identifier for particular requests. This can be useful for several purposes, +such as de-duplicating requests from parallel processes, ensuring the safety of +retries, or auditing. + +The purpose of idempotency keys is to provide idempotency guarantees: allowing +the same request to be issued more than once without subsequent calls having +any effect. In the event of a network failure, the client can retry the +request, and the server can detect duplication and ensure that the request is +only processed once. + +## Guidance + +APIs **may** add a `string idempotency_key` parameter to request messages +(including those of standard methods) in order to uniquely identify particular +requests. + +```proto +message CreateBookRequest { + // The parent resource where this book will be created. + // Format: publishers/{publisher} + string parent = 1 [ + (google.api.field_behavior) = REQUIRED, + (google.api.resource_reference) = { + child_type: "library.example.com/Book" + }]; + + // The book to create. + Book book = 2 [(google.api.field_behavior) = REQUIRED]; + + // A unique identifier for this request. Restricted to 36 ASCII characters. + // A random UUID is recommended. + // This request is only idempotent if a `idempotency_key` is provided. + string idempotency_key = 3 [(google.api.field_info).format = UUID4]; +} +``` + +- Providing an idempotency key **must** guarantee idempotency. + - If a duplicate request is detected, the server **should** return the + response for the previously successful request, because the client most + likely did not receive the previous response. + - APIs **may** choose any reasonable timeframe for honoring idempotency keys. +- The `idempotency_key` field **must** be provided on the request message to + which it applies (and it **must not** be a field on resources themselves). +- Idempotency keys **should** be optional. +- Idempotency keys **should** be able to be UUIDs, and **may** allow UUIDs to + be the only valid format. The format restrictions for idempotency keys + **must** be documented. + - Idempotency keys that are UUIDs **must** be annotated with the + `google.api.FieldInfo.Format` value `UUID4` using the extension + `(google.api.field_info).format = UUID4`. See [AEP-202](../0202/aep.md) for + more. + +### Stale success responses + +In some unusual situations, it may not be possible to return an identical +success response. For example, a duplicate request to create a resource may +arrive after the resource has not only been created, but subsequently updated; +because the service has no other need to retain the historical data, it is no +longer feasible to return an identical success response. + +In this situation, the method **may** return the current state of the resource +instead. In other words, it is permissible to substitute the historical success +response with a similar response that reflects more current data. + +## Further reading + +- For which codes to retry, see [AEP-194](https://aep.dev/194). +- For how to retry errors in client libraries, see + [AEP-4221](https://aep.dev/4221). + +## Rationale + +### Naming the field `idempotency_key` + +The original content from which this AEP is derived defines a `request_id` +field; we define `idempotency_key` instead for two reasons: + +1. There is an [active Internet-Draft][idempotency-key-draft] to standardize an + HTTP header named `Idempotency-Key`. +1. There may be edge cases in which separately identifying idempotent requests + is useful; `request_id` would be more appropriate for such use cases. For + example, an API producer might be testing the idempotency behavior of the + API server, and might want to issue multiple requests with the same + `idempotency_key` and trace the behavior of each request separately. + + +[idempotency-key-draft]: https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/ + + +### Using UUIDs for request identification + +When a value is required to be unique, leaving the format open-ended can lead +to API consumers incorrectly providing a duplicate identifier. As such, +standardizing on a universally unique identifier drastically reduces the chance +for collisions when done correctly. + +## Changelog + +- **2023-11-21**: Adopt AEP from from Google's AIP; rename field from + `request_id` to `idempotency_key`. +- **2023-10-02**: Add UUID format extension guidance. +- **2019-08-01**: Changed the examples from "shelves" to "publishers", to + present a better example of resource ownership. diff --git a/aep/general/0155/aep.yaml b/aep/general/0155/aep.yaml new file mode 100644 index 00000000..a91ba302 --- /dev/null +++ b/aep/general/0155/aep.yaml @@ -0,0 +1,9 @@ +--- +id: 155 +state: approved +slug: idempotency +created: 2019-05-06 +placement: + category: design-patterns + order: 60 +redirect_from: '/idempotency' From 79cdf96ae8f77f16c6ea124a6a8e164697501f56 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Wed, 29 Nov 2023 16:26:35 -0600 Subject: [PATCH 2/9] Address Yusuke's comments. --- aep/general/0155/aep.md | 9 ++++++--- aep/general/0155/aep.yaml | 3 +-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index 5835e4ae..b6fd980c 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -2,8 +2,11 @@ It is sometimes useful for an API to have a unique, customer-provided identifier for particular requests. This can be useful for several purposes, -such as de-duplicating requests from parallel processes, ensuring the safety of -retries, or auditing. +such as: + +- De-duplicating requests from parallel processes +- Ensuring the safety of retries +- Auditing The purpose of idempotency keys is to provide idempotency guarantees: allowing the same request to be issued more than once without subsequent calls having @@ -41,7 +44,7 @@ message CreateBookRequest { - If a duplicate request is detected, the server **should** return the response for the previously successful request, because the client most likely did not receive the previous response. - - APIs **may** choose any reasonable timeframe for honoring idempotency keys. + - APIs **should** honor idempotency keys for at least an hour. - The `idempotency_key` field **must** be provided on the request message to which it applies (and it **must not** be a field on resources themselves). - Idempotency keys **should** be optional. diff --git a/aep/general/0155/aep.yaml b/aep/general/0155/aep.yaml index a91ba302..5787cc93 100644 --- a/aep/general/0155/aep.yaml +++ b/aep/general/0155/aep.yaml @@ -4,6 +4,5 @@ state: approved slug: idempotency created: 2019-05-06 placement: - category: design-patterns + category: requests order: 60 -redirect_from: '/idempotency' From 1aff7c136b5b04f1484c002f5fc67953be7af596 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Wed, 6 Dec 2023 01:23:15 -0500 Subject: [PATCH 3/9] Update aep/general/0155/aep.md --- aep/general/0155/aep.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index b6fd980c..f7c8e845 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -48,7 +48,7 @@ message CreateBookRequest { - The `idempotency_key` field **must** be provided on the request message to which it applies (and it **must not** be a field on resources themselves). - Idempotency keys **should** be optional. -- Idempotency keys **should** be able to be UUIDs, and **may** allow UUIDs to +- Idempotency keys **must** be able to be UUIDs, and **may** allow UUIDs to be the only valid format. The format restrictions for idempotency keys **must** be documented. - Idempotency keys that are UUIDs **must** be annotated with the From bf854cebdb2a103dc871a6bf55138157d057d1d2 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Tue, 5 Dec 2023 22:24:56 -0800 Subject: [PATCH 4/9] lint --- aep/general/0155/aep.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index f7c8e845..1b08318f 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -48,9 +48,9 @@ message CreateBookRequest { - The `idempotency_key` field **must** be provided on the request message to which it applies (and it **must not** be a field on resources themselves). - Idempotency keys **should** be optional. -- Idempotency keys **must** be able to be UUIDs, and **may** allow UUIDs to - be the only valid format. The format restrictions for idempotency keys - **must** be documented. +- Idempotency keys **must** be able to be UUIDs, and **may** allow UUIDs to be + the only valid format. The format restrictions for idempotency keys **must** + be documented. - Idempotency keys that are UUIDs **must** be annotated with the `google.api.FieldInfo.Format` value `UUID4` using the extension `(google.api.field_info).format = UUID4`. See [AEP-202](../0202/aep.md) for From f4573c2b0d29a1b47e2593155985cf37b04f104e Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Tue, 5 Dec 2023 22:29:29 -0800 Subject: [PATCH 5/9] Qualify proto-specific guidance. --- aep/general/0155/aep.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index 1b08318f..5316b5ce 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -51,10 +51,10 @@ message CreateBookRequest { - Idempotency keys **must** be able to be UUIDs, and **may** allow UUIDs to be the only valid format. The format restrictions for idempotency keys **must** be documented. - - Idempotency keys that are UUIDs **must** be annotated with the - `google.api.FieldInfo.Format` value `UUID4` using the extension - `(google.api.field_info).format = UUID4`. See [AEP-202](../0202/aep.md) for - more. + - When using protocol buffers, idempotency keys that are UUIDs **must** be + annotated with the `google.api.FieldInfo.Format` value `UUID4` using the + extension `(google.api.field_info).format = UUID4`. See + [AEP-202](../0202/aep.md) for more. ### Stale success responses From 30bb416774393015e119bd6cf87d910b3f463aff Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Thu, 14 Dec 2023 15:36:17 -0600 Subject: [PATCH 6/9] Update to reflect new common components and live discussion. --- aep/general/0155/aep.md | 81 +++++++++++++++++++++++++++++++---------- 1 file changed, 62 insertions(+), 19 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index 5316b5ce..782aef35 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -16,9 +16,10 @@ only processed once. ## Guidance -APIs **may** add a `string idempotency_key` parameter to request messages -(including those of standard methods) in order to uniquely identify particular -requests. +APIs **may** add a `aep.api.IdempotencyKey idempotency_key` parameter to +request messages (including those of standard methods) in order to uniquely +identify particular requests. API servers **must not** execute requests with +the same `idempotency_key` more than once. ```proto message CreateBookRequest { @@ -33,28 +34,57 @@ message CreateBookRequest { // The book to create. Book book = 2 [(google.api.field_behavior) = REQUIRED]; - // A unique identifier for this request. Restricted to 36 ASCII characters. - // A random UUID is recommended. - // This request is only idempotent if a `idempotency_key` is provided. - string idempotency_key = 3 [(google.api.field_info).format = UUID4]; + // This request is only idempotent if `idempotency_key` is provided. + // + // This key will be honored for at least one hour after the first time it is + // seen by the server. + // + // The key is restricted to 36 ASCII characters. A random UUID is recommended. + aep.api.IdempotencyKey idempotency_key = 3 [ + (aep.api.field_info).minimum_lifetime = { seconds: 3600 } + ]; } ``` +- [`aep.api.IdempotencyKey`][] has a `key` and a `first_sent` timestamp. + + - `key` is simply a unique identifier. + - Providing an idempotency key **must** guarantee idempotency. - - If a duplicate request is detected, the server **should** return the - response for the previously successful request, because the client most - likely did not receive the previous response. + + - If a duplicate request is detected, the server **must** return either: + + - A response equivalent to the response for the previously successful + request, because the client most likely did not receive the previous + response; OR + - An error indicating that the `first_sent` field of the idempotency key is + invalid (expired, in the future, or differs from a previous `first_sent` + value with the same `key`); OR + - An error, if returning an equivalent response is not possible. + + For example, if a resource was created, then deleted, and then a + duplicate request to create the resource is received, the server **may** + return an error if returning the previously created resource is not + possible. + - APIs **should** honor idempotency keys for at least an hour. + - When using protocol buffers, idempotency keys that are UUIDs **must** be + annotated with a minimum lifetime using the extension + [`(aep.api.field_info).minimum_lifetime`]. + - The `idempotency_key` field **must** be provided on the request message to which it applies (and it **must not** be a field on resources themselves). + + - The `first_sent` field can be used by API servers to determine if a key is + expired. API servers **must** reject requests with expired keys, and + **must** reject requests with keys that are in the future. When feasible, + API servers **should** reject requests that use the same `key` but have a + different `first_sent` timestamp. + - The `key` field **must** be able to be a UUID, and **may** allow UUIDs to + be the only valid format. The format restrictions for idempotency keys + **must** be documented. + - Idempotency keys **should** be optional. -- Idempotency keys **must** be able to be UUIDs, and **may** allow UUIDs to be - the only valid format. The format restrictions for idempotency keys **must** - be documented. - - When using protocol buffers, idempotency keys that are UUIDs **must** be - annotated with the `google.api.FieldInfo.Format` value `UUID4` using the - extension `(google.api.field_info).format = UUID4`. See - [AEP-202](../0202/aep.md) for more. ### Stale success responses @@ -91,6 +121,8 @@ field; we define `idempotency_key` instead for two reasons: [idempotency-key-draft]: https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/ +[`aep.api.IdempotencyKey`]: https://buf.build/aep/api/file/main:aep/api/idempotency_key.proto#L21 +[`(aep.api.field_info).minimum_lifetime`]: https://buf.build/aep/api/file/main:aep/api/field_info.proto#L35 ### Using UUIDs for request identification @@ -102,8 +134,19 @@ for collisions when done correctly. ## Changelog -- **2023-11-21**: Adopt AEP from from Google's AIP; rename field from - `request_id` to `idempotency_key`. +- **2023-11-21**: Adopt AEP from from Google's AIP with the following changes: + - Rename field from `request_id` to `idempotency_key` (plus some minor + releated rewording). + - Add a common component [`aep.api.IdempotencyKey`][] and use this rather + than `string` for the `idempotency_key` field; add related guidance about + `IdempotencyKey.first_seen`. + - Remove guidance about annotating `idempotency_key` with + `(google.api.field_info).format`. + - Add guidance about annotating `idempotency_key` with + [`(aep.api.field_info).minimum_lifetime`]. + - Update guidance about responses to be more explicit about success and error + cases, while allowing "equivalent" rather than identical responses for + subsequent requests. - **2023-10-02**: Add UUID format extension guidance. - **2019-08-01**: Changed the examples from "shelves" to "publishers", to present a better example of resource ownership. From 2b999adee2c7fbdcc43671c12456076adc3db4a3 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Wed, 20 Dec 2023 12:41:53 -0600 Subject: [PATCH 7/9] Minor rephrasing of list. --- aep/general/0155/aep.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index 782aef35..acb48d9d 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -52,14 +52,14 @@ message CreateBookRequest { - Providing an idempotency key **must** guarantee idempotency. - - If a duplicate request is detected, the server **must** return either: + - If a duplicate request is detected, the server **must** return one of: - A response equivalent to the response for the previously successful request, because the client most likely did not receive the previous - response; OR + response. - An error indicating that the `first_sent` field of the idempotency key is invalid (expired, in the future, or differs from a previous `first_sent` - value with the same `key`); OR + value with the same `key`). - An error, if returning an equivalent response is not possible. For example, if a resource was created, then deleted, and then a From 567c4abbb34877fe61f8c202cc0ef4c98cca2684 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Wed, 20 Dec 2023 12:42:54 -0600 Subject: [PATCH 8/9] Clarify invalid `first_sent`. --- aep/general/0155/aep.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index acb48d9d..a7b7037d 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -58,8 +58,8 @@ message CreateBookRequest { request, because the client most likely did not receive the previous response. - An error indicating that the `first_sent` field of the idempotency key is - invalid (expired, in the future, or differs from a previous `first_sent` - value with the same `key`). + invalid or cannot be honored (expired, in the future, or differs from a + previous `first_sent` value with the same `key`). - An error, if returning an equivalent response is not possible. For example, if a resource was created, then deleted, and then a From a14604533246f14711b8a28be1bfef8b29179e84 Mon Sep 17 00:00:00 2001 From: Richard Frankel Date: Wed, 20 Dec 2023 13:56:33 -0600 Subject: [PATCH 9/9] Remove section about stale success responses. --- aep/general/0155/aep.md | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/aep/general/0155/aep.md b/aep/general/0155/aep.md index a7b7037d..2daa9068 100644 --- a/aep/general/0155/aep.md +++ b/aep/general/0155/aep.md @@ -70,7 +70,7 @@ message CreateBookRequest { - APIs **should** honor idempotency keys for at least an hour. - When using protocol buffers, idempotency keys that are UUIDs **must** be annotated with a minimum lifetime using the extension - [`(aep.api.field_info).minimum_lifetime`]. + [`(aep.api.field_info).minimum_lifetime`][]. - The `idempotency_key` field **must** be provided on the request message to which it applies (and it **must not** be a field on resources themselves). @@ -86,18 +86,6 @@ message CreateBookRequest { - Idempotency keys **should** be optional. -### Stale success responses - -In some unusual situations, it may not be possible to return an identical -success response. For example, a duplicate request to create a resource may -arrive after the resource has not only been created, but subsequently updated; -because the service has no other need to retain the historical data, it is no -longer feasible to return an identical success response. - -In this situation, the method **may** return the current state of the resource -instead. In other words, it is permissible to substitute the historical success -response with a similar response that reflects more current data. - ## Further reading - For which codes to retry, see [AEP-194](https://aep.dev/194). @@ -134,7 +122,7 @@ for collisions when done correctly. ## Changelog -- **2023-11-21**: Adopt AEP from from Google's AIP with the following changes: +- **2023-23-20**: Adopt AEP from from Google's AIP with the following changes: - Rename field from `request_id` to `idempotency_key` (plus some minor releated rewording). - Add a common component [`aep.api.IdempotencyKey`][] and use this rather @@ -147,6 +135,8 @@ for collisions when done correctly. - Update guidance about responses to be more explicit about success and error cases, while allowing "equivalent" rather than identical responses for subsequent requests. + - Temporarily removed the section about stale success responses, pending + further discussion. - **2023-10-02**: Add UUID format extension guidance. - **2019-08-01**: Changed the examples from "shelves" to "publishers", to present a better example of resource ownership.