From c67892ab4a0c115c1c8703ca38a19ebe084c03ea Mon Sep 17 00:00:00 2001 From: Si Beaumont Date: Thu, 19 Oct 2023 15:01:36 +0100 Subject: [PATCH] Add proposal SOAR-0008: OpenAPI document filtering (#303) ### Motivation We'd like to run a proposal for filtering the OpenAPI document for just the required parts prior to generating. ### Modifications - Add SOAR-0008: OpenAPI document filtering (See the proposal itself for details)[^1] ### Result n/a ### Test Plan n/a ### Related Issues #285 [^1]: https://github.com/apple/swift-openapi-generator/pull/303/files --------- Signed-off-by: Si Beaumont --- .../Documentation.docc/Proposals/Proposals.md | 1 + .../Documentation.docc/Proposals/SOAR-0001.md | 2 +- .../Documentation.docc/Proposals/SOAR-0002.md | 2 +- .../Documentation.docc/Proposals/SOAR-0003.md | 2 +- .../Documentation.docc/Proposals/SOAR-0004.md | 2 +- .../Documentation.docc/Proposals/SOAR-0005.md | 2 +- .../Documentation.docc/Proposals/SOAR-0007.md | 2 +- .../Documentation.docc/Proposals/SOAR-0008.md | 491 ++++++++++++++++++ 8 files changed, 498 insertions(+), 6 deletions(-) create mode 100644 Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0008.md diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/Proposals.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/Proposals.md index 28fa2533..2e505a09 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/Proposals.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/Proposals.md @@ -49,3 +49,4 @@ If you have any questions, tag [Honza Dvorsky](https://github.com/czechboy0) or - - - +- diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0001.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0001.md index 91aff9b2..300903a0 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0001.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0001.md @@ -6,7 +6,7 @@ Improved mapping of OpenAPI identifiers to Swift identifiers. - Proposal: SOAR-0001 - Author(s): [Denil](https://github.com/denil-ct) -- Status: **In Preview** +- Status: **Accepted, available since 0.2.0.** - Issue: https://github.com/apple/swift-openapi-generator/issues/21 - Implementation: - https://github.com/apple/swift-openapi-generator/pull/89 diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0002.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0002.md index 4c3f7204..5c6b9bd8 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0002.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0002.md @@ -6,7 +6,7 @@ Improved naming of content types to Swift identifiers. - Proposal: SOAR-0002 - Author(s): [Honza Dvorsky](https://github.com/czechboy0) -- Status: **In Preview** +- Status: **Accepted, available since 0.2.0.** - Issue: N/A, was part of multiple content type support: [apple/swift-openapi-generator#6](https://github.com/apple/swift-openapi-generator/issues/6) and [apple/swift-openapi-generator#7](https://github.com/apple/swift-openapi-generator/issues/7) - Implementation: - [Landed behind a feature flag as part of apple/swift-openapi-generator#146](https://github.com/czechboy0/swift-openapi-generator/blob/4555f8e998b24aa65a462a63828d9195c50dcc23/Sources/_OpenAPIGeneratorCore/Translator/Content/ContentSwiftName.swift#L23-L42) diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0003.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0003.md index da775f40..c93c9bd8 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0003.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0003.md @@ -6,7 +6,7 @@ Generate a dedicated Accept header enum for each operation. - Proposal: SOAR-0003 - Author(s): [Honza Dvorsky](https://github.com/czechboy0), [Si Beaumont](https://github.com/simonjbeaumont) -- Status: **In Preview** +- Status: **Accepted, available since 0.2.0.** - Issue: [apple/swift-openapi-generator#160](https://github.com/apple/swift-openapi-generator/issues/160) - Implementation: - [apple/swift-openapi-runtime#37](https://github.com/apple/swift-openapi-runtime/pull/37) diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0004.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0004.md index db42d68b..6d381d2e 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0004.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0004.md @@ -6,7 +6,7 @@ Represent HTTP request and response bodies as a stream of bytes. - Proposal: SOAR-0004 - Author(s): [Honza Dvorsky](https://github.com/czechboy0) -- Status: **Ready for Implementation** +- Status: **Accepted, available since 0.3.0.** - Issue: [apple/swift-openapi-generator#9](https://github.com/apple/swift-openapi-generator/issues/9) - Implementation: - [apple/swift-openapi-generator#245](https://github.com/apple/swift-openapi-generator/pull/245) diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0005.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0005.md index 234acf1e..5f7ce8a3 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0005.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0005.md @@ -6,7 +6,7 @@ Adopt the new ecosystem-wide Swift HTTP Types package for HTTP currency types in - Proposal: SOAR-0005 - Author(s): [Honza Dvorsky](https://github.com/czechboy0) -- Status: **Ready for Implementation** +- Status: **Accepted, available since 0.3.0.** - Issue: [apple/swift-openapi-generator#101](https://github.com/apple/swift-openapi-generator/issues/101) - Implementation: - [apple/swift-openapi-generator#245](https://github.com/apple/swift-openapi-generator/pull/245) diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0007.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0007.md index 5b676488..ff732fff 100644 --- a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0007.md +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0007.md @@ -7,7 +7,7 @@ operation outputs. - Proposal: SOAR-0007 - Author(s): [Si Beaumont](https://github.com/simonjbeaumont) -- Status: **Implemented** +- Status: **Accepted, available since 0.3.0.** - Review period: 2023-09-22 – 2023-09-29 - [Swift Forums post](https://forums.swift.org/t/proposal-soar-0007-shorthand-apis-for-inputs-and-outputs/67444) - Issue: diff --git a/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0008.md b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0008.md new file mode 100644 index 00000000..e9ececf6 --- /dev/null +++ b/Sources/swift-openapi-generator/Documentation.docc/Proposals/SOAR-0008.md @@ -0,0 +1,491 @@ +# SOAR-0008: OpenAPI document filtering + +Filtering the OpenAPI document for just the required parts prior to generating. + +## Overview + +- Proposal: SOAR-0008 +- Author(s): [Si Beaumont](https://github.com/simonjbeaumont) +- Status: **Accepted, available since 0.3.1.** + - Review period: 2023-09-28 – 2023-10-05 + - [Swift Forums post](https://forums.swift.org/t/proposal-soar-0008-openapi-document-filtering/67574) +- Issue: + - [apple/swift-openapi-generator#285](https://github.com/apple/swift-openapi-generator/issues/285) +- Implementation: + - [apple/swift-openapi-generator#319](https://github.com/apple/swift-openapi-generator/pull/319) +- Feature flag: n/a +- Affected components: + - generator +- Related links: + - [Project scope and goals](https://swiftpackageindex.com/apple/swift-openapi-generator/documentation/swift-openapi-generator/project-scope-and-goals) +- Versions: + - v1 (2023-09-28): Initial version + - v2 (2023-10-05): + - Filtering by tag only includes the tagged operations (cf. whole path) + - Add support for filtering operations by ID + +### Introduction + +When generating client code, Swift OpenAPI Generator generates code for the +entire OpenAPI document, even if the user only makes use of a subset of its +types and operations. + +Generating code that is unused constitutes overhead for the adopter: +- The overhead of generating code for unused types and operations +- The overhead of compiling the generated code +- The overhead of unused code in the users codebase (AOT generation) + +This is particularly noticeable when working with a small subset of +a large API, which can result in O(100k) lines of unused code and long +generation and compile times. + +The initial scope of the Swift OpenAPI Generator was to focus only on +generating Swift code from an OpenAPI document, and any preprocessing of the +OpenAPI document was considered out of scope. The proposed answer to this was +to preprocess the document before providing to the generator[[0]]. + +Even with tooling, filtering the document requires more than just filtering the +YAML or JSON document for the deterministic keys for the desired operations +because such operations likely contain JSON references to the reusable types in +the document's `components` dictionary, and these components can themselves +contain references. Consequently, in order to filter an OpenAPI document for +a single operation requires including the transitive closure of the operations +referenced dependencies. + +Furthermore, it's common that Swift OpenAPI Generator adopters do not own the +OpenAPI document, and simply vendor it from the service owner. In these cases, +it presents a user experience hurdle to have to edit the document, and +a maintenance burden to continue to do so when updating the document to a new +version. + +Because this problem has a general solution that is non-trivial to implement, +this proposal covers adding opt-in, configurable document filtering to the +generator, to improve the user experience for those using a subset of a large +API. + +### Motivation + +Many real-world APIs have hundreds of endpoints and types. Consider the Github +API, whose OpenAPI document is >230k lines. It describes ~900 endpoints and >1200 +reusable components. Running the generator with `--mode types` for this API +takes 41 seconds[^1] and results in >450k LOC, which presents a bottleneck +in the build when compiling the generated code. + +```console +% cat api.github.com.yaml | wc -l +231063 + +% cat api.github.com.yaml | yq '.paths.* | keys' | wc -l +898 + +% cat api.github.com.yaml | yq '.components.* | keys' | wc -l +1260 + +% time ./swift-openapi-generator.release \ + generate \ + --mode types \ + --config openapi-generator-config.yaml \ + api.github.com.yaml +Writing data to file Types.swift... + +real 0m41.397s +user 0m40.912s +sys 0m0.456s + +% cat Types.swift | wc -l +458852 +``` + +OpenAPI has support for grouping operations by tag. For example, the OpenAPI +document for the Github API has the following tags: + +```console +% cat api.github.com.yaml | yq '[.tags[].name] | join(", ")' +actions, activity, apps, billing, checks, code-scanning, codes-of-conduct, +emojis, dependabot, dependency-graph, gists, git, gitignore, issues, licenses, +markdown, merge-queue, meta, migrations, oidc, orgs, packages, projects, pulls, +rate-limit, reactions, repos, search, secret-scanning, teams, users, +codespaces, copilot, security-advisories, interactions, classroom +``` + +If a user wants to make use of just the parts of the API that relate to Github +issues, then they could work with a much smaller document. For example, +filtering for only operations tagged `issues` (including all components on +which those operations depend) results in an OpenAPI document that is just 25k +lines with 40 operations and 90 reusable components, comprising a ~90% +reduction in these dimensions. + +Running the generator with `--mode types` with this filtered API document +takes just 1.6 seconds[^1] and results in < 15k LOC, which is 20x faster and +a 95% reduction in generated code. + +```console +% cat issues.api.github.com.yaml | wc -l +25314 + +% cat issues.api.github.com.yaml | yq '.paths.* | keys' | wc -l +40 + +% cat issues.api.github.com.yaml | yq '.components.* | keys' | wc -l +90 + +% time ./swift-openapi-generator.filter.release \ + generate \ + --mode types \ + --config openapi-generator-config.yaml \ + issues.api.github.com.yaml +Writing data to file Types.swift... + +real 0m1.638s +user 0m1.595s +sys 0m0.031s + +% cat Types.swift | wc -l +14691 +``` + +### Proposed solution + +We propose a configuable, opt-in filtering feature, which would run before +generation, allowing users to select the paths and schemas they are interested +in. + +This would be driven by a new `filter` key in the config file used by the +generator. + +```yaml +# filter: +# paths: +# - ... +# tags: +# - ... +# operations: +# - ... +# schemas: +# - ... +``` + +For example, to filter the document for only paths that contain operations +tagged with `issues` (along with the components on which those paths depend), +users could add the following to their config file. + +```yaml +# openapi-generator-config.yaml +generate: +- types +- client + +filter: + tags: + - issues +``` + +When this config key is present, the OpenAPI document will be filtered, before +generation, to contain the paths and schemas requested, along with the +transitive closure of components on which they depend. + +This config key is optional; when it is not present, no filtering will take +place. + +The following filters will be supported: + +- `paths`: Includes the given paths, specified using the same keys as '#/paths' + in the OpenAPI document. +- `tags`: Includes the operations with any of the given tags. +- `operations`: Includes the operations with these explicit operation IDs. +- `schemas`: Includes any schemas, specifid using the same keys as + '#/components/schemas' in the OpenAPI document. + +When multiple filters are specified, their union will be considered for +inclusion. + +In all cases, the transitive closure of dependencies from the components +object will be included. + +[Appendix A](#appendix-a-examples) contains several examples on a real OpenAPI document. + +### Detailed design + +The config file is currently defined by an internal Codable struct, to which +a new, optional property has been added: + +```diff +--- a/Sources/swift-openapi-generator/UserConfig.swift ++++ b/Sources/swift-openapi-generator/UserConfig.swift +@@ -27,6 +27,9 @@ struct _UserConfig: Codable { + /// generated Swift file. + var additionalImports: [String]? + ++ /// Filter to apply to the OpenAPI document before generation. ++ var filter: DocumentFilter? ++ + /// A set of features to explicitly enable. + var featureFlags: FeatureFlags? + } +``` + +```swift +/// Rules used to filter an OpenAPI document. +struct DocumentFilter: Codable, Sendable { + + /// Operations with these tags will be included. + var tags: [String]? + + /// Operations with these IDs will be included. + var operations: [String]? + + /// These paths will be included in the filter. + var paths: [OpenAPI.Path]? + + /// These schemas will be included. + /// + /// These schemas are included in addition to the transitive closure of + /// schema dependencies of the included paths. + var schemas: [String]? +} +``` + +Note that these types are not being added to any Swift API; they are just used +to decode the `openapi-generator-config.yaml`. + +### API stability + +This change is purely API additive: + +- Additional, optional keys in the config file schema. + +### Future directions + +#### Providing a `fitler` CLI command + +Filtering the OpenAPI document has general utility beyond use within the +generator itself. In the future, we could consider adding a CLI for filtering. + +### Alternatives considered + +#### Not supporting including schema components + +While the primary audience for this feature is adopters generating clients, +there are use cases where adopters may wish to interact with serialized data +that makes use of OpenAPI types. Indeed, OpenAPI is sometimes used as +a language-agnostic means of defining types outside of the context of a HTTP +service. + +#### Supporting including other parts of the components object + +While we chose to include schemas, for the reason highlighted above, we chose +_not_ to allow including other parts of the components object (e.g. +`parameters`, `requestBodies`, etc.). + +That's because, unlike schemas, which have standalone utility, all other +components are only useful in conjuction with an API operation. + +--- + +### Appendix A: Examples + +#### Input OpenAPI document + +```yaml +# unfiltered OpenAPI document +openapi: 3.1.0 +info: + title: ExampleService + version: 1.0.0 +tags: +- name: t +paths: + /things/a: + get: + operationId: getA + tags: + - t + responses: + 200: + $ref: '#/components/responses/A' + delete: + operationId: deleteA + responses: + 200: + $ref: '#/components/responses/Empty' + /things/b: + get: + operationId: getB + responses: + 200: + $ref: '#/components/responses/B' +components: + schemas: + A: + type: string + B: + $ref: '#/components/schemas/A' + responses: + A: + description: success + content: + application/json: + schema: + $ref: '#/components/schemas/A' + B: + description: success + content: + application/json: + schema: + $ref: '#/components/schemas/B' + Empty: + description: success +``` + +#### Including paths by key + +```yaml +# openapi-generator-config.yaml +filter: + paths: + - /things/b +``` + +
+Click to expand filtered document + +```yaml +# filtered OpenAPI document +openapi: 3.1.0 +info: + title: ExampleService + version: 1.0.0 +tags: +- name: t +paths: + /things/b: + get: + operationId: getB + responses: + 200: + $ref: '#/components/responses/B' +components: + schemas: + A: + type: string + B: + $ref: '#/components/schemas/A' + responses: + B: + description: success + content: + application/json: + schema: + $ref: '#/components/schemas/B' +``` +
+ +#### Including operations by tag + +```yaml +# openapi-generator-config.yaml +filter: + tags: + - t +``` + +
+Click to expand filtered document + +```yaml +# filtered OpenAPI document +openapi: 3.1.0 +info: + title: ExampleService + version: 1.0.0 +tags: +- name: t +paths: + /things/a: + get: + tags: + - t + operationId: getA + responses: + 200: + $ref: '#/components/responses/A' +components: + schemas: + A: + type: string + responses: + A: + description: success + content: + application/json: + schema: + $ref: '#/components/schemas/A' +``` +
+ +#### Including schemas by key + +```yaml +# openapi-generator-config.yaml +filter: + schemas: + - B +``` + +
+Click to expand filtered document + +```yaml +# filtered OpenAPI document +openapi: 3.1.0 +info: + title: ExampleService + version: 1.0.0 +tags: +- name: t +components: + schemas: + A: + type: string + B: + $ref: '#/components/schemas/A' +``` +
+ +#### Including operations by ID + +```yaml +# openapi-generator-config.yaml +filter: + operations: + - deleteA +``` + +
+Click to expand filtered document + +```yaml +# filtered OpenAPI document +openapi: 3.1.0 +info: + title: ExampleService + version: 1.0.0 +tags: +- name: t +paths: + /things/a: + delete: + operationId: deleteA + responses: + 200: + $ref: '#/components/responses/Empty' +components: + responses: + Empty: + description: success +``` +
+ +--- + +[^1]: Compiled in release mode, running on Apple M1 Max.