-
-
Notifications
You must be signed in to change notification settings - Fork 288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Address perspective and channel reuse issues through introducing 'endpoint' concept #599
Comments
Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request. Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue. |
Love how much effort you've put into this proposal 👏 ❤️ I have a few comments/questions:
I wonder how you would have modeled this proposal if you remove the backward-compatibility constraint. I challenge you to reimagine it 😄 |
@fmvilas awesome feedback, thank you. On point 6, I see what you mean. I'm not as good at JSON as I should be. Could we do something similar to OpenAPI spec:
|
We're already doing that at Schema Object level (same as in OpenAPI). The Schema Object is a pure JSON Schema object, not something OpenAPI or AsyncAPI defines. |
That was a long read! well done. |
@jessemenning I assume you want to champion this through? 🙂 Or can we consider this issue as needs champion? 🤔 |
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
@jessemenning do you want to drive it further? |
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
2021.08.09 - Tried to create a non recursive definition of endpoint.
Summary
Issue
The AsyncAPI community acknowledges that there are significant issues related to the understandability of the spec for new users, the flexibility to address various async topologies and the reuse of channels across multiple applications.
Proposed Resolution
To address these issues, the proposal allows:
Result
The proposal gives users the ability within AsyncAPI to describe how:
Technical Summary
The proposal introduces a top-level
endpoint
entity to AsyncAPI, and relaxes restrictions on mandatory AsyncAPI properties and entities. In order to ensure consistent tooling behavior in the face of this new flexibility, it also introduces several business rules.General Discussion
In v2.1 an AsyncAPI spec "MUST describe the operations an application accepts." Stated another way, an AsyncAPI v2.1 spec describes how an endpoint may interact with a single application. When used for code generation, the single application is assumed to implement the inverse of what is described in the spec. In other words, if interacting endpoints can subscribe from named application, the spec assumes that named application publishes those messages.
For many use cases this is sufficient. This tends to true with users that are familiar with OpenAPI and implementing asynchronous client-server interaction.
However, as noted in the perspective issue, the mental gymnastics of "inverse implementation" remain a challenge for both new and experienced users, particularly in the context of code generation.
Additionally, AsyncAPI struggles to capture some async topologies. For instance, broker-mediated topologies are not exclusively (or even predominantly) one-to-one interaction. In fact, async architectures that use event brokers are typically a network of decoupled nodes interacting in complex ways. Here the assumption of inverse relationships fails, as components can:
Discussion of Specific Proposed Changes
Address additional use cases and issues by allowing explicit
Endpoint
s☐ Introduce a top-level
endpoint
entity to AsyncAPI.An endpoint is entity that sends, receives, or sends and receives messages via a channel. In AsyncAPI, an
endpoint
object represents either the implementation of a single endpoint, or a defines the capabilities of a generic class of endpoint.In AsyncAPI an endpoint explicitly defines, from the perspective of the endpoint:
endpoint
implement?The resulting AsyncAPI structure looks like:
The introduction of
endpoint
and the resulting loose coupling addresses both the perspective and the channel reuse issue. Because endpoints are defined from their own perspective, using intuitive verbs, the new user experience improves and code generation becomes less problematic. And by decoupling endpoints from channels by using references, channels can be reused by multiple endpoints.Further Abstract
Channel
by genericizing verbsThe existing verbs in v2.1 (
publish
andsubscribe
) are linked to the pub-sub model of async interaction. Queues, mailboxes, websocket endpoints all can serve as asynchronous channels, but do not require either publishing or subscribing.This proposal
☐ Deprecates the
subscribe
verb, replaces withendpointSends
.☐ Deprecates the
publish
verb, replaces withendpointReceives
(During a transition period, tooling which reads AsyncAPI specs should continue to accept
subscribe
andpublish
. Tooling which writes AsyncAPI specs should be updated to useendpointSends
andendpointReceives
.)This change further abstracts the base
Channel
concept away from any implementation choice. The implementation details can in turn be specified as bindings, either attached to the channel or the endpoint-operation, depending if they apply to the channel, or to an individual endpoint’s use of the channel.On a practical level, further abstraction of Channel allows point-to-point asynchronous communication, either brokered or unbrokered, to be represented by Channels without bending the common definitions of "publish" and "subscribe". This in turn increases consistency across a wider range of interaction styles.
Emphasize multiple physical realizations of underlying logical model
In examples to follow, the endpoint, channel and message entities are frequently presented as separate files. However, this should not be taken as a requirement. In fact, the increased use of references allows for a wide range of implementation options including:
To facilitate this flexibility, the proposal
☐ Makes optional all top level entities (
info, servers, channels, endpoints, components
) .The spec is unopinionated on what the best way to organize, govern, persist or access AsyncAPI. In fact, a single organization may use different physical AsyncAPI representations of the same underlying set of logical async interactions. And, depending on use case, an organization may choose to present only a piece of the overall architecture to an endpoint for reasons of security, simplicity, etc.
Differentiate Interface and Implementation when using endpoints
Currently AsyncAPI defines how a component MAY interact with the application. This reflects the intention of the document to serve as a true API, closely paralleling OpenAPI. For consistency and backwards compatibility, this will continue to be the assumption in the absence of explicitly defined endpoints.
That said, a separate and legitimate use of AsyncAPI is generating and documenting a particular endpoint’s implementation. OpenAPI (and REST) is tightly coupled to the HTTP/S protocol, so that verbs, URL, etc. can be assumed to be implemented using HTTP/S. In contrast, the asynchronous world uses a wide variety of protocols. Evidence of this is seen by provision of the specific binding information present in many AsyncAPI files and the growing number of code generation options. The need to explicitly provide implementation information in the asynchronous world forces a more explicit spec that allows for implementation details to be recorded in a systematic way.
To accommodate this reality, this proposal:
☐ Adds an optional field called
intent
toendpoint
with allowed valuesinterface
andimplementation
☐ Adds an optional field called
required
tooperations
, with allowed valuestrue
andfalse
intent
indicates whether the purpose of this endpoint definition is aninterface
(defines a of class of endpoint) or theimplementation
of a particular endpoint.For an
interface
endpoint, operations marked asrequired:true
must be present in implementations of that interface. For example, if there is a an interface called StreetLight with a required operation, then an implementation of that interface (StreetLightAt123MainStreet), must implement that operation.☐ Adds the following business rules:
Clarify Inheritance rules when using references
To achieve decoupling and reuse, the proposed spec relies heavily on references from one entity to another. This (and the introduction of separate implementation and interface concepts) forces the spec to formalize the requirements with regards to inheritance when referencing another entity.
☐ Adds the following business rules:
In the case where Entity A references another entity B (e.g. another endpoint, channel, message), Entity A MUST inherit all the properties of Entity B (and likewise all the Entities that have been previously referenced). Entity A MAY modify a property of Entity B (and previously referenced entities) IF AND ONLY IF ONE OR MORE OF THESE CONDITIONS ARE TRUE:
Summary of Proposed Changes
All changes are non-breaking.
☐ Deprecate
publish
in favor ofendpointSends
☐ Deprecate
subscribe
in favor ofendpointReceives
☐ Introduce
endpoints
entity to allow explicit definition of endpoints, including whether the endpoint represents an interface or an implementation☐ Allow for messages to be associated directly with channels
☐ Make all top level entities optional
☐ Make verbs within the channel definition optional.
These changes evolve AsyncAPI towards more robust enterprise functionality and resolve lingering issues that could prevent wider adoption.
Specific Examples
WORK IN PROGRESS
To establish a common frame of reference, the following examples use the Streetlights tutorial from AsyncAPI.com as a baseline. As described in the tutorial, the company is called "Smarty Lighting and you do smart-city lighting systems". To better explain the proposal's implications, each example adds additional complexity to the baseline tutorial.
As-is Streetlights with AsyncAPI v2 Implicit endpoint
Demonstrates: Backwards compatibility with existing AsyncAPI specs
Adds to Streetlights: Nothing
The baseline Streetlights tutorial describes the API of a Message Broker.
The AsyncAPI v2.1 representation of the Streetlights is:
Streetlight API
As there are no explicit endpoints, it is assumed the API describes:
So, the implicit clients may choose to send LightMeasured events to the event broker.
It's important to note this would be valid AsyncAPI file in both v2 and v3. Tooling would continue to read the "publish" and "subscribe" verbs, although it would be expected that new versions of tooling would write "endpointSends" and "endpointReceives"
Here we can see the two issues:
Perspective issue: AsyncAPI code generators create an application that subscribes to the events. This has proven to be difficult to explain to both new and existing users.
Channel reuse: the light/measured channel definition is tightly coupled to the application and to the pub-sub architecture. If another application wishes to use light/measured, it would need to be included in its own AsyncAPI definition.
Explicit endpoints (AsyncAPI v3 style)
Demonstrates: More explicit definitions of actors could be beneficial in some situations
Adds to Streetlights: Conversion to AsyncAPI v3
In proposed v3 it becomes possible--though not mandatory--to explicitly create endpoints and/or classes of endpoints. The same Streetlights example becomes:
endpoint_streetlight.yaml
channels.yaml
As there are explicit endpoints, it is assumed the API describes:
The potential benefits of using the v3 approach?
Channel reuse
Demonstrates: Increased ability to reuse channels
Adds to Streetlights: Additional endpoints and channels
In proposed v3 it becomes possible--though not mandatory--to explicitly create endpoints and/or classes of endpoints. As endpoints interact in increasingly complex ways, decoupling the endpoints from channels increases reuse.
To demonstrate, this example adds a Control Hub endpoint, which receives light measurements from the streetlight. In turn, it sends out commands which can turn the streetlight on. Control Hub and Streetlight both reuse the same two channels.
endpoint_streetlight.yaml
endpoint_controlhub.yaml
channels.yaml
The potential benefits of using the v3 approach?
Use of Interface/Implementation and required tags
Demonstrates: Additional possible interpretations of AsyncAPI
Adds to Streetlights: Additional streetlights, including some of which are external to the organization and describe implementations.
As event APIs become more sophisticated, there is an increased need for flexibility of inpretations and enterprise functionality.
In this example, the Streetlight API is provided to outside organizations. Streetlight owners can graciously crowd source their light measurements by sending them to Control Hub. However, external organizations do not need to receive Control Hub lightOn events.
To emphasize this, the
intent
ofinterface_streetlight.yaml
is set tointerface
. And while the light/measured operation is marked asrequired:true
, the control/lightOn operation remains optional (the default value).In contrast the implementation (
endpoint_streetlight_833Sparrow.yaml
) is set tointent:implementation
.endpoint_streetlight_833Sparrow.yaml
interface_streetlight.yaml
endpoint_controlhub.yaml
channels.yaml
Here, the third possible intpretation of AsyncAPI is shown:
In addition, enterprises can advertise their evented API products while clearly showing the functionality that the external API can and must provide.
Use of inheritance to specify which schema an endpoint uses amongst many in a channel
Demonstrates: Inheritance and override logic
Adds to Streetlights: Multiple possible messages within in a channel
As applications evolve, they will introduce breaking changes in the messages that they send. These breaking changes can cause failures for downstream applications. This is particularly true in brokers such as Kafka, where topics persist messages indefinitely, so that there may be multiple version of a message (or multiple message types) on the topic.
The decoupling of endpoints from channels allows implementers multiple ways to address this issue.
Wild West
Channel does not specify associated messages. Instead, each endpoint specifies which message it sends. Implies a lack of central governance, which means that applications receiving messages must be ready to adjust to breaking changes.
endpoint_controlhub.yaml
channels.yaml
Channel centric
Each channel specifies a single associated message, or a list of associated messages using
oneOf
. In turn, each endpoint specifies which message it sends and/or receives. If a channel contains a list of associated messsages, endpoints must select an option from that list . This topology likely implies some sort of central governance or tooling of shared channels, as applications must modify both their own AsyncAPI spec and that of the shared channel(s).endpoint_controlhub.yaml
channels.yaml
Defer to schema registry
Lastly, enterprises could defer schema management to a schema registry. Rather than list specific versions, references point to a schema registry, which can be used at either code-time, compile-time or run-time.
endpoint_controlhub.yaml
channels.yaml
Use of Interfaces for external API products
Forthcoming
Specifying Binding details.
Under this proposal, channels become much more logical. Channels know only the following
Optionally, and to align with the schema registry use case, AsyncAPI allows channels to know
Making channels logical objects defers implementation details to protocol-specific bindings. This compartmentalizes implementation details, which has several nice properties:
Symmetrical bindings- Input binding same as output binding
The simplest case is one in which the input binding will always be the same as the output binding. This is the case in Kafka. Endpoints sending data use the same topic as endpoints receiving data. In this case, existing bindings should work and should be at the channel level.
Asymmetrical bindings- Input binding different than output binding
This needs more work, and I would love input from @nictownsend, @dalelane, @jhigginbotham and @damaru-inc on this.
On the other hand, several async protocols can have asymmetric channels, where input and output bindings are are not the same (e.g. AMQP topic to queue, IBM MQ queue to topic, Solace SMF topic to queue). To accommodate these use cases, the spec needs to have the ability to specify different channel input and output bindings.
Ideally, to keep the channel as logical as possible, the channel object itself would not know if it's symmetrical or asymmetrical. That implementation detail would be left to the binding.
The question is where asymmetric bindings belong. Thinking about things, there seem to be multiple possibilities:
Async Request/Reply (Websockets, NATS)
In general, these can be described as two different channels. Modifying the @derberg example:
References to servers from bindings
Forthcoming
The text was updated successfully, but these errors were encountered: