How can we handle schema annotations in DPDS?

[andrea-gioia]

Let's brainstorm here on how to handle schema annotations in DPDS.
The ideas collected in this thread could become the foundation for a new RFC proposal.

What is a schema annotations anyway?
A dataset schema is the definition of how data is organized within a dataset. Schema definition can be used to validate shared data structure (schema validation), annotate data structure with metadata (schema annotation), or create the data structure on a specific datastore (schema creation or data definition language).

In DPDS dataset schemas are used to describe the data shared by ports. The schema definition is not part of the specification. It is delegated to the specification used to describe the API of the specific port. The schema definition format depends on the format used by the API specification to serialize exchanged data. The following tables show some commonly used schema definition format

Schema definition format	Data serialization format
JSON Schema	JSON
YAML Schema	YAML
AVRO Schema	AVRO
XML Schema Definition	XML

Why it matters?
Through schema annotation is possible to provide metadata to describe the underlying data beyond the way it is structured. Examples of information that can be encoded through schema annotations are:

• human-readable descriptions of the dataset and its field
• data usage guidelines
• technical information on how data is stored
• constraints on data exchanged that can be then converted into quality checks
• how the dataset and its fields can be linked to external datasets (syntactic linking)
• how the dataset and its fields can be linked to external ontologies (semantic linking)

While each schema definition format has different modalities to define annotations it's important to define a common vocabulary for the information that can be used to enrich the schema definition. In other words, it's important to define the admissible annotations and their meaning.

Describing the schema of a dataset using standard schema definition format and annotations is better than defining a custom structure to store schema metadata for the following reasons:

• it is possible to have only one schema for serialization, validation, and metadata
• metadata are embedded in the schema shared between producer and consumers. Once the consumer gets the schema gets also its metadata no matter what tool manages the schema repository
• using standard schema format permits to leverage the ecosystem of tools supporting the format (es. validator, linter, registry, fiff, etc...)

Some useful resources

1. schemata
2. proposed annotations

[andrea-gioia]

Annotations examples

The following examples define in all relevant formats a simplified schema for a dataset storing data about Persons. To all the schema the examples show how to add the following annotations

properrty	type	applicability	description
displayName	string	entities, fields	The human readable name of the item. It SHOULD be used by the frontend tool to visualize the data item's name in place of the name property.
owner	string	entities	The owner of the entity schema. If the schema is not shared it is the owner of the data product that defines the schema
encoding	string	fields	The name of the encoding scheme used by the field (ex. UTF-8)

JSON Schema

Annotations management

In JSON Schema, annotations are special keywords that provide additional information about the schema without affecting its validation process. Any keyword not reserved for validation can be used as an annotation. Annotation keywords can also be formalized through vocabularies.

• JSON Schema Vocabularies

Example

{
  "$schema": "http://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["givenName", "familyName"],

   "displayName": "Customer",
   "owner": "Andrea Gioia", 

  "properties": {
    "givenName": {
      "type": "string",

      "displayName": "First Name"
   },
   "familyName": {
      "type": "string",

      "displayName": "Last Name"
   }
   "age": {
      "type": "integer",
      "minimum": 0
   },
   "email": {
      "type": "string",
      "format": "email",

      "encoding": "UTF-8"
   }
 }
}

Validate JSON schema here

YAML Schema

Annotations management

todo

Example

$schema: http://json-schema.org/draft/2020-12/schema
type: object
required:
  givenName
  familyName
#annotations
displayName: Customer
owner: Andrea Gioia

properties:
  givenName:
    type: string
    #annotations
    displayName: First Name
  familyName:
    type: string
    #annotations
    displayName: Last Name
  age:
    type: integer
    minimum: 0
  email:
    type: string
    format: email
    #annotations
    encoding: UTF-8

Validate YAML schema here

AVRO Schema

Annotations management

In AVRO all attributes not defined in the specification document are permitted as metadata, but must not affect the format of serialized data.

Example

{
  "type": "record",
  "name": "Person", 
  "displayName": "Customer",
  "owner": "Andrea Gioia",
  "fields": [
    {
      "name": "givenName",
      "type": "string",
      "displayName": "First Name"
    },
    {
      "name": "familyName",
      "type": "string",
      "displayName": "Last Name"
    },
    {
      "name": "age",
      "type": "int" 
    },
    {
      "name": "email",
      "type": "string",
      "encoding": "UTF-8"
    }
  ]
}

PROTOBUF Schema

Annotations management

Protobuf offers a limited form of annotations through options. You can define options as key-value pairs attached to messages, fields, enums, and services. These options are not processed by the protobuf compiler itself, but you can use them for custom tooling or code generation.

Example

syntax = "proto3";

message Person {
   // annotations
   option(dispalyName) = "Customer";
   option(entity).owner = "Andrea Gioia";

  string given_name = 1;
  [(dispalyName)  = " First Name"]; // annotations

  string family_name = 2;
  [(dispalyName)  = " First Name"]; // annotations

  int32 age = 3;

  string email = 4;
  option(field).encoding = "UTF-8"; // annotations
}

XSD Schema

Annotations management

In XML Schema (XSD), <xs:annotation> is a special element that allows you to add extra information to your schema. This information is not part of the actual data itself, but rather serves as documentation for humans or other processing tools.

Here's a breakdown of how it works:

• Purpose: Add comments and extra info to your XSD schema.
• Content: It can contain two main types of elements:
  • <xs:appinfo>: This is for application-specific data that processing tools might use.
  • <xs:documentation>: This is for human-readable comments that explain the schema or specific parts of it.
• Placement: You can use xs:annotation with most elements within your XSD schema, providing additional details about that particular element.

Overall, <xs:annotation> is a way to enhance your XSD schema by making it clearer and more informative.

Example

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://your.schema.namespace" elementFormDefault="qualified" attributeFormDefault="unqualified">

  <xs:element name="Person">
     <xs:annotation>
             <xs:appinfo>
                   displayName: Customer
                   owner: Andrea Gioia
             </xs:appinfo>
     </xs:annotation>
      
    <xs:complexType>
      <xs:sequence>
        <xs:element name="givenName" type="xs:string" minOccurs="1">
            <xs:annotation>
                <xs:appinfo>
                      displayName: First Name
                </xs:appinfo>
           </xs:annotation>
         </xs:element>

        <xs:element name="familyName" type="xs:string" minOccurs="1">
            <xs:annotation>
                <xs:appinfo>
                      displayName: First Name
                </xs:appinfo>
           </xs:annotation>
        </xs:element>

        <xs:element name="age" type="xs:int" minOccurs="0">
          <xs:restriction base="xs:int">
            <xs:minInclusive value="0"/>
          </xs:restriction>
        </xs:element>
        <xs:element name="email" type="xs:string" minOccurs="0">
          <xs:restriction base="xs:string">
            <xs:pattern value="^[^@\s]+@[^@\s]+\.[^@\s]+$"/>
          </xs:restriction>
        <xs:annotation>
                <xs:appinfo>
                      encoding: UTF-8
                </xs:appinfo>
           </xs:annotation>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

</xs:schema>

[andrea-gioia]

Ok, we have collected enough ideas to open a new RFC. Let's create an issue.