Support GraphQL (AppSync)

[sam-goodwin]

GraphQL has a really nice way of modeling APIs and Types and how they're fulfilled by many different backend calls (via adapters). This creates an interesting opportunity to leverage Punchcard's abstraction of Build and Run time to build generic and re-usable APIs on top of diverse and complex AWS resources.

[sam-goodwin]

Interesting code-first approach for GraphQL: https://github.com/MichalLytek/type-graphql

[sam-goodwin]

Woah! I really like the look of type-graphql. Nice usage of decorators - something we haven't used in Punchcard yet. This might be a better way to define things like Shapes!

I'm gonna see if Punchcard can integrate with its dependency injection framework. It would be amazing to have an end-to-end of a type-graphql app backed by the CDK.

Then, if we can figure out how to do UI (integrate with React), then we'd have a full stack.

[sam-goodwin]

Also: https://oss.prisma.io/graphqlgen/

[pocesar]

decorators plus the generation of metadata from them are really powerful ❤️ and also fits perfectly for the shapes in general

food for thought: create another example of how the full react stack should look like (like using Cloudfront, S3, and even Route53 with an "graphql" endpoint). this one will surely attract a lot of people to know more about how great is punchcard and CDK in general

[sam-goodwin]

Just not quite sure how to generate the DSLs for things like DynamoDB if we move to annotated classes.

E.g.

const MyShape = struct({
  key: string({maxLength: 1})
});

// vs

class MyShape {
  @string({maxLength: 1})
  public key: string;
}

If we use MyShape in a Table:

const table = new DynamoDB.Table(..,.., {
  attributes: MyShape,
  partitionKey: 'key'
});

// and then the generated DSL:

await table.put({
  item: ...,
  if: item => item.key.equals('key')
});

Perhaps a JS proxy could help us here?

We'd have to change how the type-mapping works. Shapes currently encode the "runtime shape" in the type: e.g. class StringShape extends Shape<string>. I.e. the runtime type is annotated on the build-time value. A decorated class is the other way around: you define the runtime type and then use decorators to assign build-time details. It's also more idiomatic, so I like that. But is it as capable?

[sam-goodwin]

Here's a succinct way to integrate Shapes with Classes:

const dataShape = Symbol();

class MyShape implements RuntimeShape<MyShape[typeof dataShape]> {
  public readonly [dataShape] = struct({
    key: string()
  });

  public key: string;
}

[sam-goodwin]

Simplifying this further - remove the struct concept and use a class instead:

class MyShape {
  @Annotation
  public readonly key = string();
}

Only caveat is that a runtime value is not an instance of this class, it is an instance of RuntimeShape<MyShape>.

[sam-goodwin]

This seems really clean.

export interface ClassType<T = any> {
  new (...args: any[]): T;
}

type Value<T> = {
  [prop in keyof T]:
    T[prop] extends Shape ? RuntimeShape<T[prop]> :
    T[prop] extends ClassType<infer V> ? Value<V> :
    never;
};

class OtherType {
  key = string();
}

class Rate {
  rating = integer();
  key = string();
  otherType = OtherType;
  nested = Rate;
}

const rate = new Rate();
const rateValue: Value<Rate>;
rateValue.key; // string
rateValue.otherType; // Value<OtherType>
rateValue.nested.key; // string

Even supports recursive types by the looks of it.

[pocesar]

I was checking the struct-overhaul branch, it reminded me of how kotlin does json (de)serialization, the readability is really nice! even though the code itself is over my head, the generated metadata is really interesting and the use o Proxy while providing type-safety recursively without having to mutate the input 🥇

[sam-goodwin]

Do you have a link to an example of how Kotlin does it?

It reminds me of Django/Google App Engine for Python, where class members are values that represent a type.

https://docs.djangoproject.com/en/3.0/topics/db/models/

from django.db import models

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

Except I de-coupled the constraints (such as max_length) from the type itself with a new concept, Trait. Traits are a lot like Decorators, except they augment the type of what they are decorating. Ordinary decorators are also supported, but they don't maintain type information.

An example of this in action:

class MyType {
  /**
   * Field documentation.
   */
  id = string
    .apply(MaxLength(1))
    .apply(MinLength(0))
    .apply(Pattern('.*'));

  // can be thought of like
  // @MaxLength(1)
  // @MinLength(0)
  // @Pattern('.*)
  // id = string
}

// the type of the derived schema retains all information, even the literal values
// such as maxLength: 1, instead of maxLength: number
// note: the type is inferred, I've explicitly included it here to illustrate
const schema: ObjectSchema<{
  id: StringSchema<{
    maxLength: 1;
    minLength: 0;
    pattern: ".*"; 
  }>;
}> = JsonSchema.of(MyType);

This issue is related: microsoft/TypeScript#7169 - support for the reification of generic types by the TS compiler for use with JS reflection.

My plan for the struct-overhaul change is to provide a modular/re-usable framework for defining types and then deriving type-safe Abstract Syntax Trees (ASTs) from them. E.g. JSON Schema, JSON Path, the DDB DSL already built in Punchcard, etc. This one type-system will support any data format - i'm envisioning Avro, Parquet, Thrift, Smithy and even an ORM for type-safe SQL DSLs.

The framework in its current state already supports all the features I need. I'm just in the middle of porting the existing work in punchcard to separate @punchcard/shape-* libraries. I'm super excited to have these libraries as stand-alone - the dynamodb library, for example, will be usable without punchcard, which should (in my opinion) provide a better experience than https://github.com/awslabs/dynamodb-data-mapper-js

[sam-goodwin]

Some cool (potentially useful) type-machinery is the use of this metadata to build type-safe predicates:

class MyType {
  count = number
    .apply(MultipleOf(2))
}

function requireEven(schema: NumberSchema<{multipleOf: 2}>) {
  // this function requires the schema to represent even numbers
}

const schema = JsonSchema.of(MyType);

// compiles
requireEven(schema.properties.count);

Compilation breaks if I now comment out the .apply(MultipleOf(2)).

I wonder if we could do operations like <, >, etc. It would likely require machinery similar to Shapeless's Nat, a type-level representation of natural numbers: https://github.com/milessabin/shapeless/blob/master/core/src/main/scala/shapeless/nat.scala

I found this, https://github.com/thomaseding/nat-ts, not sure how useful it is though.

[sam-goodwin]

This has turned out so much better than I'd expected. It's a generic, type-safe AST transformer. Shapes + Traits modeled in the TS compile-time type-system mean I can transform a Shape declaration into any other format (AST) without loss of type information metadata. In fact, that metadata information can even be used to influence the result.

This is effectively a generic code generator that doesn't generate code - it generates types. Look how easy it is to generate code:

class MyType {
  /**
   * Field documentation.
   */
  id = string
    .apply(MaxLength(1))
    .apply(MinLength(0))
    .apply(Pattern('.*'))
    ;

  count = number
    .apply(Maximum(1))
    .apply(Minimum(1, true))
    .apply(MultipleOf(2))
    ;

  nested = Nested;
  array = array(string);
  complexArray = array(Nested);
  set = set(string);
  complexSet = set(Nested);
  map = map(string);
  complexMap = map(Nested);
}

// generate an interface representing this object at runtime
interface MyRuntimeRepresentation extends Runtime.OfType<MyType> {}

// generate an interface representing this object's JSON schema reprsentation
interface MyJsonSchemaRepresentation extends JsonSchema.OfType<MyType> {}

Implementing a AST transformer is relatively simple too. This is all that's required to map a Shape to its JavaScript runtime value:
https://github.com/punchcard/punchcard/blob/f2f01e04667cede0746971efc412fcf902311b45/packages/%40punchcard/shape-runtime/lib/runtime.ts#L1-L49

[pocesar]

well, I'm 5 minutes'ish late, but here's how kotlin + moshi does it https://proandroiddev.com/getting-started-using-moshi-for-json-parsing-with-kotlin-5a460bf3935a
android Room is similar, but using SQLite but needs a lot of boilerplate to generate the relations (for obvious reasons), but they all use decorators https://android.jlelse.eu/android-room-using-kotlin-f6cc0a05bf23. dynamodb-data-mapper-js suffer from the same boilerplate of Room

EDIT: gotta say, never knew you could reference "this" directly inside a interface like that plus access it like an object haha. while it DOES seem simple, after it's ready, I'm not being able to understand everything, I plan to study punchcard source thoroughly to be able to help, at least the thought process

EDIT2: after checking some of the asserts, TS 3.7 have assert types now, might come in handy https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#assertion-functions it's the cousin of "arg is something", it's like a boolean on steroids

function isString(val: any): val is string {
    return typeof val === "string";
}

function yell(str: any) {
    if (isString(str)) {
        return str.toUppercase();
    }
    throw "Oops!";
}

[sam-goodwin]

Came up with another approach with a key benefit: the class MyClass is the runtime type instead of requiring the mapping Value.Of<typeof MyClass>. Is this more idiomatic/familiar?

Instead of just having a class, we can extend a call to a function that sets up static values for the class and hides the shape-values in the class under a symbol, all while remaining type-safe.

const class = Symbol.for('class');

// psuedo code
export function data<T extends {[prop: string]: Shape;}>(shape: T): ClassType<{
  // class contains a member for each of the values - awesome!
  [K in keyof T]: Value.Of<T[K]>;
} & {
  // put the shape on the class instance, like getClass() in Java.
  [class]: Shape.Of<T>; 
}> & {
  // static reference to the class (shape data)
  readonly class: Shape.Of<T>;
}{
  // dynamically construct a class
  class TheClass {}
  TheClass.class = Shape.of(shape); // static class
  TheClass.prototype.class = TheClass.class; // reference to class on the instance
  return TheClass as any;
}

Usage:

class MyData extends data({ // we extend the result of the data function call, which is a dynamically generated class
  key: string
    .apply(Decorator) // traits can be dynamically applied like this, unlike TS decorators, because we use ordinary code that isn't restricted to only running on module-load
}) {
  // i can implement methods on the class now!
  public getKey(): string {
    return this.key; // compiles
  }
}

MyData.class; // contains the static information about this class and its members, like in Java. We've basically built a static type-system for TS.

// MyData is the representation at runtime, instead of the ugly Value.Of<typeof MyData>
const value = new MyData({
  key: 'this is a key' // type-safe constructor
}); 
value.key; // of type: string
value[class]; // this instance's class.

Only caveat is that you can't extend a "data class" to add new properties.

// won't work, bummer.
class MyExtendedData extends MyData {
  key = string; // won't be used
}

This would align with Smithy's "no inheritance" tenet for DDLs. Maybe it's a good thing?

Thoughts?

[sam-goodwin]

If we went with this approach, reflection would be identical to Java:

const queue = new SQS.Queue(,, MyData.class); // pass the .class instance - just like Java

[pocesar]

oh, good idea, it's also how kotlin does java interop as well by default, imo it's definitely a way forward, might be weird for JS/TS only coders though.
I don't like inheritance much myself, but using "extends" as a "trait" is a nice approach, plus that would make passing around your dependencies much easier, and you can always rely on .class.

side note: won't this clash with #100?

[sam-goodwin]

Bundling (#100) refers to the running of webpack to create a minified JS file for running inside Lambda. It shouldn't clash with this change - the new Shape system is ordinary JS/TS.