linkml-arrays chat

[sneakers-the-rat]

Wanted to make a general discussion thread for linkml-arrays. as a meta-thread, let's try and use the in-post replies as much as possible to group conversations <3

plz edit this root post/propose edits to the root post as relevant.

Status

NDArrays

• metamodel syntax declared: First pass at native NDArray support. linkml-model#181
• generators
  • pydantic: list of lists - [pydanticgen] Add list of lists array implementation #1887
  • pydantic: nptyping - (@sneakers-the-rat will do)
  • json schema: Add support for linkml:array in jsonschemagen #1889
  • RDF/Shacl/ShEx: decide on strategy for serializing NDArrays as RDF and validating using shacl/shex #1890

Labeled Arrays

• metamodel syntax declared: (still WIP i think?? linkml-arrays has the current draft
• generators
  • pydantic: list of lists
  • pydantic: nptyping
  • ...

References

repo: https://github.com/linkml/linkml-arrays/

example data, particularly for the labeled arrays spec

.. (lmk what else)

[sneakers-the-rat]

@rly i had this thought and wanted to jot it down rl quick and realized we didn't have a public meta-discussion place and so made this. ok so part of what i have been trying to puzzle about with the labeled arrays spec is the sort of 'unseatedness' of the label axes - that it is good that they can be declared independently, but it still feels like it would be an implementation and authorship challenge to keep track of the labeling/index dimensions as basically parallel slots/attrs to the main array slot.

in trying to refactor the pydantic generator i got to thinking how nice it would be if we could just have a 1:1 map between metamodel properties and generator methods - if a slot has this prop, mutate the SlotDefinition in this way, and so on. the reason that's hard is because the relationship the props isn't/is hard to schematize - "range changes meaning when array is also defined."

I think that these problems can be seen as the same problem - It's difficult to schematize "horizontal" relationships between things at the same level. Since linkml is recursively defined, where the metamodel is in the syntax of the schema, could the same syntax be used to schematize relationships between props as eg. relationship of the indexing dimensions to the array?

just wanted to jot the thought down quickly without doing a full investigation of eg. existing ways it's possible to describe horizontal dependencies, but something as simple as a syntax like

array:
  modifies:
    - range

as a generic indication that "other things about this property range are subsumed by this property array, which is a superset of it." so in the context of a generator, you would skip the normal computation of the range and just use the array method

or if it's specified with a dict, the value specifies the way that the interaction happens

array:
  modifies:
    range: after

so "array" is to be handled "after" the usual range computation - do everything else you would do normally, and then this one. so in the case of a generator, you would compute the range as normal, and then pass it in to the array method.

and then the parameterized form (to complete the usual trirumvirate of array, Dict[str,str] and Dict[str,dict]

array:
  modifies:
    range:
      type: after
      param1: whatever

so eg. for generators, if a parameter had to specify it came in a specific order in a processing chain, etc.

To know when a keyword is applicable, it could use the usual metamodel way of defining itself

slots:
  after:
    is_a: linkml:modifier
    domain:
      - SlotExpression
      - SlotDefinition

and so then with the case of the labeled arrays you could have something like

MyClass:
  attributes:
    values:
      range: int
      array: 
      
    index1:
      range: TemperatureType
      modifies:
        values: indexes
    
    # or
    index2:
      range: LatitudeType
      modifies:
        values:
          type: indexes
          dimension: 0
          curie: # ...

[rly]

Thanks for the discussion @sneakers-the-rat !

re: naming - labeled_by et al.

keyed_by makes sense to me, though it is more of a technical term that makes more sense in relational databases. I know LinkML has key but one might prefer to use identifier which does the same thing (my impression is identifier is more common than key but IDK). So I have a slight preference toward labeled_by.

I believe your example is equivalent to mine. However, if there is a third dimension to temperature, e.g., time, how would that work? If the temperature array has shape (X, Y, T), then would the T time values have to be copied X * Y times, and the unraveled X * Y latitude values have to be copied T times?

---

re: naming - labeling_slot

I like labeling_slot.

---

syntax - array specs

NOT referencing another class that has an array slot is likely the primary use case and simplest example. However, I am thinking of cases in NWB and the Xarray tutorial example, where the same LatitudeInDegSeries array might be used in multiple classes, e.g., for both a temperature dataset and a precipitation dataset. Also in NWB, you may have multiple time series objects that use the same timestamps because they were recorded from the same acquisition system. I think that defining the axis as a separate array class, e.g., LatitudeInDegSeries or TimestampInSSeries, and referencing that object (as opposed to inlining the axis where it is used) makes this data reuse / linking possible. This use case is probably uncommon, but arrays can be expensive to store, so I think we need to consider ways to not duplicate the array data. This is more of a storage concern, but the schema constraints the storage in LinkML.

I agree that this is fragile and adds boilerplate when data is not being reused, but I don't know of a better way.

I suppose one way is to explicitly reference the latitude data in another object:

root:
  containers: 
  - name: temperature
    type: TemperatureDataset
    latitude_in_deg: [[0, 1, 2], [3, 4, 5]]
    temperatures_in_K: [[1, 1, 1], [2, 2, 2]]
  - name: precipitation
    type: PrecipitationDataset
    latitude_in_deg: ../temperature/latitude_in_deg
    precipitation_in_mm: [[3, 3, 3], [4, 4, 4]]

This is similar to how NWB handles linking. But I don't think this path-based or object-based referencing is supported by the LinkML language yet?

It might also help if LinkML could allow a slot to be either inlined or referenced, as opposed to requiring one or the other. See https://github.com/orgs/linkml/discussions/2185.

---

syntax - labeling dimensions as a list

Just to check, labeled_by is a (yaml) array because one could equivalently do this (assuming that all the values within a slice of the axis have the same value, rather than each cell being allowed to have its own latitude and longitude)

Yes

and then in the case that each cell of the array can have a totally independent label along all its dimensions then one would use the array mapping syntax above.

Yes

---

syntax - label mapping

We might want to have some syntax like this to indicate which dimension in the labeling array maps onto the labeled array, right?

Yes. It is required that each axis of the source (labeling) array is used (I cannot think of an example where that is not the case), and that the order of indices in labeled_dimensions is meaningful such that source array axis i is used to label target array axis at labeled_dimensions[i] .

I think the more explicit syntax is good, but the simplified syntax is so clean! Unless we think it is confusing, in which case, I suppose a few more lines in the schema to be clear is not so bad.

---

usage - requiredness

labeled_by only applies in the context of a class, right?

Yes.

So then if a labeled array slot was defined as a slot rather than an attribute, that would imply that the label slot would be required to be present on any class that uses it?

Yes. We should validate that.

[melonora]

Sorry coming to this discussion late.

re: naming - labeled_by et al.

I agree with both of you here with a preference for labeled_by with the options suggested so far. Main reason for that being that the concept of keys in relational databases is quite different from the concept of labeled dimensions in arrays. The concept here is more the concept of additional descriptors for each array cell besides the actual array values itself. Overall, in the array world I see more uptake of terminology related to labeled dimensions so that for people who would use it labeled_by might be more intuitive with meaning.
However, I would like to note that the concept of indexing by coordinates does exist also in xarray so perhaps label_indices is also an option? This would be similar to pandas Index as well which is used by xarray under the hood. Subsequently the slot could then be label_index_slot or labeling_index_slot for example. Perhaps this would also take some ambiguity away from how indexing is used in the context of relational databases.

[melonora]

Also regarding this

This use case is probably uncommon, but arrays can be expensive to store, so I think we need to consider ways to not duplicate the array data

This is perhaps not as uncommon as you might think particular when dealing with multimodal imaging (spatial transcriptimics, proteomics) as one requires to align the images so they live in one coordinate space. In this case it would be nice to share the labeled dimensions.

[oruebel]

here with a preference for labeled_by with the options suggested so far. Main reason for that being that the concept of keys in relational databases is quite different from the concept of labeled dimensions in arrays

Terms like indexed_by and keyed_by seem to imply that they are unique identifier (e.g., timestamps). In contrast, terms like labeled_by and annotations do not seem to imply uniqueness of labels (e.g., when annotating data with events in time that may repeat over time). With that in mind, I am also leaning towards labeled_by. However, we may want to consider having some additional key to identify whether the labels are a required to be unique (hence functioning as an index), e.g.,:

labeled_by:
        - label_slot: latitude
          labeled_dimensions: 0
          label_values_unique: 1

Or maybe that is a property that is better described as part of the labeling slot latitude itself, rather than being part of the labeled_by annotation.

[sneakers-the-rat]

agreed with y'all: let me rescind my muddying of the water and say let's not consider keyed_by :)

Re: uniqueness - I think this is already handled on the labeling slot side by list_elements_unique : https://linkml.io/linkml-model/latest/docs/list_elements_unique/ and that makes more sense to me as a property of the labeling slot than the labeled array - we dont want nonlocal assertions on a slot, which is one of the reasons we switched from labels to labeled_by above.

However, I would like to note that the concept of indexing by coordinates does exist also in xarray so perhaps label_indices is also an option?

I think this current draft is based on indexing/slicing using the labeling dimensions? But I also might not be getting what you're saying, maybe an example of what you mean here?

However, if there is a third dimension to temperature, e.g., time, how would that work?

I think that each of the labeled_by entries should be independent, ie. The interpretation of one is not influenced by presence or value of any other. Xarray allows multiple overlapping labels to be applied and that seems like a nice property. So we have to allow partial labeling sets - if only lat/long are labeled, then selecting a lat/long returns a vector of that position across time.

would the T time values have to be copied X * Y times, and the unraveled X * Y latitude values have to be copied T times?

I think that defining the axis as a separate array class, e.g., LatitudeInDegSeries or TimestampInSSeries, and referencing that object (as opposed to inlining the axis where it is used) makes this data reuse / linking possible.

As with not including details like column ordering in the regular array spec, we should try to avoid building details like how arrays are stored into the spec itself, and I agree this requires a referencing syntax.

Linkml spec does have this referencing syntax, though I am not sure of implementation: https://linkml.io/linkml-model/latest/docs/specification/02instances/#instance-accessor-syntax

I think this is largely a weakness of the current linkml yaml data model, which requires a single root collection type and all class identities are implicit within the dataset (I.e. you only know what class something is a dataset by specifying it in the loader call). IMO linkml datasets should look more like this, where referencing would be straightforward.

linkml_meta: # the only special name
  imports:
    - name: mySchema
      uri: example.com/the-schema-used-in-this-dataset

# each top-level thing is identified by its key, and then it declares its class with `is_a`
EarthPositions:
  is_a: mySchema.Positions
  latitude: #. ..
  longitude:  #...

EarthTemperatures:
  is_a: mySchema.Temperatures
  latitude: EarthPositions.latitude
  temps: #...

Also important to keep in mind the distinction between the schema and data levels, where all the schema needs to do is indicate the type of slot that can serve as the labels, so references don't need to be resolved there.

[sneakers-the-rat]

First array representation just dropped, u can use it today!

Guide: https://linkml.io/linkml/schemas/arrays.html
Docs: https://linkml.io/linkml/generators/pydantic.html#arrays

this builds on the broader refresh of the pydantic generator and its templating system: https://linkml.io/linkml/generators/pydantic.html#templates

more 2 come <3

[rly]

On the topic of how to define an array slot within a DataArray class as an array class instead of as a slot, so that one can

1. reuse the class in multiple DataArray objects (e.g., you may want to store an array of a million timestamps once and use it to label multiple other arrays)
2. associate attributes / metadata to the array
3. extend the class in the schema

@cmungall and I thought about creating a simple, degenerate Array class that follows a particular pattern where exactly one attribute is an array. These classes would be identified using implements: linkml:Array (for now at least). I mocked that up in this example here:
https://github.com/linkml/linkml-model/pull/190/files#diff-84cb9e7e732cf37a327db45b4a6e1098fc0da6c6c423f6145c812918623abfb9

This raises an issue where it does not really make sense to put indexed_by on the TemperatureMatrix class because it does not know about / cannot access the indexes. So I put indexed_by on the temperatures_in_K attribute under the array key in the higher-level DataArray. This is kind of an odd usage of the array key. Perhaps it should be array_usage or something similar? (Here is a case where a modifies key might be useful.)

Open to thoughts and ideas!

[sneakers-the-rat]

since array currently is a SlotDefinition property, could we just also extend its domain to include ClassDefinition ?

classes:
  MyClass:
    array: # ...

[rly]

In that case, where would we specify other SlotDefinition properties like:

range: float
unit:
  ucum_code: deg
minimum_value: 0

[sneakers-the-rat]

Hmm hmm hmm.

Yes this is sort of what I was thinking about with that "can the implements keyword be a sort of "level shifter" half-idea. So eg. Say all the terms that modify an array are in some subset "array". then by doing implements: array what we are doing are bringing all those terms into scope in the containing context.

Trying to do 2 things: avoid special cases like one special dimension, and also give some reusable intuition/uniform implementation for what "implements" does. I def see the tension of a slot-only array and the want to also have a class-level array, so yes we need some way of doing this

[rly]

Something to follow but is not totally relevant to this effort is the Python Array API standard: https://data-apis.org/array-api/latest/index.html

Also potentially relevant is ZEP0004 on Zarr Conventions: zarr-developers/zarr-specs#262

[sneakers-the-rat]

ooo very interesting. i have some things to add to this esp re the typing section :)

[sneakers-the-rat]

oh boy that's heavy https://data-apis.org/array-api/latest/API_specification/index.html

[sneakers-the-rat]

Important difference between "api" and "interchange format" or "specification," lots of useful stuff there

[rly]

Also adding a note from @melonora that this package/effort may be useful https://github.com/loco-philippe/ntv-numpy

[melonora]

Question, I am not certain whether we actually cover this particular use case even with labeled arrays. Particularly when describing transformations you have a transformation matrix in which you have particular labels. For example in ome with the affine transform:

At the moment though with labeled arrays you could have a matrix being labeled, you can't necessarily add a description for an individual value, am I right? If so I think there are other use cases for this that would warrant a try for supporting it.

[sneakers-the-rat]

I think since its a linked data format we could do something like this at the schema level rather than the metamodel level, what do ya think?

(In pseudocode for illustration purposes)

classes:
  Affine:
    attributes:
      target_array:
        range: ArrayExpression
      transform:
        array:

[melonora]

sorry for not responding earlier. I am not certain whether we are talking about the same thing. I mean to describe what each value of the transformation matrix means, the matrix when you have yx instead of xy looks different for example from the affine matrix when having xy.

In any case with the transforms, usually the coordinate space is named and then transforms are described for how to get data in a particular coordinate system.

[sneakers-the-rat]

oh sorry, I thought you meant like 'can we have an additional transformation array describe an existing labeled array' . rather than 'can we label individual values in an array.'

no we haven't covered that yet, but i do see the case you're talking about here with a specific stereotyped array where each element is meaningful (as opposed to arrays as we have been thinking about them where it's the dimension that's the meaningful thing). In the past we had decided not to put things like column/row major encoding into the top-level schema and treat that as an implementation detail, and that sort of seems like the same thing here: like the xy also needs to match the ordering of the array it's being applied to, etc. and we get into a situation where the schema specification is overstrict. like this would be something i would probably lean towards being at a schema rather than metamodel level, but i would want to try and implement it and see if it is necessary to be in the metamodel because i also see what you're saying.

[melonora]

Yeah I agree this should be more defined at the schema level, but it would be good to test whether the metamodel allows for specifying this at the schema level. Main reason is that different kinds of software define the affine transformations in different ways, so if you are able to provide a schema on how your software defines it, it can make the ecosystem more interoperable:) I am not certain whether there are other usecases like this but I just wasn't certain whether the current metamodel allows to specify this in a schema.

[sneakers-the-rat]

Could an array range be a type that has a label annotation? Just spitballing

[melonora]

hmm think with the next session we could have a small discussion. Optionally we could also define a labeled array which consists out of a collection of elements which have indices and also individually can have a label and description, which I think is slightly different from the other labeled array implementation.

[sneakers-the-rat]

@melonora see nptyping's handling of explicitly labeled array elements: https://github.com/ramonhagenaars/nptyping/blob/master/USERDOCS.md#dimension-breakdowns

so something like

myslot:
  array:
    dimensions:
    - alias: first_dim
      items: [x, y]
    - alias: second_dim
      items: [a, b]

for when one wants to label the entire nth element of that whole dimension, ie.

		second_dim
		a	b
first_dim	x	(x, a) = 0	(x, b) = 2
	y	(y, a) = 1	(y, b) = 3

where the value 0 is labeled as x in the first dimension and y in the second dimension. That would imply that exact_cardinality: len(items) and should probably be mutually exclusive.

or like

myslot:
  array:
    dimensions:
      - alias: first_dim
        exact_cardinality: 2
      - alias: second_dim
        exact_cardinality: 2
    items: [[a, b], [x, y]]

for when you want to label each item in an array independently. That would mean that the shape of items would have to match that of all the specified dimensions, and i don't want to also be able to switch row v. column major in the items specification so i would be in favor of just picking one. that is also sort of annoying where you might just want to annotate a single item, but idk if i also want to support yet another syntax like

items:
  - first_dim: 0
    second_dim: 1
    item: x
  - first_dim: 1
    second_dim: 1
    item: y

and so on.

Ideally it would be cool if instead of just strings we could specify classes/slots so that the item annotations can have their own metadata like a linked data URI or whatnot, but then we get into cool recursion territory where an annotation has an array which has item annotations...

[sneakers-the-rat]

Returning to work partially, pushed an update to numpydantic that takes over control of the shape syntax (part of the longer term plan to get rid of nptyping and make a cleaner, more inspectable syntax) and adds shape ranges so annotations dont have to be so verbose:
https://numpydantic.readthedocs.io/en/latest/syntax.html#ranges
https://github.com/p2p-ld/numpydantic/releases/tag/v1.2.0
p2p-ld/numpydantic@07ab3d1

Should be able to pull the numpydantic generator early next week, really not much work at all since a lot of the work is done in the npd package. Then with that linkml arrays become extremely implementation general and ready to use in normal contexts where ppl are usually using array libraries instead of lists of lists.

Some fun stuff in there like direct support for videos, and will be standing by to add in support for labeled arrays and an interface to xarray.

[sneakers-the-rat]

What is up arrays chat we got ourselves a new array representation and its a hoot and a half- linkml about to be able to hook into whatever the heck array library you want

#2178