How to specify the LinkML equivalent of Union[List[int], str]?

[candleindark]

Consider the following Pydantic model.

from typing import Union, List

from pydantic import BaseModel


class Foo(BaseModel):
    x: Union[List[int], str]

I want to specify Foo as a class and x as a slot in a LinkML schema, but I have trouble defining the slot for representing x. In particular, I couldn't find a way to express the constraints imposed by the Union[List[int], str] type annotation in the Pydantic model in LinkML. I am aware that I can use any_of to express a constraint on the range of a slot that allows one of a set of possible types. However, the use of any_of doesn't seem to allow "re-specification" of the multivalued metamodel slot since any_of has a range of Expression (and re-specification of multivalued doesn't make sense because of monotonicity in LinkML schema).

How I can accomplish what I want in LinkML?

Thanks in advance

[cmungall]

Thanks, yes in fact in addition to monotonicity issues, there is an arbitrary restriction that multivalued is only allowed at the slot level and not the slot expression level. This restriction will be lifted in the next release of the metamodel

[candleindark]

Will the next metamodel solve the monotonicity issue as well? With monotonicity in place, the slot will have both restrictions of multivalued==True and multivalued==False.

[cmungall]

multivalued at the expression level:

linkml/linkml-model#197

[candleindark]

Thanks @cmungall, but I don't think the PR works as intended. I filed #2190.

[yarikoptic]

. This restriction will be lifted in the next release of the metamodel

Just FTR -- that particular feature was indeed included in the "next release" back then was 1.8.0 released July 3.

Allowing multivalued to be used in slot expressions as well as top level slots. by @cmungall in linkml/linkml-model#197

[sneakers-the-rat]

This, like arrays, makes me wonder if the types system is underused. It makes sense to me to be able to declare a type that is a list of ints, since it doesnt have semantic content like a slot but is just a type. so then one would do something like

types:
  intlist:
    is_a: integer
    multivalued: true

slots:
  x:
    any_of:
    - range: string
    - range: intlist

(Pseudocode bc I dont remember the type syntax offhand)

[candleindark]

It may very well be true that the types system is underused. I couldn't find documentation about it at https://linkml.io/linkml/index.html. I am able to find documentation about it at https://linkml.io/linkml-model/latest/docs/TypeDefinition/, but one has to do quite a bit of guesswork to find out how the type system work from that.

The solution you suggested works if the definition of intlist behaves as you intended. However, there is indication that TypeDefinition doesn't work the way you intended. There is no metamodel slots named is_a and multivalued in TypeDefinition, and according to the documentation, a TypeDefinition is "an element that whose instances are atomic scalar values that can be mapped to primitive types." Since an instance of List[int] is not an atomic scalar value, TypeDefinition may not be able to provide the solution we need.

[sneakers-the-rat]

Right right, just was showing that as an example of how the type system might be used, not how it works currently.

I think theres value in keeping things well defined, but I think, esp with some of what we've been doing in linkml arrays, some of that ends up getting shunted over into slots, so slots become sorta abused/amorphous to take up the slack.

From a generator POV, I could see how this would be implemented relatively neatly but it would require there to be more structure with how generators work than there is currently. Like in python-based generators this would be simple bc you just resolve the type first and use that as the "inner part" for later slot and class generation. In json-schema you would put it in a definitions block and use it by reference. Things get a little weird in RDF but one could still I think keep the type concern separated from slots/classes, just would need to restructure generator.

Semantically I think it makes sense. Types are still some composition of atomic types, and slots are the meaning given to a type. I have come to think that arrays should be types rather than a parameterization of slots, bc we're facing a decent amount of trouble with "levels" of slots referring to other slots and etc. But thats another conversation.

Just to say I think it should be possible to express this and I think it points to some gap in the type system :)

[candleindark]

Something like an array should be a type indeed. It's much more intuitive that way.

[cmungall]

I agree this is more informative from a programming language perspective, rather than a W3c/XSD/RDF perspective. But forgetting the original question for the moment and taking a simpler case of a list of ints; we *could* allow multivalued in types; this would make the following equivalent ``` types: # INTLIST=List[int] intlist: typeof: integer multivalued: true slots: x: ## x : INTLIST, or x: List[int] range: intlist ``` and ``` slots: x: ## x : List[int] range: integer multivalued: true ``` Which is fine from a programming language perspective, we're used to lots of different ways to say the same things - I wonder if this adds too much cognitive overhead for a modeling language however (worth taking up to its own top-level discussion?)

…

On Sat, Jun 15, 2024 at 4:59 PM Jonny Saunders ***@***.***> wrote: This, like arrays, makes me wonder if the types system is underused. It makes sense to me to be able to declare a type that is a list of ints, since it doesnt have semantic content like a slot but is just a type. so then one would do something like types: intlist: is_a: integer multivalued: true slots: x: any_of: - range: string - range: intlist (Pseudocode bc I dont remember the type syntax offhand) — Reply to this email directly, view it on GitHub <#2154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMMOMZN7RPN4XW6G4K3E3ZHTIODAVCNFSM6AAAAABJC6F4X2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBUGEYDO> . You are receiving this because you commented.Message ID: ***@***.***>

[candleindark]

This solution can solve the particular problem of this post, the expression of Union[List[int], str]. It can also solve another example of the general problem I am facing, the expression of Union[List[List[int]], str]. However, to solve this second problem, this solution requires defining two types. As the number of nesting increases, the number of type definitions are needed (many of them are intermediate and possibly not needed elsewhere). Is there a cleaner solution?

[sneakers-the-rat]

Yes maybe worth its own top-level discussion, I feel like there is a bit of fuzziness at the border of types and slots that would be worth hashing out - eg. Types can have some logic like regex patterns so theyre not totally just atomic types, but then often type-like logic gets shunted into slot-like logic, which is mostly fine except in instances like this and the labeled arrays where one needs to be able to "shift levels" between types and slots, compose/refer to other slots, etc.

the expression of Union[List[List[int]], str]

Speaking of arrays, the range of range is Element, so is this valid?:

slots:
  2d_array:
    range: integer
    array:
      exact_number_dimensions: 2

  my_slot:
    any_of:
    - range: 2d_array
    - range: string

Sort of a good example where we want to use a slot like a type

[candleindark]

If the above example is valid and able to express Union[List[List[int]], str], it highlights the fuzziness you point to. The slot is being used as a type.

[candleindark]

Another example of the general problem I am facing is to express Union[List[List[M]], str] where M is a class. If the type system only supports composition of atomic types, then the type system can't be used to solve this problem.

[sneakers-the-rat]

btw, not responsive to the problem directly, and it would definitely be a hack, but at least with the pydantic generator you can get this kind of behavior Union[T, str] by fiddling with the inlining settings, can't remember offhand what combo does it, but one of them always adds str as an allowed type, i think expecting the string to be the URI of the element. might also depend on whether something is an identifer, idk that part of the generator is still murky to me.

like i said, definitely a hack, but might be a workaround pending an official fix

[candleindark]

Thanks for letting me know. Unfortunately, I don't think this hack is applicable in my case.

I am actually working on converting existing models in defined in Pydantic to LinkML. The examples I gave out are just specific cases of a general problem. Sorry that I have not been completely clear about what the general problem is. Ideally, I want to be able to express a union of any Python types in LinkML. The types can be an atomic type or a composition of atomic types and user defined types. An live example is https://github.com/dandi/dandi-schema/blob/e135307e38e8fccf83676cf1d6faba9b142a8de0/dandischema/models.py#L1358.