Skip to content

Commit

Permalink
Introduce a new links category for the YAML definitions (#691)
Browse files Browse the repository at this point in the history 
* Make the reader and validator understand links

Add the new category so that it can be read

* Make code generation create necessary link code

Effectively a header per link with a typedef and a common .cc file with
registration code for all links

* Make sure that SIO backend also works

* Separate link registration macros

* Remove special casing from roundtrip tests again

* Make sure to make link collections available

* Make sure that all user facing types exist

Otherwise things start to break in cases where users explicitly use
these types at the moment.

* Add documentation for new capabilities and links

* Update available templates documentation

* Only generate SIOBlocks if necessary

* Use walrus operator to improve readability

---------

Co-authored-by: Mateusz Jakub Fila <[email protected]>
Co-authored-by: Andre Sailer <[email protected]>
  • Loading branch information
@tmadlener @m-fila @andresailer
3 people authored Dec 2, 2024
1 parent 497b1be commit 39587ee
Show file tree
Hide file tree
Showing 26 changed files with 351 additions and 92 deletions.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ version and build that automatically. This behavior is controlled via the
`USE_EXTERNAL_CATCH2` cmake variable. It defaults to `AUTO` but can also be set
to `ON` or `OFF` to fully control the desired behavior.

### Python > 3.6
### Python >= 3.8

Check your Python version by doing:

Expand Down
32 changes: 32 additions & 0 deletions doc/datamodel_syntax.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,38 @@ define which `Types` can be used with this interface class, in this case the
not allow for mutable access to their data.** They can be used in relations
between objects, just like normal `datatypes`.

## Definition of links
Podio offers a templated `Link` class ([see here for more details](links.md))
that allows one to link two arbitrary datatypes without having to introduce a
`OneToOneRelation` or `OneToManyRelation` inside the corresponding datatypes. In
order to keep the full definition of a datamodel in the YAML file it is possible
to declare `links` in the YAML file:

```yaml
links:
ExampleLink:
Description: "A link between two (podio generated) objects"
Author: "It could be you"
From: ExampleHit
To: TypeWithEnergy
```

This definition will yield the following typedefs
```cpp
using ExampleLinkCollection = podio::LinkCollection<ExampleHit, TypeWithEnergy>;
using ExampleLink = typename ExampleLinkCollection::value_type;
// this is equivalent to
// using ExampleLink = podio::Link<ExampleHit, TypeWithEnergy>;
using MutableExampleLink = typename ExampleLinkCollection::mutable_type;
// this is equivalent to
// using MutableExampleLink = podio::MutableLink<ExampleHit, TypeWithEnergy>;
```

additionally, this will generate the necessary code to enable I/O for this link
type.

### Assigning to interface types

Interface types support the same functionality as normal (immutable) datatypes.
Expand Down
1 change: 1 addition & 0 deletions doc/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Welcome to PODIO's documentation!
examples.md
frame.md
userdata.md
links.md
storage_details.md
cmake.md
advanced_topics.md
Expand Down
16 changes: 10 additions & 6 deletions doc/links.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,16 @@ For a more detailed explanation of the internals and the actual implementation
see [the implementation details](#implementation-details).

## How to use `Link`s
Using `Link`s is quite simple. In line with other datatypes that are
generated by podio all the functionality can be gained by including the
corresponding `Collection` header. After that it is generally recommended to
introduce a type alias for easier usage. **As a general rule `Links` need
to be declared with the default (immutable) types.** Trying to instantiate them
with `Mutable` types will result in a compilation error.
Using `Link`s is quite simple. The most straight forward way is to simply
declare them as part of the datamodel, [as described
here](datamodel_syntax.md#definition-of-links). That will result in code
generation that effectively does what is described below here. However, it's not
strictly necessary to do that in case non-generated code is preferred. In line
with other datatypes that are generated by podio all the functionality can be
gained by including the corresponding `Collection` header. After that it is
generally recommended to introduce a type alias for easier usage. **As a general
rule `Links` need to be declared with the default (immutable) types.** Trying to
instantiate them with `Mutable` types will result in a compilation error.

```cpp
#include "podio/LinkCollection.h"
Expand Down
29 changes: 16 additions & 13 deletions doc/templates.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,19 +26,22 @@ Note that some of the information below will only apply to either of these gener
Currently PODIO loads templates that are placed in [`<prefix>/python/templates`](/python/templates).
They are broadly split along the classes that are generated for each datatype or component from the EDM definition:

| template file(s) | content | generated file(s) |
|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
| `Component.h.jinja2` | Definition for each component | `[<package>/]<component-name>.h` |
| `Data.h.jinja2` | POD struct of each datatype (living in the POD layer) | `[<package>/]<datatype-name>Data.h` |
| `Obj.{h,cc}.jinja2` | `Obj` class for each datatype (living in the object layer) and managing resources | `[<package>/]<datatype-name>Obj.h`, `src/<datatype-name>Obj.cc` |
| `[Mutable]Object.{h,cc}.jinja2` | The user facing interfaces for each datatype (living in the user layer) | `[<package>/][Mutable]<datatype-name>.h`, `src/[Mutable]<datatype-name>.cc` |
| `Collection.{h,cc}.jinja2` | The user facing collection interface (living in the user layer) | `[<package>/]<datatype-name>Collection.h`, `src/<datatype-name>Collection.cc` |
| `CollectionData.{h,cc}.jinja2` | The classes managing the collection storage (not user facing!) | `[<package>/]<datatype-name>CollectionData.h`, `src/<datatype-name>CollectionData.cc` |
| `datamodel.h.jinja2` | The *full datamodel header* that includes everything of a generated EDM (via including all generated `Collections`). | `[<package>]/<package>.h` |
| `selection.xml.jinja2` | The `selection.xml` file that is necessary for generating a root dictionary for the generated datamodel | `src/selection.xml` |
| `SIOBlock.{h,cc}.jinja2` | The SIO blocks that are necessary for the SIO backend | `[<package>/]<datatype-name>SIOBlock.h`, `src/<datatype-name>SIOBlock.cc` |
| `MutableStruct.jl.jinja2` | The mutable struct definitions of components and datatypes for julia | `[<package>/]<datatype-name>Struct.jl`, `[<package>/]<component-name>Struct.jl` |
| `ParentModule.jl.jinja2` | The constructor and collection definitions of components and datatypes in the data model are contained within a single module named after the package-name | `[<package>/]<package-name>.jl` |
| template file(s) | content | generated file(s) |
|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
| `Component.h.jinja2` | Definition for each component | `[<package>/]<component-name>.h` |
| `Data.h.jinja2` | POD struct of each datatype (living in the POD layer) | `[<package>/]<datatype-name>Data.h` |
| `Obj.{h,cc}.jinja2` | `Obj` class for each datatype (living in the object layer) and managing resources | `[<package>/]<datatype-name>Obj.h`, `src/<datatype-name>Obj.cc` |
| `[Mutable]Object.{h,cc}.jinja2` | The user facing interfaces for each datatype (living in the user layer) | `[<package>/][Mutable]<datatype-name>.h`, `src/[Mutable]<datatype-name>.cc` |
| `Collection.{h,cc}.jinja2` | The user facing collection interface (living in the user layer) | `[<package>/]<datatype-name>Collection.h`, `src/<datatype-name>Collection.cc` |
| `CollectionData.{h,cc}.jinja2` | The classes managing the collection storage (not user facing!) | `[<package>/]<datatype-name>CollectionData.h`, `src/<datatype-name>CollectionData.cc` |
| `datamodel.h.jinja2` | The *full datamodel header* that includes everything of a generated EDM (via including all generated `Collections`). | `[<package>]/<package>.h` |
| `selection.xml.jinja2` | The `selection.xml` file that is necessary for generating a root dictionary for the generated datamodel | `src/selection.xml` |
| `SIOBlock.{h,cc}.jinja2` | The SIO blocks that are necessary for the SIO backend | `[<package>/]<datatype-name>SIOBlock.h`, `src/<datatype-name>SIOBlock.cc` |
| `LinkCollection.h.jinja2` | The header that is generated for each *Link* containing effectively typedefs only | |
| `DatamodelLinksSIOBlock.cc.jinja2` | The .cc file that is necessary for enabling SIO based I/O for *Link*s | |
| `DatamodelLinks.cc.jinja2` | The global .cc file that is necessary to enable I/O for all *Link*s | |
| `MutableStruct.jl.jinja2` | The mutable struct definitions of components and datatypes for julia | `[<package>/]<datatype-name>Struct.jl`, `[<package>/]<component-name>Struct.jl` |
| `ParentModule.jl.jinja2` | The constructor and collection definitions of components and datatypes in the data model are contained within a single module named after the package-name | `[<package>/]<package-name>.jl` |


The presence of a `[<package>]` subdirectory for the header files is controlled by the `includeSubfolder` option in the yaml definition file.
Expand Down
27 changes: 7 additions & 20 deletions include/podio/LinkCollection.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,17 @@
#define PODIO_ENABLE_SIO 0
#endif

/// Macro for registering links at the CollectionBufferFactory by injecting the
/// corresponding buffer creation function.
#define PODIO_REGISTER_LINK_BUFFERFACTORY(FromT, ToT) \
/// Main macro for declaring links. Takes care of registering the necessary
/// buffer creation functionality with the CollectionBufferFactory.
#define PODIO_DECLARE_LINK(FromT, ToT) \
const static auto PODIO_PP_CONCAT(REGISTERED_LINK_, __COUNTER__) = \
podio::detail::registerLinkCollection<FromT, ToT>(podio::detail::linkCollTypeName<FromT, ToT>());

/// Macro for registering the necessary SIOBlock for a Link with the SIOBlock factory
#define PODIO_REGISTER_LINK_SIOFACTORY(FromT, ToT) \
const static auto PODIO_PP_CONCAT(LINK_SIO_BLOCK_, __COUNTER__) = podio::LinkSIOBlock<FromT, ToT>{};

#if PODIO_ENABLE_SIO && __has_include("podio/detail/LinkSIOBlock.h")
#include "podio/detail/LinkSIOBlock.h"
/// Main macro for declaring links. Takes care of the following things:
/// - Registering the necessary buffer creation functionality with the
/// CollectionBufferFactory.
/// - Registering the necessary SIOBlock with the SIOBlock factory
#define PODIO_DECLARE_LINK(FromT, ToT) \
PODIO_REGISTER_LINK_BUFFERFACTORY(FromT, ToT) \
PODIO_REGISTER_LINK_SIOFACTORY(FromT, ToT)
#else
/// Main macro for declaring links. Takes care of the following things:
/// - Registering the necessary buffer creation functionality with the
/// CollectionBufferFactory.
#define PODIO_DECLARE_LINK(FromT, ToT) PODIO_REGISTER_LINK_BUFFERFACTORY(FromT, ToT)
#include <podio/detail/LinkSIOBlock.h>
/// Macro for registering the necessary SIOBlock for a Link with the SIOBlock factory
#define PODIO_DECLARE_LINK_SIO(FromT, ToT) \
const static auto PODIO_PP_CONCAT(LINK_SIO_BLOCK_, __COUNTER__) = podio::LinkSIOBlock<FromT, ToT>{};
#endif

#endif // PODIO_LINKCOLLECTION_H
41 changes: 38 additions & 3 deletions python/podio_gen/cpp_generator.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,14 +91,16 @@ def pre_process(self):

return {}

def post_process(self, _):
def post_process(self, datamodel):
"""Do the cpp specific post processing"""
self._write_edm_def_file()

if "ROOT" in self.io_handlers:
self._prepare_iorules()
self._create_selection_xml()

if the_links := datamodel["links"]:
self._write_links_registration_file(the_links)
self._write_all_collections_header()
self._write_cmake_lists_file()

Expand Down Expand Up @@ -207,6 +209,23 @@ def do_process_interface(self, _, interface):
self._fill_templates("Interface", interface)
return interface

def do_process_link(self, _, link):
"""Process a link definition and generate the necessary code"""
link["include_types"] = []
for rel in ("From", "To"):
rel_type = link[rel]
include_header = f"{rel_type.bare_type}Collection"
if self._is_interface(rel_type.full_type):
# Interfaces do not have a Collection header
include_header = rel_type.bare_type
link["include_types"].append(
self._build_include_for_class(
include_header, self._needs_include(rel_type.full_type)
)
)
self._fill_templates("LinkCollection", link)
return link

def print_report(self):
"""Print a summary report about the generated code"""
if not self.verbose:
Expand Down Expand Up @@ -506,8 +525,10 @@ def _write_list(name, target_folder, files, comment):

def _write_all_collections_header(self):
"""Write a header file that includes all collection headers"""

collection_files = (x.split("::")[-1] + "Collection.h" for x in self.datamodel.datatypes)
collection_files = (
x.split("::")[-1] + "Collection.h"
for x in list(self.datamodel.datatypes.keys()) + list(self.datamodel.links.keys())
)
self._write_file(
os.path.join(self.install_dir, self.package_name, f"{self.package_name}.h"),
self._eval_template(
Expand All @@ -520,6 +541,20 @@ def _write_all_collections_header(self):
),
)

def _write_links_registration_file(self, links):
"""Write a .cc file that registers all the link collections that were
defined with this datamodel"""
link_data = {"links": links, "incfolder": self.incfolder}
self._write_file(
"DatamodelLinks.cc",
self._eval_template("DatamodelLinks.cc.jinja2", link_data),
)
if "SIO" in self.io_handlers:
self._write_file(
"DatamodelLinkSIOBlock.cc",
self._eval_template("DatamodelLinksSIOBlock.cc.jinja2", link_data),
)

def _write_edm_def_file(self):
"""Write the edm definition to a compile time string"""
model_encoder = DataModelJSONEncoder()
Expand Down
Loading

0 comments on commit 39587ee

Please sign in to comment.