Glaze as a reflection library

[fregate]

Hi @stephenberry !

I've really enjoyed using glaze as a json parser (and serializator, for sure).

For my project I have a few formats to receive and serialize to json, for example. One of these formats is Erlang binary term format. And I've realized that my current library is too... ugly, I guess, and slow in comarsion with glaze.

So I try to made glaze-reflection as underlying library for Erlang format and it works! I've made only a few changes in glaze to handle new format. But it works (only for read at this moment).

Do you think about split glaze (in fact it already splitted in core, json and beve) to "reflection library" and "format parsing"?

In any case, I put my repo here - If you would be so kind to check my glaze patch (thirdparty/glaze.diff) and applicable it to main repository (or make PR is better way to discuss these changes?)

https://github.com/fregate/termlib/tree/reflection

Thanks in advance

PS My current lib is, like, proof of concept (and I think, some constructions with template magic I'm not fully understand), but it works and I will improve it (writing, at least)

[stephenberry]

@fregate
Wow, this is really great! We've tried to design Glaze so that it is straightforward to add more formats. I would be happy to add you as a collaborator to Glaze so that you can push/pull to main and add support for the Erlang format.

I've recently made Glaze even better for developing new formats. Features like glz::custom just automatically work with any format in Glaze, this means you gain a ton of features for free when adding new formats.

Anyway, can I add you as a collaborator to Glaze? You could then pull Glaze, make a branch for Erlang from your code, and then create a pull request. I can look over your pull request and offer tips/explain the architecture more.

Reading is typically more complicated, so writing should be straightforward.

Also, the core reflection utilities in Glaze have become even more separated from formats since version 3, so your ideas here of splitting out utilities (reflection vs serialization) is very much a focus.

[fregate]

Hi, @stephenberry !

What a great offer! ) Be a part of this excellent library as a contributor! )

Right now Im on a vacation and in a few next weeks can't put some effort in. Sorry. But later - of course I will continue.

So I never was a contributor to community projects before - this will my first experience to create something useful and public available )

[stephenberry]

Cool! I sent you an invite. No rush on doing any of this, just let me know when you'd like me to take a look at your pull request. Have a great vacation!

[fregate]

Hello, @stephenberry !

I've started some refactoring for support new format and go deeper and deeper.

My first objective, of course is the new format, but while adding it, I've realized that there is some new necessary (from my point) changes I suggesting to make. For example make glz::opt suitable only for current working format and not to use one big struct with 30+ bools (for json mainly). Change it to uint64 and use like bitmask (not class, class std::bitset is constexpr only in C++23+). And also it removes unreadable errors like read<{1, 0, 1, 0, 0,..... <many times>}, char> to read<{1, 12312312}, char>

This is my first commit to change opts to a new way.

And second point like separate writer and reader like some entities, with own options. After that you can easily write something like beve_to_json, but with any formats you want (csv_to_beve, beve_to_erlang and so on).

And according to my first, make error_code just ints, not enum, because many error are format specific and if split every format to separate space, base on core, extending enum is a wrong way to handle core and new formats.

This is my first thoughts. What do you think about?

I saw you started discussion about v5 API changes. I can write these thoughts there.

[stephenberry]

A agree with you that glz::opts needs to be reworked into a bit-packed structure (I've actually attempted this twice already and there are internal options that are bit packed). The challenge is making this as clean as possible for compile time programming. I would like to be able to bitwise or options like this: glz::opts{glz::opt::null_terminated | glz::opt::new_lines_in_arrays | glz::opt::validate_skipped}.

This means that the glz::opt enum would have to bit align its fields and we could only fit 64 options in this primary enumeration. If we have more than 64 options then we need another options enumeration. But, right now 64 options would cover all our boolean options. But, there are also options that are not boolean such as indentation_width. Which I think needs to remain a uint8_t within the struct.

I like the idea of making the error code simply an integer that has its meaning based on the format. But, I also could see simply organizing the errors within the enum because so many formats have the same errors. I would hate to duplicate a hundred lines of error codes for each format that is added. If we moved to integers then we'd want to reserve ranges of numbers for error codes that are common across formats and ranges for format specific errors, this would allow us to use common error codes.

I don't quite get what you're aiming for with this comment:

And second point like separate writer and reader like some entities, with own options. After that you can easily write something like beve_to_json, but with any formats you want (csv_to_beve, beve_to_erlang and so on).

If you could explain a bit more that would be helpful. I do have ideas for making the boilerplate code for new formats much cleaner.

Feel free to add thoughts to the discussion for v5 breaking changes. I think improving glz::opts is definitely an aim.

[fregate]

Also I think about bitwise and I like that approach, but in my suggestion all the options put as bits even indentation_char or similar. And right now I can't realize how to do that.

Like every option is struct that contains bit number and width. After that it could work, but you have to make enum and corresponding structs. Worth try, I think

[fregate]

Ok. Thanks. I will write my thought about new opts.

But my second point was also about it. Try to be more clear. Right now, glaze always uses same opts for read<> and write<> function-set. And my thought that it can be splitted to 2 entity - reader and writer with own set of options (so if opts became bitmask it expand bit-space for options). A little picture.

If we think about writers, than write to struct is one of possible writer, like "typed_writer". With "json_writer" you can write to json, you can write to beve with "beve_writer" and etc. Same API for all. But different writers. And different options.

And this actual and for readers. You read from some data-stream and write to other format (even to data-stream with, for examlpe, prettifier), like "json_reader(input)" -> "json_writeropts:pretty(output)", or "beve_reader(input)" -> "json_writer(output)". No need to store in some external struct, because you don'y use a "typed_writer". For "typed_writer" it is like "json_reader(input)' -> "typed_writer(struct_instance)".

So a new syntax will like...

auto json = "..."s;
my_struct s{};
auto r = json_reader<writer_options>(json); // it like view for data, if no one uses it - nothing happend
auto w = typed_writer<reader_options>(r, s);

and of course, you can create some helpers like beve_to_json, or just write_json with these reader and writer internally.

PS I read this and think, this is very messy )

[stephenberry]

I'll quickly dump another idea I had for cleaner compiler time options. The template errors are long because the options are embedded in the template parameters. However, if we instead passed a pointer to options as the template parameter, then the template is simply built upon a 64bit pointer. It looks like compilers will give very clean template errors in this case.

Here is an example on compiler explorer: https://godbolt.org/z/zK4M69ndK

Notice that the compiler doesn't declare any fields of the options, but rather only returns: int func() [with auto* Opts = (& opts)]
This is because the template is unique by the pointer that is passed to it.

This would allow us to use the old options structure and keep adding all kinds of new options without bit packing and the template errors would always remain small.

I'll reply to your other thoughts soon, but I just wanted to throw this idea out there before you developed a lot of code, because I think this might be the cleanest solution.

[fregate]

Wow. Interesting. It's way easier that packing into a bitmask. And flexible, because you can use different underlying structs for different formats

[stephenberry]

Yeah, it does make having different sets of options clean as well.

The really important thing to note is that we will want to use a global pointer, otherwise the template instantiations will need to be recomputed for every pointer instance, which would be terrible for compile times.

[fregate]

Maybe concept for Opts is enough? Like,

template <typename T>
concept opt_t = requires (T o) {
typename T::format;
is_integral<decltype(o.format)>;
// some common options for all formats
}

template <opt_t T>
concept json_opt_t = requires (T o) {
o.internal;
/// other checks to handle all the json features
}

template <opt_t T>
concept beve_opt_t = requires...

// and so on

After that in core we will use template <opt_t auto Opts, ...> and json-specific functions template <json_opt_t auto Opts, ...>
And in this case API won't changed significantly, only format-specific functions, but you can use own structs for all the format without unnecessary fields.
And compile time will be the same, I guess.

[stephenberry]

A do like the idea of using a concept.

There is an issue that needs to be solved before pointer based options can be implemented. That issue is that Glaze often sets options internally, but these settings must be assigned to copies of the options, because template parameters are immutable. Creating these copies is no problem and how the code functions currently. The problem is that any copy will introduce a new pointer, which would cause a new template instantiation down a path that uses the same exact template parameters.

This is a major issue, because we want the template to be instantiated only once for a unique set of options, but if we make copies of options then we get different pointers and different instantiations.