Add a rule-based file editor to CAF

[jouvin]

In some circumstances it is required to edit some files (typically provided by a RPM) to change some configuration parameters and doing it on the template site is impractical or unmaintainable because the file provided by the RPM evolves and you just want to ensure that some critical parameters have a defined value. Doing it on the template site would require to update the templates at each new version of the RPM and would lead to a tricky dependency management between the RPM version and the template version. A typical example are EMI DPM and Xrootd, managed by ncm-dpmlfc and ncm-xrootd. Puppet for example is using the "template approach" and this is a nigthmare to express the dependency between the Puppet module version and the RPM version.

An approach that has been successfully used for many years in ncm-dpmlfc and ncm-xrootd is to use a rule-based editor in the component perl module. The basic idea is that you express the variable/keyword to be edited in your config file with some value from your Quattor configuration if some conditions are met, like a configuration option defined in a certain part of the configuration. This is very flexible and powerful. Currently, this is a (long) piece of Perl code that is inside both configuration modules (this is the same code copied/pasted, the ncm-dpmlfc being slightly more recent with a few additional features.

As this is not good to duplicate such a piece of code and because it could also be beneficial for other use cases, I'd like to add it as a new CAF module, for example FileRuleBasedEditor. Before going furtther, I'd like to discuss if there is an agreement on the features provided by the current rule-based editor.

Let start by describing how rules are constructed (see comments in https://github.com/jouvin/configuration-modules-grid/blob/dpmlfc/ncm-dpmlfc/src/main/perl/dpmlfc.pm around line 147 and the rule examples below):

• For each file that you want to edit, you define a hash containing a set of rules.
• For each entry in the hash (unordered), the key is matched against a "keyword" in the configuration file (the exact way of matching a keyword depends on the line format, e.g. shell script, apache config file...) and the value is defining when/how to update the configuration line. This value (a string) has the following format:

[condition->]option_name:option_set[,option_set,...];line_fmt[;value_fmt]

With:

• condition: an option or an option set that must exist for the rule to be applied. Both option_set and option_name:option_set are accepted (see below). Only one option set is allowed and only existence, not value is tested. In addition, the condition may be negated (option or option_set must not exist) by prepending a '!' to it.
• option_name: the name of an option that will be retrieved from the configuration
• option_set: the set of options the option is located in (for example 'dpnsHost:dpm' means 'dpnsHost' option of 'dpm' option set (an option set is a sub-hash of the option hash passed to the rule-based editor). 'GLOBAL' is a special value for 'option_set' indicating that the option is a global option, instead of belonging to a specific option set (a global option is an option at the first level of the option hash).
• line_fmt` indicates the line format for the parameter. The following formats are currently supported (names reflect their original usage and should be changed when moving this into CAF):
• envvar : a sh shell environment variable definition (export VAR=val)
• param : a sh shell variable definition (VAR=val)
• xrdcfg : a keyword value line, as used by Xrootd or Apache config files.
• xrdcfg_setenv : a setenv line, as used by Xrootd config files or csh/tcsh script.
• xrdcfg_set : a set line, as used by Xrootd config files.
• line_fmt allows to specify how to process lists and nlists. In particular if an option is a pan list, the rule-based editor can display the value of the list separated by the selected separator.
• 'value_fmtallows special formatting of the value. This is mainly used for boolean values to get them printed asyesorno`.

Inline comments are not supported in xrdcfg family of formats whereas every line updated by Quattor receive an inline comment in envvar and paramformats.

In addition, if the keyword (hash key) is starting with a -, this means that the matching configuration line must be removed/commented out (instead of added/updated) from the configuration file if present. If it is starting with a '?', this means that the matching line must be removed/commented out if the option is undefined (note that this is also possible to specify at the file level, for the whole ruleset, if a line matching one rule but whose condition is not met must removed/commented out).

[stdweird]

@jouvin quite a long post. i'm not going to comment on all sections at once though

a. the disadvantages of filewriter/metaconfig are indeed what you sum up, but at least for metaconfig, extending support for a new option can typically be done by extending the schema, and textrender supports multiversioned templates, but the coupling with the rpm version is indeed manual.
you however do not mention how fileeditor handles rpm updates with newer configs (the rpm files are typically renamed to .rpmnew, and as such not seen by the component/editor). also, when adding support for new options, you also have to modify the component as well.

b. the way you distribute the rules should not be perl code. similar to TT files, one could ship editor rules like

# first section are file paths match
/etc/httpd/conf.d/ssl.conf
---
# section some global options, like case insensitive matching of the variable name (as in apache config files)
---
# last section are the rules, one line per rule

ofcourse this example format is heavily influenced by the regexptests from TextRender; but this is static data, not code.
(you would also need version support etc etc, but that is also the case in the perl version)

c. the rules should also indicate where missing data is added (imagine adding a single entry only to the 2nd virtual host in a apache config file)

d. i once tought about something similar based on TextRender: make TT files to generate the config for single line values (and than splitting by line and looping over them to apply to the whole text), and provide the conditions in such a way that the regexp pattern for each can also be rendered by the same TT files.
e.g. TT file

VARX='[% x %]'

would create the correct configuration VARX='value' using the regular rendering methods with configdata '{x => 'value'}, and provided with pattern for x {x => '[^']*'}, it would generate VARX='[^']*', which is generate the regular expression condition (maybe convert to ^VARX... (and apply ->\s* conversion etc etc)
but this is very very hard to generalise without duplicating e.g. type data (which we already have in the schema, no need to duplicate that information in the code again) or making a very complicated description.

[jouvin]

I understand your points and I don't want to claim that the proposed solution addresses any use cases. The actual added value of what I developed, IMO, is not that this is a powerful templating system (it is clearly not!) but the coupling with a hierarchy of options through this option set concept. You could certainly mimics thing in TT template code but this would hardly be generic as you said (if I understood properly!). What is the real feature/power here is the condition expression that allows somebody to write a rule even for a completely new use case without caring about how it is implemented. You just need to pass an option hash with the proper structure, something easy to do from the pan configuration (look at ncm-dpmlfc which does this).

You are right that the Apache case you mentioned is not addressable by the current implementation. I already thought a little bit about it, I think it's doable to add the support for it but as I had no need for it until now, I delayed working on it. At some point it may become too complex and not worth the effort and the metaconfig approach would be a better compromise. Again, I certainly don't want to advocate that this rule-based editor is in any way an alternative to metaconfig. If we agree to put it in CAF, we should probably make it clear that its use should be reserved to cases where the metaconfig approach is too suboptimal.

As for the support of new parameters, you are right that if you want to be able to configure them, you need to update the configuration modules managing the files (schema and rules) but what is great in the proposed approach if that if a new configuration parameter is added by a new version of the RPM and the default value is ok for your, at least to start, you can still use the same version of the configuration module to manage the file as it will not break anything.

I didn't want to cover everything in my original post so I didn't mention how you update your config file when a new one is provided by a new version of the RPM with extension .rpmnew. You are right that the rule-based editor doesn't help directly but it does help indirectly. In your configuration module, you could easily check for example if a .rpmnew file more recent than your config file exists and decide to first overwrite your file with it and then apply the rules to customize it. This way you need no configuration module update in sync with the RPM update and this is for me a great advantage for some services (again I used it so far for 2 related services, DPM and Xrootd, and I have no plan to make a more general use currently as in most services metaconfig is a better approach).

I hope I made the intention a bit more clear! And that hopefully I convinced you that this is a feature complementary to others we have to manage files that is adding to the power of Quattor.

[jouvin]

BTW, I forgot to give an example of rule hash (this is from ncm-dpmlfc (this uses many constants not described here):

my %dpm_config_rules = (
      "DPMCONFIGFILE" => "dbconfigfile:GLOBAL;".LINE_FORMAT_PARAM,
      "RUN_DPMDAEMON" => "ALWAYS->role_enabled:dpm;".LINE_FORMAT_PARAM.";".LINE_VALUE_BOOLEAN,
      "ULIMIT_N" => "maxOpenFiles:dpm;".LINE_FORMAT_PARAM,
      "GLOBUS_THREAD_MODEL" => "globusThreadModel:dpm;".LINE_FORMAT_ENVVAR,
           );

I didn't understand the remark about rules being code rather than data. Rules are Perl hashes which for me are data (the only code used is the string concatenation)...

[jouvin]

Thinking a little bit more at the implementation details, I think that the issue is misworded and that there is no need for a new CAF module. What I propose is in fact just and extension of the FileEditor with a set of new functions to do the rule-based edition. A new module would just inherit from FileEditor and add new functions without modifying anything to the standard behaviour.

[jrha]

16.2 is unrealistic, please don't set a milestone unless you have code in progress.

[jrha]

Resolved by #151.