DBS configuration for NATS integration

[vkuznet]

No description provided.

[muhammadimranfarooqi]

@vkuznet , Valentin

Should I merge this PR?

Best
M Imran

[vkuznet]

Not yet, it is related to dmwm/DBS#617, I'm still awaiting for ok from Yuyi (@yuyiguo )

[yuyiguo]

I haven't got time to review the DBS PR, but why this PdmV related to DBS? I won't have time to review and test the DBS one for a while. @vkuznet, can you separate them?

[vkuznet]

Yuyi, I'm not sure I understood your comment. The dmwm/DBS#617 provides integration with NATS, this PR only provide how to configure DBS to use or not NATS. In this PR I explicitly disable NATS such that default DBS configuration will not use it and it serves as an example how configuration should be done.

Regarding PdmV use case, it is listed here: https://its.cern.ch/jira/browse/CMSMONIT-161, they want to monitor datasets info (e.g. change of dataset status and growths of datasets).

[yuyiguo]

@vkuznet
Following this configuration and the server info you gave me, I did not receive NATS message. I don't know why and have no time to debug this for this release. I have to move on to other things as we discussed on the phone.
Thanks,
Yuyi

[vkuznet]

Yuyi, I don't know when you performed tests, but today CERN network struggles completely, see full message https://hypernews.cern.ch/HyperNews/CMS/get/cernCompAnnounce/1393/1.html We had issues with frontends, CRAB, Monitoring, Cephfs, etc. And I don't know if CERN IT fixed the problem yet.

Said that I checked nats server logs and I saw records from dbs3-test2. Therefore DBS was talking to NATS server and I checked DBS logs on dbs3-test2 and there is no NATS errors either. Therefore it is probably a subscriber. I setup one for DBS in monitoring cluster now, therefore if your test instance is running and you'll do some dataset manipulation (e.g. change dataset status or inject new blocks/datasets), then I'll check the subscriber and report.

If you'll do any tests (apart from NATS) with dataset changes just report me a timestamp that I can check the logs.

[yuyiguo]

Thu Jan 23 19:10:43 CET 2020
I did a write unit test that has a lot of writing.

[vkuznet]

Yuyi, I think we have invalid DBS configuration (may be my mistake, I think I posted in PR the original config I was trying to use).

The config you have defines NATS parameters in view section (the last lines of config), while it should be part of dbs section. In my k8s configuration I have this

# NATS integration
config.dbs.nats_server=xxxxx
config.dbs.use_nats=True
config.dbs.nats_topics=[]

instead of parts in view section of your configuration. So the NATS configs goes to config.dbs part instead of dbconf as you have now.

Can you make this change and re-run unit tests?

[vkuznet]

Yuyi, also according to DBS logs on dbs3-test2 the NATS manager was never called. According to the code https://github.com/dmwm/DBS/blob/master/Server/Python/src/dbs/web/DBSWriterModel.py#L43-L51 we should have an entry in DBS logs. But it never happens. That's why I think we improperly configured DBS.

Since I never understood to full details DMWM configuration system I always struggle to find out proper place in it for additional stuff.

[vkuznet]

I can modify this PR if necessary with the change I mentioned in previous comments, but it would be nice if you'll prove in your environment/VM the proper configuration place for NATS attributes.

[yuyiguo]

Valentin,

I don't have time to try again today. The preprod are having problem and I don't think we have time for this release.

Thanks,
Yuyi