-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changing from a plugin to a standalone CLI application #68
Conversation
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left comments regarding pipeline names in the scripts.
I tested calculating judgments with Chorus and got a bunch of errors connected to the JSON structure and fields of the UBI events and queries. I did some local changes and fixed the findings but I think it makes sense to look at them together to make sure we make changes in the right place (here vs Chorus vs the specification vs the UBI JS client).
Other findings:
- Running the query set generator with a
querySetSize
of 5 resulted in 6 queries in the query set. - Running a query set reported a successful run (
Query set run complete: ...
). However there were no calculated metrics stored.
Let's have a look together.
Found and fixed this one. |
I got the metrics indexing now. I had that part removed for a bit. (doh) |
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
Signed-off-by: jzonthemtn <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
I had to make a few changes to the data model and with the last run I only saw judgments of value 0. Might be data related (the calculation is correct, the event data I have is just rubbish), might be something in the code (the calculation should give us something else for the event data). I'll have a closer look in the next review.
src/main/java/org/opensearch/eval/model/ubi/event/EventObject.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/eval/model/ubi/query/UbiQuery.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One last request, a renaming request. Other than that: My brief test worked. I tested functionally (=does everything compile and run without errors?)
I got all 0s for all judgments (and as a consequence also for the search metrics) but again, that's something that may well be data-specific that has nothing to do with the implementation.
I'd do these test outside of this PR that really is meant to move from plugin to standalone app.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Changing from a plugin to a standalone CLI application.