[RFE] Add more advanced filtering options to steve

[cmurphy]

Builds on #38427

SURE-5698

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
From Jira:

We need to be able to partial match a free text string across multiple fields

• We can only AND filters together and not OR
• See below. Both name and namespace are filtered with the free text of b . Row 1 is included even though it's name does not include b
  (missing image)

Describe the solution you'd like

Current API allows you to filter like

?filter=metadata.name=foo&metadata.namespace=foo

to get you objects that have both the name 'foo' and namespace 'foo'.

We need a way to get object that match either the name 'foo' OR namespace 'foo' not necessarily both.

Something like

?filter=metadata.name|metadata.namespace=foo

Also, be able to use the filter to exclude resources from the list, e.g. exclude all resources with a system namespace.

[richard-cox]

@cmurphy Just double checking, in Q2 we will be able to filter results on 0 or more projects or namespaces?

For instance the user might select...

• 3 Projects and 2 Namespaces
• 1 Project, 1 Namespace
• 1 Project
• 1 Namespace

In these cases a resource is returned if it appears in any of the project or namespaces provided

• NOT in 3 Namespaces

In this case a resource is returned if is not in any of the provided namespaces

[cmurphy]

@richard-cox yes, for projects+namespaces, including "NOT", I'm working on that in #40140 since it's slightly different than the general k8s resource filtering use case.

[cmurphy]

Filter syntax proposal as discussed in meeting:

?filter=metadata.name=example,metadata.namespace=example&filter=...&

Use "," as the OR separator.

Append &filter= to add another AND clause.

Consider also NOT operator ("!"), not implemented in UI to date but has been asked for by users.

[richard-cox]

@cmurphy #40141 (comment) looks good. Possible improvement at the end. I've added some common use cases and how the proposed query params would look. Only slight issue is the NOT requirement, i always forget this feature also covers namespace/project filtering where the NOT in namespace is needed

Question - Will the NOT case for namespaces/projects still be covered in 2.7 Q2

Use case 1 - List Filter by free text

• If the value exists in any of the supplied properties
• Commonly this will return rows where the value appears in any one of the supplied fields
• Example ?filter=prop1=abc,prop2=abc,prop3=abc,prop4=abc

Namespace filters using the same kind of querys....

Use case 2 - List showing namespaces resources - 1 namespace selected

• Example ?filter=metadata.namespace=fullnamespacename

Use case 3 - List showing namespaces resources - 3 namespace selected

• Example ?filter=metadata.namespace=fullnamespacename1,metadata.namespace=fullnamespacename2,metadata.namespace=fullnamespacename3

Use case 4 - List showing project resources

• Question - How would we specify the field for a project? If this is not super duper straight forward the UI can supply the underlying NS from the selected projects (it does this already locally, which includes shenanigans about 'system' resources)

Use case 5 - List Filtered by free text in a specific namespace

• Example ?filter=prop1=abc,prop2=abc,prop3=abc&filter=metadata.namespace=fullnamespacename

Request
To reduce the size of the url would it be possible to reduce duplication? If required we can pause on this until the UI implements and plays around with the feature?

• When searching a different set of properties with the same search term (see use case 1)
• When searching the same set of properties with different search term (see use case 3)
• When using a combination of them both (see use case 5)

[cmurphy]

Question - Will the NOT case for namespaces/projects still be covered in 2.7 Q2

Yes, that is covered in #40140, it will be in Q2. The filtering syntax will be different from what I'm proposing here because it is being implemented in a different component.

In last week's meeting I thought URL size wasn't as much of a concern. If we exclude the case for filtering by projects and namespaces (because that is being implemented separately and you won't have to pass 10 or 100 namespaces in to filter by a project), do you still have concerns?

To deduplicate for use case 1, that could look like the initial syntax I proposed:

?filter=metadata.name,metadata.namespace=foo

And for use case 3 it could look the opposite

?filter=metadata.name=foo,bar,baz

And for use case 5 we would also still need what I proposed in #40141 (comment)

Supporting all these different formats will add quite a bit of complexity, especially if we also have to consider NOT for all these cases. Is this flexibility a hard requirement, if we have alternative ways of filtering the projects/namespaces? The main use case that we're trying to replicate that exists today is use case 1, right?

[richard-cox]

Ok, that's good. If namespace/project filtering feature will be disconnected from filter query param then agree with all your points

• URL length isn't as important
  • we shouldn't be filtering by more than 10 fields and filter text will be limited by input field
• Optimising duplicated filter fields and terms
  • When searching a different set of properties with the same search term (see use case 1)
    • Nice to have, can be tackled later -
  • When searching the same set of properties with different search term (see use case 3)
    • Namespace/Project specific, not required
  • When using a combination of them both (see use case 5)
    • Namespace/Project specific, not required

[gaktive]

Connects to epic: rancher/dashboard#5243

[cmurphy]

Similar to #38427 we'll dev-validate this one. Integration tests have been added to rancher. Docs are added in https://github.com/rancher/steve#readme .

No release note for now since it's still not user-facing.

[richard-cox]

This can be tested as part of rancher/dashboard#9478

[richard-cox]

As part of rancher/dashboard#9478 i validated most of these. There were some additional issues identified in #42767, though from a dev validate side of things would be ok closing this