Inefficient loops negatively impacts performance

[pemsith]

What happened?

The playbook contains a task that uses a loop with the command module to annotate PersistentVolumes (PVs) individually. This approach invokes the command module separately for each item in the loop, leading to unnecessary overhead and increased execution time. Such can significantly impact performance, especially in scenarios with a large number of PVs.

Example of the inefficient task in extra_playbooks/migrate_openstack_provider.yml

• name: Rewrite the "pv.kubernetes.io/provisioned-by" annotation # noqa 301
  command: "{{ bin_dir }}/kubectl annotate --overwrite pv {{ item }} pv.kubernetes.io/provisioned-by=cinder.csi.openstack.org"
  loop: "{{ pvs.stdout_lines | list }}"

What did you expect to happen?

The task should handle annotations in a more efficient manner, reducing the number of invocations of the command module. Ideally, the PVs should be annotated in bulk or with a script that processes all PVs in one execution.

Potential Optimization
Combine the annotations into a single execution to improve performance. For example:

• name: Rewrite PV annotations (optimized)
  shell: |
  {% for pv in pvs.stdout_lines %}
  {{ bin_dir }}/kubectl annotate --overwrite pv {{ pv }} pv.kubernetes.io/provisioned-by=cinder.csi.openstack.org
  {% endfor %}

Alternatively, use a script:

Write a script to iterate over the PVs and annotate them.
Call the script with command or shell

How can we reproduce it (as minimally and precisely as possible)?

Prepare a Kubernetes Cluster
Create a Few PVs
Simulate the Playbook Task
Run the Task
Notice the task runs a separate kubectl annotate command for each PV in the loop.
Measure the execution time for environments with more PVs to observe the inefficiency.

OS

CentOS stream 9

Version of Ansible

ansible [core 2.15.8]
config file = None
configured module search path = ['/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /.local/lib/python3.9/site-packages/ansible
ansible collection location = /.ansible/collections:/usr/share/ansible/collections
executable location = /.local/bin/ansible
python version = 3.9.18 (main, Jan 4 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
jinja version = 3.1.3
libyaml = True

Version of Python

3.9.18

Version of Kubespray (commit)

86b45fc

Network plugin used

calico

Full inventory with variables

NA

Command used to invoke ansible

NA

Output of ansible run

NA

Anything else we need to know

No response

[VannTen]

This is not a bug, but I see how it could be a hit if you have lots of pv.
You don't need a shell, though, since kubectl annotate accepts multiple resources.
/remove-kind bug
/kind cleanup