Manual validation of AI-based identifications

[DavidFunosas]

For the manual validation of AI-based (e.g. BirdNET) identifications, it would be very useful to be be able to follow the following procedure:

- Screen 1: Choosing a specific confidence threshold with possible values from 0.1 to 0.99

- Screen 2: Choosing, from a list, the species whose identifications we want to validate. Ideally, the list would contain all species with at least 1 BirdNET identification with conf_score >= conf_threshold yet to validate. It would be helpful as well to display the number of BirdNET identifications yet to validate next to each species name.

- Screen 3: Once we've chosen the species we want to work on, the platform should let us visualize the spectrogram containing the first identification to validate. The spectrogram on display should contain at least the whole audio chunk with the ID to validate plus a 2-second margin on both left and right. When clicking on the Play button, the recording should start playing 2 s prior to the start of the ID to validate, and it should be possible to move backward and forward within the recording in case it's necessary to access prior/posterior vocalizations of the same species to confirm the ID. 
	
	· Since the spectrogram on display would contain some sounds before and after the BirdNET prediction segment, it would be helpful to be able to visualize the beginning and the end of the ID we're validating (e.g. with vertical red bars at the start and end times, or with a horizontal label spanning across the duration of the ID).

	· Usually, the ID to validate will last 3 seconds (corresponding to the BirdNET prediction segment where the species was identified), but it's possible to have multiple overlapping IDs of the same species one after the other, especially if we choose a non-zero value for the overlap input parameter. It would be great if these IDs were merged together so that the validation process could be swifter (e.g. Alauda arvensis 0s-3s, Alauda arvensis 1s-4s, Alauda arvensis 4s-7s ---> Alauda arvensis 0s-7s).

	· Somewhere on the screen there should be 2 buttons: 1 to confirm and 1 to invalidate the current ID. Once we click on either of these, we should be redirected to the next ID to validate. The IDs to validate should be sorted by the following criteria: 1. Recording site, 2. Recording date, 3. Recording time, 4. Start time of the ID within the recording.

	· It would also be great to be able to visualize the following information on top of the spectrogram (or somewhere else on the screen): recording site, recording date and recording time.

	· Once we've finished validating all the remaining BirdNET IDs for the species currently selected, we should be redirected to Screen 2 to be offered the option to select another species to validate.

- It's not easy to estimate how much time will be needed for the manual validation of all BirdNET identifications over a given confidence threshold. If we initially estimate that we'll be able to validate all identifications with a confidence score >= 0.9 but later on we decide to lower the threshold to 0.8 in order to get a more comprehensive picture of the composition of the bird community, it would be great if the platform remembers all the identifications already validated (>= 0.9) so that we only have to validate those with confidence scores between the old and the new thresholds (0.9 >= x >= 0.8).

- Finally, it would also be useful to be able to export a CSV file with all validated and invalidated IDs, in the form of TP and FP, for each species and audio segment.

Of course I'm aware that it might not be possible to implement all these features, but this should give you an idea of what we have in mind for the validation module.

Thanks in advance!

[kdarras]

Hi David, sorry I don't get notifications on new issues, so I just found this one. What I comment below won't be entirely satisfying for you yet, but it stems from my continual efforts to offer generic, and basic functionalities instead of specialised workflows. Let us discuss and mature the ideas further.

• Screen 1: Choosing a specific confidence threshold with possible values from 0.1 to 0.99

Answer: corresponding issue: #26

• Screen 2: Choosing, from a list, the species whose identifications we want to validate. Ideally, the list would contain all species with at least 1 BirdNET identification with conf_score >= conf_threshold yet to validate. It would be helpful as well to display the number of BirdNET identifications yet to validate next to each species name.

Answer: corresponding issue: #26 (but I added the display of the numbers of tags)

• Screen 3: Once we've chosen the species we want to work on, the platform should let us visualize the spectrogram containing the first identification to validate. The spectrogram on display should contain at least the whole audio chunk with the ID to validate plus a 2-second margin on both left and right. When clicking on the Play button, the recording should start playing 2 s prior to the start of the ID to validate, and it should be possible to move backward and forward within the recording in case it's necessary to access prior/posterior vocalizations of the same species to confirm the ID.

Answer: This is already possible by zooming on the first tag, but it needs to additionally implement #380 , already planned in your development round

· Since the spectrogram on display would contain some sounds before and after the BirdNET prediction segment, it would be helpful to be able to visualize the beginning and the end of the ID we're validating (e.g. with vertical red bars at the start and end times, or with a horizontal label spanning across the duration of the ID).

Answer: BirdNET annotations are tags that are displayed by default, so these bars would be visible

· Usually, the ID to validate will last 3 seconds (corresponding to the BirdNET prediction segment where the species was identified), but it's possible to have multiple overlapping IDs of the same species one after the other, especially if we choose a non-zero value for the overlap input parameter. It would be great if these IDs were merged together so that the validation process could be swifter (e.g. Alauda arvensis 0s-3s, Alauda arvensis 1s-4s, Alauda arvensis 4s-7s ---> Alauda arvensis 0s-7s).

Answer: planned in #21

· Somewhere on the screen there should be 2 buttons: 1 to confirm and 1 to invalidate the current ID. Once we click on either of these, we should be redirected to the next ID to validate. The IDs to validate should be sorted by the following criteria: 1. Recording site, 2. Recording date, 3. Recording time, 4. Start time of the ID within the recording.

Answer: #516 will be rolled out soon

· It would also be great to be able to visualize the following information on top of the spectrogram (or somewhere else on the screen): recording site, recording date and recording time.

Answer: good point. Created #561 !

· Once we've finished validating all the remaining BirdNET IDs for the species currently selected, we should be redirected to Screen 2 to be offered the option to select another species to validate.

Answer: first, let's implement the basic functionalities. Then, the longer-term plan is to let users assemble their own workflows (not yet drafted as an issue), so that anyone can do anything they want with all the available functionalities. In the meantime, maybe #535 would help to execute this task in a batch in a more generic manner.

• It's not easy to estimate how much time will be needed for the manual validation of all BirdNET identifications over a given confidence threshold. If we initially estimate that we'll be able to validate all identifications with a confidence score >= 0.9 but later on we decide to lower the threshold to 0.8 in order to get a more comprehensive picture of the composition of the bird community, it would be great if the platform remembers all the identifications already validated (>= 0.9) so that we only have to validate those with confidence scores between the old and the new thresholds (0.9 >= x >= 0.8).

Answer: yes, we should be able to simply filter by a range of thresholds. adapted #26

• Finally, it would also be useful to be able to export a CSV file with all validated and invalidated IDs, in the form of TP and FP, for each species and audio segment.

Answer: Currently, you should download the "tags" and "reviews" tables and merge them on your own

[kdarras]

After wrapping my head around this: essentially, this will be possible after the refactoring and implementing #535 + a global user setting for padding the zoom around tags.