Bulk Encode randomly fails and creates hundreds of duplicates of videos including failed ones.

[FosterLiu-GAT]

I am using Bulk Encode to transfer videos from a youtube channel. The videos are downloaded to the server using youtube-dl commands.
Under Bulk Encode, I typed in the path to the directory where downloaded videos are stored and I selected all then hit Add on Queue. The page will freeze up for ~ 20 min (but on the backend I could see it was running)

When it's done running, in the Queue Log, about half of them will show error and there's no detail. Re-encoding button does nothing and I can only delete them.
On the frontend, there will be hundreds of duplicates of the same video. Some shows Encoder error, some shows Encoding, and some are actually published. And there will be duplicates of those published.
The directory had ~200 videos when I hit the button, only ~100 were uploaded and I ended up with 1700+ duplicates of those videos on the site.

Error log (most recent 300ln):
log.txt

Video list to show duplicated (and failed) videos.
Channel_videos.csv

Video list in dashboard as an example.

The search result showing the dupe videos on the actual site with mix of good ones and failed ones.
https://openairmedia.com/?search=old+vs&searchFieldsNames%5B%5D=v.title&searchFieldsNames%5B%5D=v.description&searchFieldsNames%5B%5D=c.name&searchFieldsNames%5B%5D=c.description&searchFieldsNames%5B%5D=v.id&searchFieldsNames%5B%5D=v.filename&catName=&created=&minViews=0

The encoder was uptodate to at most 3 days ago. I used the latest fork of the ytb-dl and it was updated yesterday.
Any insight would be greatly appreciated.

Thank you.
Regards,

[DanielnetoDotCom]

Can you please guide how exactly you are doing the bulk encode?

[FosterLiu-GAT]

1. mkdir video storage directory and run youtube-dl -f best -ciw --write-description --output "%(title)s.%(ext)s" -v "my youtube video|channel|playlist url"
2. Like stated in the OP, In Bulk Encode, put in the path to the said directory and List Files and select all and Add on Queue
3. GUi freezes but apparently it's still running in the background.
4. Gets failed uploads, duplicated videos and what not

[DanielnetoDotCom]

It’s possible that clicking the "Add to Queue" button multiple times caused the same video to be added multiple times to the queue. This can result in duplicate processing attempts and ultimately multiple entries in the database.

Additionally, I recommend checking the behavior of yt-dlp (or youtube-dl) to ensure it is not downloading the same video more than once. Use appropriate flags such as --no-overwrites or --download-archive to avoid downloading videos that already exist on the server.

As I do not use this feature myself, I cannot provide a definitive answer, but these are some common causes of duplicate entries.

[FosterLiu-GAT]

I was very cautious and definitely didn't click the button more than once, if I did it wouldn't be hundreds of times. (if you see the csv the same video shows up in different time multiple times)

--no-overwrites were indeed used -w and when I checked the downloaded files there were no dupes, let alone hundreds of them.

And that's fair. Appreciate the input.