Model directory (#172)

* refactor: Handle file exceptions when writing raw sentence and translations

This commit modifies the code in transcription-filter-callbacks.cpp to handle file exceptions when writing raw sentence and translations to files. It adds exception handling using try-catch blocks to ensure that file operations are properly handled. This change improves the robustness of the code and prevents crashes or unexpected behavior when file operations fail.

* refactor: Update models_info function to use cached models information

The models_info function in model-downloader.cpp has been updated to use a cached version of the models information. This improves performance by avoiding unnecessary file reads and JSON parsing. The function now returns a const reference to the cached models_info map. This change ensures that the models_info function is more efficient and reduces the overhead of loading the models information.

Refactor the code in model-downloader.cpp to use the updated models_info function and remove the unnecessary file read and JSON parsing code.

Closes #123

* refactor: Simplify file handling in transcription-filter-callbacks.cpp

* refactor: Add script to query Hugging Face models and update models_directory.json

This commit adds two new scripts, hugging_face_model_query.py and hugging_face_model_query_all.py, to query Hugging Face models and update the models_directory.json file. The hugging_face_model_query.py script fetches model information from the Hugging Face API and adds new models to the models_directory.json file. The hugging_face_model_query_all.py script fetches a list of models matching a specific search criteria and adds the matching models to the models_directory.json file. These scripts will help keep the models_directory.json file up to date with the latest models available on Hugging Face.

Refactor the file handling in transcription-filter-callbacks.cpp

This commit simplifies the file handling in the transcription-filter-callbacks.cpp file. The changes aim to improve the readability and maintainability of the code by reducing complexity and removing unnecessary code.

Update the models_info function to use cached models information

This commit updates the models_info function to use cached models information instead of fetching it every time the function is called. This change improves the performance of the function by reducing the number of API calls and improves the overall efficiency of the code.

Handle file exceptions when writing raw sentence and translations

This commit adds exception handling code to handle file exceptions when writing raw sentence and translations. The changes ensure that any file-related exceptions are caught and properly handled, preventing the program from crashing or producing incorrect results.

Simplify the Onnxruntime installation in FetchOnnxruntime.cmake

This commit simplifies the Onnxruntime installation process in the FetchOnnxruntime.cmake file. The changes aim to make the installation steps more concise and easier to understand, improving the overall maintainability of the code.

Update the version to 0.3.6 and adjust the website URL

This commit updates the version of the software to 0.3.6 and adjusts the website URL accordingly. The changes ensure that the software is properly versioned and the website URL is up to date.

* refactor: Add ExtraInfo struct to ModelInfo and update models_info function

* refactor: Update model names in models_directory.json and fix URL in transcription-filter.h

src/model-utils/model-downloader-types.h

@@ -16,13 +16,21 @@ struct ModelFileDownloadInfo {
 
 enum ModelType { MODEL_TYPE_TRANSCRIPTION, MODEL_TYPE_TRANSLATION };
 
+struct ExtraInfo {
+	std::string language;
+	std::string description;
+	std::string source;
+};
+
 struct ModelInfo {
 	std::string friendly_name;
 	std::string local_folder_name;
 	ModelType type;
 	std::vector<ModelFileDownloadInfo> files;
+	ExtraInfo extra;
 };
 
-extern std::map<std::string, ModelInfo> models_info;
+extern const std::map<std::string, ModelInfo> &models_info();
+extern const std::vector<ModelInfo> get_sorted_models_info();
 
 #endif /* MODEL_DOWNLOADER_TYPES_H */

src/model-utils/model-downloader.cpp

@@ -8,9 +8,16 @@
 
 std::string find_model_folder(const ModelInfo &model_info)
 {
-	if (model_info.friendly_name.empty() || model_info.local_folder_name.empty() ||
-	    model_info.files.empty()) {
-		obs_log(LOG_ERROR, "Model info is invalid.");
+	if (model_info.friendly_name.empty()) {
+		obs_log(LOG_ERROR, "Model info is invalid. Friendly name is empty.");
+		return "";
+	}
+	if (model_info.local_folder_name.empty()) {
+		obs_log(LOG_ERROR, "Model info is invalid. Local folder name is empty.");
+		return "";
+	}
+	if (model_info.files.empty()) {
+		obs_log(LOG_ERROR, "Model info is invalid. Files list is empty.");
 		return "";
 	}
 

src/tests/CMakeLists.txt

@@ -7,7 +7,6 @@ target_sources(
   PRIVATE ${CMAKE_SOURCE_DIR}/src/tests/localvocal-offline-test.cpp
           ${CMAKE_SOURCE_DIR}/src/tests/audio-file-utils.cpp
           ${CMAKE_SOURCE_DIR}/src/transcription-utils.cpp
-          ${CMAKE_SOURCE_DIR}/src/model-utils/model-infos.cpp
           ${CMAKE_SOURCE_DIR}/src/model-utils/model-find-utils.cpp
           ${CMAKE_SOURCE_DIR}/src/whisper-utils/whisper-processing.cpp
           ${CMAKE_SOURCE_DIR}/src/whisper-utils/whisper-utils.cpp

src/tests/localvocal-offline-test.cpp

@@ -21,6 +21,7 @@
 #include "audio-file-utils.h"
 #include "translation/language_codes.h"
 #include "ui/filter-replace-utils.h"
+#include "model-utils/model-downloader-types.h"
 
 #include <stdio.h>
 #include <stdlib.h>
@@ -84,6 +85,14 @@ void obs_log(int log_level, const char *format, ...)
 	printf("\n");
 }
 
+const std::map<std::string, ModelInfo> &models_info()
+{
+	static const std::unique_ptr<const std::map<std::string, ModelInfo>> cached_models_info =
+		std::make_unique<const std::map<std::string, ModelInfo>>();
+
+	return *cached_models_info;
+}
+
 transcription_filter_data *
 create_context(int sample_rate, int channels, const std::string &whisper_model_path,
 	       const std::string &silero_vad_model_file, const std::string &ct2ModelFolder,

src/transcription-filter-callbacks.cpp

@@ -112,13 +112,18 @@ void send_sentence_to_file(struct transcription_filter_data *gf,
 	}
 	if (!gf->save_srt) {
 		// Write raw sentence to file
-		std::ofstream output_file(gf->output_file_path, openmode);
-		output_file << str_copy << std::endl;
-		output_file.close();
-		if (write_translations) {
-			std::ofstream translated_output_file(translated_file_path, openmode);
-			translated_output_file << translated_sentence << std::endl;
-			translated_output_file.close();
+		try {
+			std::ofstream output_file(gf->output_file_path, openmode);
+			output_file << str_copy << std::endl;
+			output_file.close();
+			if (write_translations) {
+				std::ofstream translated_output_file(translated_file_path,
+								     openmode);
+				translated_output_file << translated_sentence << std::endl;
+				translated_output_file.close();
+			}
+		} catch (const std::ofstream::failure &e) {
+			obs_log(LOG_ERROR, "Exception opening/writing/closing file: %s", e.what());
 		}
 	} else {
 		if (result.start_timestamp_ms == 0 && result.end_timestamp_ms == 0) {
@@ -297,12 +302,15 @@ void recording_state_callback(enum obs_frontend_event event, void *data)
 	struct transcription_filter_data *gf_ =
 		static_cast<struct transcription_filter_data *>(data);
 	if (event == OBS_FRONTEND_EVENT_RECORDING_STARTING) {
-		if (gf_->save_srt && gf_->save_only_while_recording) {
+		if (gf_->save_srt && gf_->save_only_while_recording &&
+		    gf_->output_file_path != "") {
 			obs_log(gf_->log_level, "Recording started. Resetting srt file.");
 			// truncate file if it exists
-			std::ofstream output_file(gf_->output_file_path,
-						  std::ios::out | std::ios::trunc);
-			output_file.close();
+			if (std::ifstream(gf_->output_file_path)) {
+				std::ofstream output_file(gf_->output_file_path,
+							  std::ios::out | std::ios::trunc);
+				output_file.close();
+			}
 			gf_->sentence_number = 1;
 			gf_->start_timestamp_ms = now_ms();
 		}

src/transcription-filter-properties.cpp

@@ -152,15 +152,16 @@ void add_transcription_group_properties(obs_properties_t *ppts,
 	obs_property_t *whisper_models_list = obs_properties_add_list(
 		transcription_group, "whisper_model_path", MT_("whisper_model"),
 		OBS_COMBO_TYPE_LIST, OBS_COMBO_FORMAT_STRING);
+	obs_property_list_add_string(whisper_models_list, "Load external model file",
+				     "!!!external!!!");
 	// Add models from models_info map
-	for (const auto &model_info : models_info) {
-		if (model_info.second.type == MODEL_TYPE_TRANSCRIPTION) {
-			obs_property_list_add_string(whisper_models_list, model_info.first.c_str(),
-						     model_info.first.c_str());
+	for (const auto &model_info : get_sorted_models_info()) {
+		if (model_info.type == MODEL_TYPE_TRANSCRIPTION) {
+			obs_property_list_add_string(whisper_models_list,
+						     model_info.friendly_name.c_str(),
+						     model_info.friendly_name.c_str());
 		}
 	}
-	obs_property_list_add_string(whisper_models_list, "Load external model file",
-				     "!!!external!!!");
 
 	// Add a file selection input to select an external model file
 	obs_properties_add_path(transcription_group, "whisper_model_path_external",
@@ -191,7 +192,7 @@ void add_translation_group_properties(obs_properties_t *ppts)
 	// add "Whisper-Based Translation" option
 	obs_property_list_add_string(prop_translate_model, MT_("Whisper-Based-Translation"),
 				     "whisper-based-translation");
-	for (const auto &model_info : models_info) {
+	for (const auto &model_info : models_info()) {
 		if (model_info.second.type == MODEL_TYPE_TRANSLATION) {
 			obs_property_list_add_string(prop_translate_model, model_info.first.c_str(),
 						     model_info.first.c_str());

src/transcription-filter.h

@@ -20,9 +20,9 @@ void transcription_filter_show(void *data);
 void transcription_filter_hide(void *data);
 
 const char *const PLUGIN_INFO_TEMPLATE =
-	"<a href=\"https://github.com/occ-ai/obs-localvocal/\">LocalVocal</a> (%1) by "
-	"<a href=\"https://github.com/occ-ai\">OCC AI</a> ❤️ "
-	"<a href=\"https://www.patreon.com/RoyShilkrot\">Support & Follow</a>";
+	"<a href=\"https://github.com/locaal-ai/obs-localvocal/\">LocalVocal</a> (%1) by "
+	"<a href=\"https://github.com/locaal-ai\">Locaal AI</a> ❤️ "
+	"<a href=\"https://locaal.ai\">Support & Follow</a>";
 
 const char *const SUPPRESS_SENTENCES_DEFAULT =
 	"Thank you for watching\nPlease like and subscribe\n"

src/translation/translation-utils.cpp

@@ -22,7 +22,7 @@ void start_translation(struct transcription_filter_data *gf)
 		return;
 	}
 
-	const ModelInfo &translation_model_info = models_info[gf->translation_model_index];
+	const ModelInfo &translation_model_info = models_info().at(gf->translation_model_index);
 	std::string model_file_found = find_model_folder(translation_model_info);
 	if (model_file_found == "") {
 		obs_log(LOG_INFO, "Translation CT2 model does not exist. Downloading...");

src/whisper-utils/whisper-model-utils.cpp

@@ -72,13 +72,13 @@ void update_whisper_model(struct transcription_filter_data *gf)
 			// new model is not external file
 			shutdown_whisper_thread(gf);
 
-			if (models_info.count(new_model_path) == 0) {
+			if (models_info().count(new_model_path) == 0) {
 				obs_log(LOG_WARNING, "Model '%s' does not exist",
 					new_model_path.c_str());
 				return;
 			}
 
-			const ModelInfo &model_info = models_info[new_model_path];
+			const ModelInfo &model_info = models_info().at(new_model_path);
 
 			// check if the model exists, if not, download it
 			std::string model_file_found = find_model_bin_file(model_info);