upload PM3-bench

HKU-BAL · Oct 20, 2024 · 1db702b · 1db702b
1 parent 327fce8
commit 1db702b
Show file tree

Hide file tree

Showing 4 changed files with 29 additions and 1 deletion.
diff --git a/PM3-Bench/PM3_Bench_data.json b/PM3-Bench/PM3_Bench_data.json
diff --git a/PM3-Bench/README.md b/PM3-Bench/README.md
@@ -0,0 +1,24 @@
+# PM3-Bench
+
+## Introduction
+The [ClinGen Evidence Repository](https://erepo.clinicalgenome.org/evrepo/)  provides expert-curated assertions, they are written in plain English, posing a difficult challenge for automated evaluation of benchmarks. To address this, we created PM3-Bench, a comprehensive dataset for PM3 literature evidence extraction, based on the ClinGen Evidence Repository
+
+![](../images/PM3-bench.png)
+
+---
+
+## Description
+We provide the `PM3-Bench.json` in this repo, which includes the following fields:
+| Column Name | Description | 
+|---| --- |
+| ClinGen ID | The original ID in the ClinGen Evidence Repository |
+| Variant Name | The HGVS name of the variant (DNA change) |
+| Condition | The condition reported in ClinGen |
+| Criterion | The met ACMG criteria |
+| Raw Comment |The expert-submitted comment |
+| PMID | The PubMed ID of the literature evidence  |
+| Number of Patients | Extracted number of patients based on the comments |
+| In trans Variants  | List of in trans variants extracted from the comments, augmented in all possible formats, separated by space. "NA" means no in trans variant was mentioned in the raw comments |
+| labels | `eval`: variant-publication pairs for evaluations; `others`: truncated publication XML file in NCBI API, removed in evaluation; `fine-tune`: remaining samples, where non-empty comments used for fine-tuning|
+
+
diff --git a/README.md b/README.md
@@ -23,6 +23,7 @@ We introduce AutoPM3, a method for automating the extraction of ACMG/AMP PM3 evi
 - [Usage](#usage)
     - [Quick Start](#quick-start)
     - [Advanced Usage](#advanced-usage-of-the-python-script)
+- [PM3-Bench](#pm3-bench)
 - [TODO](#todo)
 ---
 
@@ -47,7 +48,6 @@ mkdir ollama_models
 export OLLAMA_MODELS=./ollama_models
 ```
 
-3. Launch Ollama server:
 
 ```bash
 
@@ -108,5 +108,8 @@ python AutoPM3_main.py
 --model_name_text llama3_loraFT-8b-f16 ## change to llama3:70b or other hosted models as the backend of RAG as you prefer, noted that you need pull the model in Ollama in advance.
 ```
 
+## PM3-Bench
+* We released PM3-Bench used in this study, details listed in [PM3-Bench tutorial](PM3-Bench/README.md)
+
 ## TODO
 * A fast set up for AutoPM3.
diff --git a/images/PM3-bench.png b/images/PM3-bench.png