resolve conflicts

.gitignore

@@ -141,3 +141,19 @@ deprecated/*
 
 #backup
 backup/*
+
+#otree
+**/venv
+**/staticfiles
+**//db.sqlite3
+**/.idea
+*~
+**/*.sqlite3
+**/_static_root
+**/_bots*s
+**/__temp*
+**/__pycache__/
+**/*.py[cod]
+**/.DS_Store
+**/merge.ps1
+**/*.otreezip

docs/all_the_issues.md

@@ -3,6 +3,10 @@
 
 Large batch size may cause some episodes to be skipped. This is due to the fact that the server may not be able to handle the load. Try reducing the batch size. But you can also use the script in `examples/fix_missing_episodes.py` to fix the missing episodes.
 
+## How to serialize the data saved in the database?
+
+Check out `Episodes_to_CSV/JSON` in the `notebooks/redis_stats.ipynb` notebook.
+
 ## Where I can find the data?
 
 For the full data:

examples/fix_missing_episodes_with_tag.py

@@ -145,6 +145,7 @@ def get_combo_model_map(
     bad_gpt4_rewards_count = 0
     bad_combo_count = 0
     bad_lite_mode_count = 0
+    bad_background_count = 0
     episodes_to_delete = []
 
     # iterate through episodes
@@ -181,6 +182,15 @@ def get_combo_model_map(
         #     bad_lite_mode_count += 1
         #     continue
 
+        # check if the background is present
+        # if not "lite" in curr_ep.tag:
+        #     concat_text = "\n".join(rendered_ep[:-2])
+        #     if "'s background" not in concat_text:
+        #         print("Background not found in: ", curr_ep.pk)
+        #         episodes_to_delete.append(curr_ep.pk)
+        #         bad_background_count += 1
+        #         continue
+
         if len(interaction_list) <= 5:
             # bad_rewards_count += 1
             episodes_to_delete.append(curr_ep.pk)
@@ -211,7 +221,8 @@ def get_combo_model_map(
             # episodes_to_delete.append(curr_ep.pk)
 
             bad_combo_count += 1
-    print("Bad lite mode count: ", bad_lite_mode_count)
+    # print("Bad lite mode count: ", bad_lite_mode_count)
+    # print("Bad background count: ", bad_background_count)
     # exit(0)
     print("-" * 20 + "Deleting Bad Combos" + "-" * 20)
     for ep_pk in episodes_to_delete:

human_eval/Procfile

@@ -0,0 +1,2 @@
+web: otree prodserver1of2
+worker: otree prodserver2of2

human_eval/README.md

@@ -0,0 +1,93 @@
+# Human Evaluation for Sotopia Social Conversation
+
+To conduct highly customized human evaluation experiments for Sotopia-$\pi$, we utilize **Prolific** platform to get high-quality annotators and utilize **oTree** to build the full-stack evaluation system.
+
+## File Structure
+
+The overall structure of this provided human evaluation system includes two parts: `pilot_study` and `official_study`. For `pilot_study`, it is used to select qualified annotators from the prolific. For `official_study`, it is used to get the final human evaluation results based on qualified annotators.
+
+The important files for the otree project is described as:
+
+```
+human_eval/
+├─ official_study_payment_info/ # for annotator payment checking page
+│   ─ __init__.py # payment information
+│   ─ PaymentInfo.html # payment HTML page
+├─ pilot_study_payment_info/ # for annotator payment checking page
+│   ─ __init__.py # payment information
+│   ─ PaymentInfo.html # payment HTML page
+├─ sotopia_official_study/ # for annotator instruction and answering pages
+│   ─ __init__.py # data point distribution logic and data processing logic
+│   ─ SotopiaEval.html # annotator filling answer page
+│   ─ SotopiaEvalInstruction.html # annotator task instruction page
+├─ sotopia_pilot_study/ # for annotator instruction and answering pages
+    ─ __init__.py # data point distribution logic and data processing logic
+    ─ SotopiaEval.html # annotator filling answer page
+    ─ SotopiaEvalInstruction.html # annotator task instruction page
+...
+```
+
+## Local Machine Development
+
+We can locally launch our otree project based on the following command:
+
+```bash
+pip install -r requirements.txt
+otree devserver
+```
+
+After this operation, we can visit the website via https://localhost:8000 for debugging.
+
+## Project Deployment
+
+#### Step1. Create Apps on Heruko
+
+**Step1.1**
+
+First, we need to create a Heruko apps as our base app to deploy.
+
+**Step1.2**
+
+Secondly, we need to link our apps with Heroku Postgres.
+
+![heruko_db](figs/heruko_db.png)
+
+**Step1.3**
+
+Thirdly, we need to change the environment config var under setting to support release of our evaluation task including avoiding debugging information and admin login.
+
+![heruko_env_config](figs/heruko_env_config.png)
+
+### Step2. Deploy oTree project via oTree Hub
+
+**Step2.1**
+
+After finishing modifying the otree project, we can run `otree zip` to have a `otree_project.otreezip` for deployment.
+
+**Step2.2**
+
+Secondly, this zip file is used to deploy on oTree Hub. Each oTree Hub should be linked with one or more Heruko apps so that we can directly deploy based on that. (https://www.otreehub.com/my_projects/). More instructions about how to use otree-hub can be found at https://otree.readthedocs.io/en/latest/server/heroku.html.
+
+![otree_hub](figs/otree_hub.png)
+
+**Step2.3**
+
+Thirdly, after deploying on oTree Hub, we need to reset our database to make sure our database is empty (which is optional but recommended).
+
+**Step2.4**
+
+Finally, we can get a deployment link similar with https://sotopia-eval-8cd8c447c673.herokuapp.com/demo that is deployed on Heruko server.
+
+### Step3. Release on Prolific
+
+**Step3.1**
+
+To release on Prolific, we need to get a release link that annotators can directly access to. To get this release link, we need to click into the deployment link and create new sessions with sufficient participants. Therefore, we can get a session-wide link that is used for annotators like https://sotopia-eval-8cd8c447c673.herokuapp.com/join/tikanoma.
+
+![release_link](figs/release_link.png)
+
+**Step3.2**
+
+Secondly, we just put the release link into the Prolific project setting.
+
+![prolific_release](figs/prolific_release.png)

human_eval/_templates/global/Page.html

@@ -0,0 +1,2 @@
+{{ extends "otree/Page.html" }} {{ load otree }} {{ block global_styles }} {{
+endblock }} {{ block global_scripts }} {{ endblock }}

human_eval/official_study_payment_info/PaymentInfo.html

@@ -0,0 +1,41 @@
+{{ block title }}Thank you{{ endblock }} {{ block content }}
+
+<div class="panel panel-default" style="margin-bottom: 10px">
+  <div class="panel-body">
+    <p><b>For prolific annotators</b></p>
+    <p>
+      <strong
+        >If you are directly guided to this page without annotation, it
+        indicates that there is no left data for annotation now.</strong
+      >
+    </p>
+    <p>
+      <strong
+        >You could join the annotation multiple times and we would assign
+        different data points for you automatically.</strong
+      >
+    </p>
+    <p>
+      Thank you a lot for participating the official test for the social
+      evaluation test.
+    </p>
+    <p>
+      Please redirect to
+      <a href="https://app.prolific.com/submissions/complete?cc=xxxxxxxx"
+        >here</a
+      >
+      to get paid
+    </p>
+    <p>
+      Alternatively, you can use <strong>xxxxxxxx</strong> as your code to get
+      money.
+    </p>
+    <p>
+      Each annotator would be able to get paid after we approved all the
+      annotation results in a few hours after the submissions.
+    </p>
+    <p>Please leave me a message if you have any questions.</p>
+  </div>
+</div>
+
+{{ endblock }}

human_eval/official_study_payment_info/Procfile

@@ -0,0 +1,2 @@
+web: otree prodserver1of2
+worker: otree prodserver2of2

human_eval/official_study_payment_info/__init__.py

@@ -0,0 +1,44 @@
+from typing import Dict
+
+from otree.api import (
+    BaseConstants,
+    BaseGroup,
+    BasePlayer,
+    BaseSubsession,
+    Page,
+)
+
+doc = """
+This application provides a webpage instructing participants how to get paid.
+Examples are given for the lab and Amazon Mechanical Turk (AMT).
+"""
+
+
+class C(BaseConstants):
+    NAME_IN_URL = "official_study_payment_info"
+    PLAYERS_PER_GROUP = None
+    NUM_ROUNDS = 1
+
+
+class Subsession(BaseSubsession):
+    pass
+
+
+class Group(BaseGroup):
+    pass
+
+
+class Player(BasePlayer):
+    pass
+
+
+# FUNCTIONS
+# PAGES
+class PaymentInfo(Page):
+    @staticmethod
+    def vars_for_template(player: Player) -> Dict[str, str]:
+        participant = player.participant
+        return dict(redemption_code=participant.label or participant.code)
+
+
+page_sequence = [PaymentInfo]

human_eval/pilot_study_payment_info/PaymentInfo.html

@@ -0,0 +1,33 @@
+{{ block title }}Thank you{{ endblock }} {{ block content }}
+
+<div class="panel panel-default" style="margin-bottom: 10px">
+  <div class="panel-body">
+    <p><b>For prolific annotators</b></p>
+    <p>
+      Thank you a lot for participating the qualification test for the social
+      evaluation test.
+    </p>
+    <p>
+      We would verify your results and invite you to continue participating in
+      our official test later.
+    </p>
+    <p>
+      Please redirect to
+      <a href="https://app.prolific.com/submissions/complete?cc=xxxxxxxx"
+        >here</a
+      >
+      to get paid
+    </p>
+    <p>
+      Alternatively, you can use <strong>xxxxxxxx</strong> as your code to get
+      money.
+    </p>
+    <p>
+      Each annotator would be able to get paid after we approved all the
+      annotation results in a few hours after the submissions.
+    </p>
+    <p>Please leave me a message if you have any questions.</p>
+  </div>
+</div>
+
+{{ endblock }}

human_eval/pilot_study_payment_info/Procfile

@@ -0,0 +1,2 @@
+web: otree prodserver1of2
+worker: otree prodserver2of2

human_eval/pilot_study_payment_info/__init__.py

@@ -0,0 +1,46 @@
+from typing import Dict
+
+from otree.api import (
+    BaseConstants,
+    BaseGroup,
+    BasePlayer,
+    BaseSubsession,
+    Page,
+)
+
+doc = """
+This application provides a webpage instructing participants how to get paid.
+Examples are given for the lab and Amazon Mechanical Turk (AMT).
+"""
+
+
+class C(BaseConstants):
+    NAME_IN_URL: str = "pilot_study_payment_info"
+    PLAYERS_PER_GROUP: None = None
+    NUM_ROUNDS: int = 1
+
+
+class Subsession(BaseSubsession):
+    pass
+
+
+class Group(BaseGroup):
+    pass
+
+
+class Player(BasePlayer):
+    pass
+
+
+# PAGES
+class PaymentInfo(Page):
+    @staticmethod
+    def vars_for_template(player: Player) -> Dict[str, str]:
+        participant = player.participant
+        redemption_code: str = (
+            participant.label or participant.code
+        )  # Assuming both label and code are strings.
+        return dict(redemption_code=redemption_code)
+
+
+page_sequence = [PaymentInfo]

human_eval/requirements.txt

@@ -0,0 +1,6 @@
+# oTree-may-overwrite-this-file
+# IF YOU MODIFY THIS FILE, remove these comments.
+# otherwise, oTree will automatically overwrite it.
+otree==5.10.4
+psycopg2>=2.8.4
+sentry-sdk>=0.7.9