From 9351c5e3646187e8d142a54c4c55cd455ffed76e Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Fri, 7 Jun 2024 11:50:24 -0400 Subject: [PATCH] Tutorial notebooks that can be directly opened in colab (#90) * add jupyter notebooks that can be directly opened on colab * update the links in the readme * Prompt to rerun task sampling when no episodes are retrieved --- README.md | 4 +- docs/public/favicon.png | Bin 0 -> 4563 bytes docs/public/favicon.svg | 19 +- notebooks/tutorials/1.1-setup.ipynb | 354 ++++++++++++++++++++++ notebooks/tutorials/1.2-browse-data.ipynb | 334 ++++++++++++++++++++ 5 files changed, 696 insertions(+), 15 deletions(-) create mode 100644 docs/public/favicon.png create mode 100644 notebooks/tutorials/1.1-setup.ipynb create mode 100644 notebooks/tutorials/1.2-browse-data.ipynb diff --git a/README.md b/README.md index 528a8b85c..fb27cfc0a 100644 --- a/README.md +++ b/README.md @@ -46,8 +46,8 @@ If you want to try it out on Google Colab first, please check out our Colab Tuto
  1. Basic
      -
    1. 1.1 Playing with Sotopia in a Google Colab Notebook
      2. -
    2. 1.2 Browsing sotopia data
      3. +
    3. 1.1 Playing with Sotopia in a Google Colab Notebook
      4. +
    4. 1.2 Browsing sotopia data
  2. diff --git a/docs/public/favicon.png b/docs/public/favicon.png new file mode 100644 index 0000000000000000000000000000000000000000..ef6b7c1af91506e6f374f9445914f5a48e27e39f GIT binary patch literal 4563 zcmX9?c~p|y*9LP+aY#!Gr?8ve($G>6zAy*m{S^{CJLE|CWhhk(eL}?oU@+2_BrcaXP>>FXTO;jFCdi^zzP5WKs3&&-!BJ0s4xM*#qeYQG2Y6YK5<0I)0ltONW?ToIq;9QnmjLw2do z)9b?1V%CC#ATVf7I>TA@mvn`*>fG;1ES+&d^_w)%*(cJHltUUnT~#o9Wz|>=Pw{t= zItviQh(xbldTCZa3$K-WZ6n&_1-oEe6wpUu$^vV@wY-~Z^z!k)i8GYRTbgj3HP--O z@MGxts@)!yZdnizi3WPM8sH&sM( zkpFoQtH`=_pBN%Tq+;@6G1Iosl!#Gb3O)vU+9|p5YV*}eyG_k8=6j;K%#gHN8#x@u zzsS7&`Kqt}@Or*HQ922#7r1@npkJluseiAxHm9{Mt!ux62T`xy{-AjObV!!2Q zaU^B2$O8);i+Jsc4XCNLLD&y<52ivB*#{dFY6EIKu@*Jylm4+Wj*3K3>N!BmgVmr< zUZ4|;u026N9{D$ZaokB`oCX1jSf>$g23ZtJ}lm-p6ktEpi!C*kSW4U|;#DSyL}u2BzUEbZx3?B)Kj_li9A z9e0boQfN8I>FXq{t?5*ybs|rKW()FG`sPolv~@uGd`ZTOVXZl({1;`U|8ln$C}6NXpOReT*>QAj>hd;Tm)LNDd`bHp zUyArO)w$jJ77v=d>*TUV-b*yl(~3JM-yWTj1n5Q!W#Z5_29~3O@Vs4qJ8LUcUae?* z>RN2x_#9akB%p|%qIcnzmS>lv6^NkmhAHnePOj{Ad(wT3m8PEeSXtuEjFZO|`BFb0 zrJ9UQ{VlF4@qCb5qkNb9u>WBvgk;=^kwe%LgK&$qrXy>b>ROjgE^)pc$2cpqX8qL= zx_Nb7_O`%quuE)xc@z+O&H|hHG>U&ruZ~&=iE$nn{1utL_8%xcDZ^BcJC*}Xgqyh|+> z#}IBHYnvc%q<5+Z3=Jv{==09bLp0Av ze7Cp`B8D2+L$ktBzS-B*h1IwAyw84`1*ha1={FP(`#?A0@y z?T>qzV?xa_oR+b=-M~>mORjOry$LkmdmeC!a?{M$n<9$h>tC0|zsk;}bx*9KZM@(- zUmyIoj}}{Yki#1@xLkKDt&BK-IXJWi_W2N`sQkM-1T}EmfRiGUG3WtKjZUmJ2ix8I zs3K5w(!?#!yaJ-GCz%vnOMy76!NhT4K2!6Rh!=(1HJ)-xGvf`_pT=(iTH*yJH5sUD z6BFenZ^~6d(ixo%6X_?@qgI4>@U{=;GF(^JM_~o`AkJV|!gej-zlSPAdJ(kH)-2@X zEzEcUn)3Gn^;=w_&L_A%iJmXKkHaU8a=lg zf46`)H@&_D-A;Q^~;+3p7JT(=)` za@uw{1bzGY%1t>;dc>h=4N+@ggTegxUic(LI$mi?1H$lt7VLV+WfJN*D@W{e?ya9P z`%kuag!bfFp1h9e%{FkYZljT+xv2-(cH}Vt09sk*!uO~VO4p`k`Z?P=AoEAzy(6d? zIz9hO$be3J78h7|<6QR$ZW{B*g{6Opv50QSu>_)MlXFS{mUJ8s^!Nn>cQaN~tg3Kt8w zE4P#v6~-^}=zH2FEI&sAfZfi)!xI&!+yIlsQ64)t9UA_)Y&Fp9bPgec79`HNw7lCq z`{NbI?!$M&GDk!BQ7et=9isXCp2zL@UTfY%okh}yYr~UeEAFWLW$nIGucCxEP=K`u z_4nQS>W|Bjt;`O9#0Wj$#D&0=bE zvP}KT(%zJpa-~GCh3d1{FMJq=Vl%o4q8Xhw+YS9T4_E6XIv|x@M+c13n$Gj>iI0b{ zmJ>0I`n|#}2?0s+T4yk#q+Qf;=V;h9i?(}OEAX$+Q4^mBajObc!P{#=n8jv>)mlaxUGJlMUW;Jest&SYsyo|@_B!I;~sL{by-^!Uh72flXZ2)2)RoQWa< z=JjC2<~nxb<>{|6jBmb-&_?*)zQQn36@CVw6b ztp@Ja(D=qFq=8ZS=cQo8BXz1LNMfx(fAGp>^s@{41}oB-957uxK`o5!{(umP1t-^6kvaM z^!T{a4aST}LlzoyJH7bzvg{pR30|bZp6?rpE8HuLUH~{9K_NTqzqFSno`CxJ$g40z zDSuJbqXX;9E4e}Zt7{p?YX8W`eto2m77Fv@%gRMtqlw8o*)HXHxHBCVVv>=Q_}va` z^y8LQwDS-x(QXj>^{D#`qt!T za6nA3PwwRM;xlipQjDyDjR5YF+?#UlO9s?OK08B*dO4X2r=KtLq~`pw zB!d+JYM=BUO-G2xRW0vAooG!v+d|3%dy=lv2|-kpoF_l%3npJujD>Re}(0 z{Occ3pqz7>eEwHl<#i`#NyRbcA#OLXpD6jo7ms=^X!p@hsOD%!OVv;`nGL$cF`L(?J04u4V5Og2j zaM}@2cPF{4EQe0P9PnYR2y;bSt|a;S#la6Ob-=dN5`EMzr1|W|4r;&N7RHn}?u%@w zwjhVciViKRSQnoh+lKsdFfT$ztr{@YLFB)>3F=Cxlqbx;gsK0!=z$4hWF{~sTtaLM zPn>BdsOwb67=Oe*!u;S$LNZ-^%P z{9CeGbD_H&^|UKQl8vDfnYapL29W-h&VH>tmyQOSR|QBzN)MniT60wHOB}+C3)&&)WN;v_BFG2We5Kqm{_0w{j>VwmPvhRuca zpJZ)NJZKRr6Tp~R=}l}IHKtm-r9iUbBO6$FcE9G1tZ>Dp4iC^RfekDnt(GiO9FXGt zaI`(Ii+j7N4(;lk?n|^AGR9^XIC8l^cW3LcY!xBli?KI)X6oa~_;$jV3f;(7%ZiR! z9BIQkx}o2pC%|h9ht?S$XKQ6h2;!~UkHDS_%bydy_os%@!!_)o=OVmxbHQ?6t({qh zw|~8$IJ0vgFz;H`ywry6NPHOLVZ^^zS35iq84Diuh25OqFUi)?V-4`QQ=JYyol>Cq z+a=`wQXS7RCG4{|-=7%*e$z5OO7vs`vcOBYyTbOquF!OU4l^Ge^pj~a<;Pq@1E3F; zMu1GpMvh{{ou*~B8+}t>nbjQ-R>htxdQFWj?|iZHy}>G45Bab@Ch$&`@>Ff18RyP7 zrn$t%R+u*Nf9=_0HYefFOU)lkEt8dTi|y4wW3p&Yv!U^mZgIJZYyeU(<6G03?!w%1 z)h>;wZ1IF2QZS|rnR<$bbVtxwDRTaP%Ue(COrTw><&KT?P&MD4AfKKa%io6UOq5yl z*mP4(Nsk?=zbg88$uP(h)K|9t%~f7tV-~yg(kMP|Hk^Ng=N>faOf6eqc9mcG!_2v` z_#TVd>;;8PGbp4zZK{+{&=*Kab3kF4ca*KKxPYtxRO{}RL{hUbKHCl^_O+C!uena7 zgW!1bW$a!o5T j2dCw-W`}TC-_+AV1jht1=MPH$(g4n$zTi-6@00R>sS_AH literal 0 HcmV?d00001 diff --git a/docs/public/favicon.svg b/docs/public/favicon.svg index ed06047cb..1a4f0aff1 100644 --- a/docs/public/favicon.svg +++ b/docs/public/favicon.svg @@ -1,14 +1,7 @@ - - - - - + + + + + + diff --git a/notebooks/tutorials/1.1-setup.ipynb b/notebooks/tutorials/1.1-setup.ipynb new file mode 100644 index 000000000..f00e7d33b --- /dev/null +++ b/notebooks/tutorials/1.1-setup.ipynb @@ -0,0 +1,354 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "EIQMxzgqlRX7" + }, + "source": [ + "

    \"drawing\"

    " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bIf4OMHUz6ma" + }, + "source": [ + "# Sotopia Tutorial Series\n", + "\n", + " \"Open \n", + "\n", + "The following is the first tutorial of this series to teach how to quickly build your socially intelligent agents on the sotopia platform.\n", + "\n", + "## Quick links\n", + "\n", + "1. Basics\n", + " \n", + " 1.1 [Playing with Sotopia in a Google Colab Notebook](https://colab.research.google.com/drive/14hJOfzpA37PRUzdlFgiqVzUGIhhngqnz?usp=sharing)\n", + " \n", + " 1.2 [Browsing sotopia data](https://colab.research.google.com/drive/1Gi2U3cA0KG1nekw1A0N0DZpx6gKrj8Nc?usp=sharing)\n", + "\n", + "2. Build your agents (coming soon!)\n", + "3. Evaluate your agents (coming soon!)\n", + "4. Extending Sotopia (coming soon!)\n", + "\n", + "Please contact [Hao Zhu](https://zhuhao.me) for any questions!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Fe0-ncx5U8q7" + }, + "source": [ + "# Guide on running Redis Stack for [Sotopia](https://sotopia.world)\n", + "\n", + "Author: [Hao Zhu](https://zhuhao.me)\n", + "\n", + "After the release of Sotopia package, I received feedback on the reliance of Sotopia on Redis, a KV database for maintaining profiles, interaction logs, error logs, and communication between the FastAPI server and agents. When hearing about database, people often think of a huge overhead. However, I want to show you in this tutorial that setting up Redis backend for Sotopia is actually extremely easy -- just run the following blocks." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DYRSxB4pWfDS" + }, + "source": [ + "## Install Redis\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "IJRC8SkzRteQ" + }, + "outputs": [], + "source": [ + "# @title The following block downloads Redis stack server 7.2.0, and libssl 1.1.\n", + "%%capture\n", + "!curl -fsSL https://packages.redis.io/redis-stack/redis-stack-server-7.2.0-v10.focal.x86_64.tar.gz -o redis-stack-server.tar.gz\n", + "!tar -xvf redis-stack-server.tar.gz\n", + "# Installs libssl1.1 for Ubuntu 22 source: https://stackoverflow.com/questions/72133316/libssl-so-1-1-cannot-open-shared-object-file-no-such-file-or-directory\n", + "!wget http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb\n", + "!sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb\n", + "%pip install redis" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "GlqtKYDmR2Ij", + "outputId": "d69d6466-5f8b-4904-ffce-7c1dbec0ca53" + }, + "outputs": [], + "source": [ + "# @title Just one line of code to spin up your redis server:\n", + "\n", + "\n", + "!./redis-stack-server-7.2.0-v10/bin/redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bIf7Nni3XEQk" + }, + "source": [ + "If the above cell results in no error message, you have successfully set up your Redis server." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "gNgwAngBR7vC", + "outputId": "70c0d1ca-6cd9-4adb-b1e0-b7d4859494d9" + }, + "outputs": [], + "source": [ + "# @title Try it out:\n", + "\n", + "import redis\n", + "\n", + "client = redis.Redis(host=\"localhost\", port=6379)\n", + "client.ping()\n", + "client.set(\"foo\", \"Yes it works.\")\n", + "client.get(\"foo\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RISoihCJXdiL" + }, + "source": [ + "## Run Sotopia with it" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "o0-S_Ug2d4vB" + }, + "source": [ + "To simulate an episode in Sotopia, there's one thing you will have to do: adding your `OPENAI_API_KEY` to the secret to left. Click on the little 🔑 shape button, and then toggle \"Notebook access\", input \"OPENAI_API_KEY\" under Name and the key string under Value.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ne2r-KmxXpNS" + }, + "outputs": [], + "source": [ + "# @title Run the following cell to add it to the environment variable.\n", + "%%capture\n", + "%pip install sotopia\n", + "import os\n", + "from google.colab import userdata\n", + "\n", + "os.environ[\"REDIS_OM_URL\"] = \"redis://:@localhost:6379\"\n", + "os.environ[\"OPENAI_API_KEY\"] = userdata.get(\"OPENAI_API_KEY\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "X-3yiYuIcLVZ" + }, + "outputs": [], + "source": [ + "# @title Helper functions for adding characters and social tasks\n", + "from sotopia.database.persistent_profile import AgentProfile, EnvironmentProfile\n", + "from typing import Any\n", + "\n", + "\n", + "def add_agent_to_database(**kwargs: dict[str, Any]) -> None:\n", + " agent = AgentProfile(**kwargs)\n", + " agent.save()\n", + "\n", + "\n", + "def add_env_profile(**kwargs: dict[str, Any]) -> None:\n", + " env_profile = EnvironmentProfile(**kwargs)\n", + " env_profile.save()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "6nPHso_4Z9s-" + }, + "outputs": [], + "source": [ + "# @title Let's first add a character to the database. Only name is required when creating a new character\n", + "\n", + "\n", + "first_name = \"Alice\" # @param {type:\"string\"}\n", + "last_name = \"Smith\" # @param {type:\"string\"}\n", + "age = 22 # @param {type:\"integer\"}\n", + "occupation = \"Data Scientist\" # @param {type:\"string\"}\n", + "gender = \"Woman\" # @param [\"Man\", \"Woman\", \"Nonbinary\"]\n", + "gender_pronoun = \"she/her\" # @param {type:\"string\"}\n", + "big_five = \"extraversion, openness\" # @param {type:\"string\"}\n", + "moral_values = [\"power\", \"achievement\"] # @param {type:\"string\"}\n", + "decision_making_style = \"analytical\" # @param {type:\"string\"}\n", + "secret = \"She cheated in her code interview\" # @param {type:\"string\"}\n", + "\n", + "add_agent_to_database(\n", + " first_name=first_name,\n", + " last_name=last_name,\n", + " age=age,\n", + " occupation=occupation,\n", + " gender=gender,\n", + " gender_pronoun=gender_pronoun,\n", + " big_five=big_five,\n", + " moral_values=moral_values,\n", + " decision_making_style=decision_making_style,\n", + " secret=secret,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "jaVLJSMsdVSJ" + }, + "outputs": [], + "source": [ + "# @title Let's add another character\n", + "\n", + "\n", + "first_name = \"Bob\" # @param {type:\"string\"}\n", + "last_name = \"Johnson\" # @param {type:\"string\"}\n", + "age = 30 # @param {type:\"integer\"}\n", + "occupation = \"Chef\" # @param {type:\"string\"}\n", + "gender = \"Man\" # @param [\"Man\", \"Woman\", \"Nonbinary\"]\n", + "gender_pronoun = \"he/his\" # @param {type:\"string\"}\n", + "big_five = \"openness, conscientiousness\" # @param {type:\"string\"}\n", + "moral_values = [\"Benevolence\"] # @param {type:\"string\"}\n", + "decision_making_style = \"conceptual\" # @param {type:\"string\"}\n", + "secret = \"He has a negative net worth.\" # @param {type:\"string\"}\n", + "\n", + "add_agent_to_database(\n", + " first_name=first_name,\n", + " last_name=last_name,\n", + " age=age,\n", + " occupation=occupation,\n", + " gender=gender,\n", + " gender_pronoun=gender_pronoun,\n", + " big_five=big_five,\n", + " moral_values=moral_values,\n", + " decision_making_style=decision_making_style,\n", + " secret=secret,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "gE4iBfHjfZxM" + }, + "outputs": [], + "source": [ + "# @title Let's add a social tasks\n", + "\n", + "scenario = 'A friend is raising a fund for the \"Help the children\" charity' # @param {type:\"string\"}\n", + "social_goal_1 = \"Ask for donation of $100\" # @param {type:\"string\"}\n", + "social_goal_2 = \"Donate less than $10\" # @param {type:\"string\"}\n", + "\n", + "add_env_profile(scenario=scenario, agent_goals=[social_goal_1, social_goal_2])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "TcYo311NdJNb", + "outputId": "e03fb8f3-6cef-4160-ab95-1674747ae70a" + }, + "outputs": [], + "source": [ + "# @title Finally, run a server:\n", + "\n", + "from sotopia.samplers import UniformSampler\n", + "from sotopia.server import run_async_server\n", + "\n", + "\n", + "await run_async_server(\n", + " model_dict={\n", + " \"env\": \"gpt-4\",\n", + " \"agent1\": \"gpt-3.5-turbo\",\n", + " \"agent2\": \"gpt-3.5-turbo\",\n", + " },\n", + " sampler=UniformSampler(),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OJXZlHvJAGBS" + }, + "source": [ + "### A note on what is running under the hood\n", + "\n", + "After you added the characters and scenarios to the database, the `run_async_server` samples a pair of characters and one scenario to run the simulation, and then evaluate the simulation. All of the sampling, interaction simulation, and evaluation are handled by Sotopia. You could easily tweak them through editing `run_async_server`, or changing the LLMs, or the sampler according to your own need." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9YNRGLsRfMvG" + }, + "source": [ + "## Conclusion\n", + "\n", + "In this very short tutorial, I demonstrated the whole pipeline of setting up Redis server, installing sotopia, adding characters and social tasks, and simulating + evaluating the social interaction between two GPT-3.5 agents with GPT-4. This tutorial intends to serve as the first quick example for you. The only thing you have to do besides running all of the cells is add your own OpenAI key[[1]](#cite_note-1). You could also use other LLMs supported by LiteLLM even your own API endpoint. Let me know if you want me to explain more.\n", + "\n", + " [1](#cite_ref-1): *sorry I cannot afford to share mine.*" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.1.undefined" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/tutorials/1.2-browse-data.ipynb b/notebooks/tutorials/1.2-browse-data.ipynb new file mode 100644 index 000000000..ff500878c --- /dev/null +++ b/notebooks/tutorials/1.2-browse-data.ipynb @@ -0,0 +1,334 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "EIQMxzgqlRX7" + }, + "source": [ + "

    \"drawing\"

    " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ocrtYaex1s0h" + }, + "source": [ + "# Sotopia Tutorial Series\n", + "\n", + " \"Open \n", + "\n", + "The following is the second tutorial of this series to teach how to quickly build your socially intelligent agents on the sotopia platform.\n", + "\n", + "## Quick links\n", + "\n", + "1. Basics\n", + " \n", + " 1.1 [Playing with Sotopia in a Google Colab Notebook](https://colab.research.google.com/drive/14hJOfzpA37PRUzdlFgiqVzUGIhhngqnz?usp=sharing)\n", + " \n", + " 1.2 [Browsing sotopia data](https://colab.research.google.com/drive/1Gi2U3cA0KG1nekw1A0N0DZpx6gKrj8Nc?usp=sharing)\n", + "\n", + "2. Build your agents (coming soon!)\n", + "3. Evaluate your agents (coming soon!)\n", + "4. Extending Sotopia (coming soon!)\n", + "\n", + "Please contact [Hao Zhu](https://zhuhao.me) for any questions!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Fe0-ncx5U8q7" + }, + "source": [ + "\n", + "# 1.2 Browsing [Sotopia](https://sotopia.world) data\n", + "\n", + "Author: [Hao Zhu](https://zhuhao.me)\n", + "\n", + "Running the code in this tutorial does not require familiarity of the redis setup. But if you have finished [Tutorial 1.1](https://colab.research.google.com/drive/14hJOfzpA37PRUzdlFgiqVzUGIhhngqnz?usp=sharing), you would have a better idea of the characters (in the format of `sotopia.database.AgentProfile`) and social tasks (in the format of `sotopia.database.EnvironmentProfile`). What you would be able to learn through this tutorial are:\n", + "\n", + "1. How to load all of the existing data from sotopia(-pi) experiments, including 40 characters, hundreds of scenarios, and tens of thousands of interaction episodes.\n", + "2. How to browse them through indexing based on two awesome libraries `redis-om` and `pydantic`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DYRSxB4pWfDS" + }, + "source": [ + "## Install Redis\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "IJRC8SkzRteQ" + }, + "outputs": [], + "source": [ + "# @title Download Redis: This code block is exactly the same as [Tutorial 1.1](https://colab.research.google.com/drive/14hJOfzpA37PRUzdlFgiqVzUGIhhngqnz?usp=sharing).\n", + "%%capture\n", + "!curl -fsSL https://packages.redis.io/redis-stack/redis-stack-server-7.2.0-v10.focal.x86_64.tar.gz -o redis-stack-server.tar.gz\n", + "!tar -xvf redis-stack-server.tar.gz\n", + "# Installs libssl1.1 for Ubuntu 22 source: https://stackoverflow.com/questions/72133316/libssl-so-1-1-cannot-open-shared-object-file-no-such-file-or-directory\n", + "!wget http://nz2.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb\n", + "!sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb\n", + "%pip install redis" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "GlqtKYDmR2Ij" + }, + "outputs": [], + "source": [ + "%%capture\n", + "# @title Download existing data in a single file, and launch redis\n", + "\n", + "!mkdir -p /content/redis-stack-server-7.2.0-v10/var/db/redis-stack\n", + "!curl -L https://huggingface.co/datasets/cmu-lti/sotopia-pi/resolve/main/dump.rdb?download=true --output /content/redis-stack-server-7.2.0-v10/var/db/redis-stack/dump.rdb\n", + "!./redis-stack-server-7.2.0-v10/bin/redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bIf7Nni3XEQk" + }, + "source": [ + "If the above cell results in no error message, you have successfully set up your Redis server." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "gNgwAngBR7vC", + "outputId": "321eb7f8-29c1-4d84-aa9a-2f114a4f8f49" + }, + "outputs": [], + "source": [ + "# @title Your DB is now pre-populated with existing data:\n", + "\n", + "import redis\n", + "from redis import BusyLoadingError\n", + "import time\n", + "\n", + "client = redis.Redis(host=\"localhost\", port=6379)\n", + "while True:\n", + " try:\n", + " client.ping()\n", + " break\n", + " except BusyLoadingError:\n", + " time.sleep(1)\n", + "print(f\"Voila, you have loaded {client.dbsize()} data!\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RISoihCJXdiL" + }, + "source": [ + "## Browse the data you have just loaded" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "ne2r-KmxXpNS" + }, + "outputs": [], + "source": [ + "# @title First, install sotopia\n", + "%%capture\n", + "%pip install sotopia\n", + "import os\n", + "\n", + "os.environ[\"REDIS_OM_URL\"] = \"redis://:@localhost:6379\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 353 + }, + "id": "ImjdFUlGGA4_", + "outputId": "df1abaac-fce7-4e14-fd84-0dc3d96d8521" + }, + "outputs": [], + "source": [ + "# @title Show me a random character\n", + "import random\n", + "from rich import print\n", + "\n", + "from sotopia.database import AgentProfile\n", + "\n", + "\n", + "all_character_pks = list(AgentProfile.all_pks())\n", + "pk = random.choice(all_character_pks)\n", + "\n", + "print(AgentProfile.get(pk))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 321 + }, + "id": "Su5EwfIPHQWp", + "outputId": "47e36153-fa91-4a34-cc5b-80dea2e516dd" + }, + "outputs": [], + "source": [ + "# @title Show me a random social task\n", + "from sotopia.database import EnvironmentProfile\n", + "\n", + "all_task_pks = list(EnvironmentProfile.all_pks())\n", + "pk = random.choice(all_task_pks)\n", + "\n", + "print(EnvironmentProfile.get(pk))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "SFW9PRzzH1Pt", + "outputId": "d53ab793-6740-4e69-9fc7-3a911c892aa1" + }, + "outputs": [], + "source": [ + "# @title Show me an Episode of the random social task before\n", + "\n", + "from sotopia.database import EpisodeLog\n", + "\n", + "all_episodes = list(EpisodeLog.find(EpisodeLog.environment == pk).all())\n", + "if len(all_episodes) == 0:\n", + " print(\"No episodes found for this task, Please try to sample another task.\")\n", + "else:\n", + " episode = random.choice(all_episodes)\n", + " print(episode)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dYx2N8njZAIi" + }, + "source": [ + "### Find all young chefs in Sotopia\n", + "\n", + "Powered by `redis_om`, the database of sotopia can be easily indexed by simple logics" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 33 + }, + "id": "7mAMGMXgYGRE", + "outputId": "8b85b2fa-c3de-4054-d466-223530ceea18" + }, + "outputs": [], + "source": [ + "young_chefs = AgentProfile.find(\n", + " (AgentProfile.occupation == \"Chef\") & (AgentProfile.age < 45)\n", + ").all()\n", + "print(\n", + " f\"There are {len(young_chefs)} young chefs in sotopia. They are: {', '.join(chef.first_name for chef in young_chefs)}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8iIG2EWOa1A1" + }, + "source": [ + "### Tagging system in Sotopia\n", + "\n", + "You might notice that Sotopia episodes are all \"tagged\" which allows for tracing the source of the generation. You can list all of the tags" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "sDST-ZcJeubQ", + "outputId": "1750cc27-909d-4b93-9d5d-b3f8e8cfc82d" + }, + "outputs": [], + "source": [ + "def get_episode_tag(episode_id):\n", + " try:\n", + " episode = EpisodeLog.get(episode_id)\n", + " return episode.tag\n", + " except Exception as _:\n", + " return \"\"\n", + "\n", + "\n", + "set(get_episode_tag(episode) for episode in EpisodeLog.all_pks())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9YNRGLsRfMvG" + }, + "source": [ + "## Conclusion\n", + "\n", + "In this very short tutorial, I demonstrated how to load the existing data from sotopia(-pi) experiments and browse the data with the easy-to-use api in sotopia. You can enumerate or find a random character and social task, and you can also index the interaction episodes with the task id or character id." + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}