Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enabled saving and evaluation for moderator #271

Merged
merged 8 commits into from
Jan 7, 2025

Conversation

JXZhou0224
Copy link
Collaborator

@JXZhou0224 JXZhou0224 commented Jan 2, 2025

Closes #

📑 Description

added two new features for moderator:

saving

  1. Now LLMAgents will store their profile in a AgentProfile on redis.
  2. Later, LLMAgent can be loaded from existing AgentProfiles
  3. and moderator can access AgentProfile to generate an EpisodeLog

evaluation

  1. created evaluators for multiagent assessment (current Evaluators in sotopia only supports two agents)
  2. moderator can now evaluate an episode on demand, currently there is only one evaluator available, new evaluators will be added later.

✅ Checks

  • My pull request adheres to the code style of this project
  • My code requires changes to the documentation
  • I have updated the documentation as required
  • All the tests have passed
  • Branch name follows type/descript (e.g. feature/add-llm-agents)
  • Ready for code review

ℹ Additional Information

@JXZhou0224 JXZhou0224 marked this pull request as ready for review January 2, 2025 08:24
@XuhuiZhou XuhuiZhou self-requested a review January 6, 2025 16:16
examples/experimental/sotopia_original_replica/origin.toml Outdated Show resolved Hide resolved
examples/experimental/sotopia_original_replica/origin.toml Outdated Show resolved Hide resolved
sotopia/experimental/agents/logs.py Outdated Show resolved Hide resolved
sotopia/experimental/agents/logs.py Outdated Show resolved Hide resolved
@@ -0,0 +1,25 @@
from abc import ABC, abstractmethod
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not using the existing evaluators

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing evaluators base class can only take in message history. However, there might be cases where we need other information in EpisodeLog to generate an evaluation. So I made a new class of evaluator for future use.

@JXZhou0224 JXZhou0224 requested a review from XuhuiZhou January 7, 2025 10:03
@XuhuiZhou XuhuiZhou changed the base branch from demo to feature/multiparty January 7, 2025 19:34
@XuhuiZhou XuhuiZhou merged commit 66da649 into sotopia-lab:feature/multiparty Jan 7, 2025
1 check passed
JXZhou0224 added a commit that referenced this pull request Jan 18, 2025
feat: enabled saving and evaluation for moderator (#271)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants