Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/minecraft update #276

Open
wants to merge 22 commits into
base: minecraft
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
77462e2
create stone_pickaxe
XianzheFan Dec 24, 2024
976a440
help each other to make 3 stone pickaxes and add codeOutput to memory
XianzheFan Dec 25, 2024
ec73201
add theory of mind (goal inference)
XianzheFan Dec 25, 2024
c854d11
Delete examples/experimental/minecraft_agents/agent_output_log_202412…
XianzheFan Dec 27, 2024
81e21cc
Delete examples/experimental/minecraft_agents/agent_output_log_202412…
XianzheFan Dec 27, 2024
1bb4b99
Delete examples/experimental/minecraft_agents/agent_output_log_202412…
XianzheFan Dec 27, 2024
6a67143
add theory of mind (goal inference)
XianzheFan Dec 27, 2024
fe355fd
toml files for different tasks
XianzheFan Dec 28, 2024
f3c914b
added some new ToM questions
XianzheFan Jan 9, 2025
3e99c74
improve log
XianzheFan Jan 10, 2025
1b29ee2
read images from google cloud storage; add openai_api_predict_async
XianzheFan Jan 11, 2025
a5e0b26
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 12, 2025
7a2dff4
add chest_no_task_division
XianzheFan Jan 13, 2025
b3c0bad
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 13, 2025
e6c037b
add boat_no_division.toml
XianzheFan Jan 15, 2025
93fa00f
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 15, 2025
483ef2b
add stone_pickaxe & wooden pickaxe (no task division)
XianzheFan Jan 16, 2025
5876b37
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 16, 2025
9375428
add door_no_task_division.toml
XianzheFan Jan 16, 2025
f9142bb
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 16, 2025
3d4ead0
Automatically generate TOML files
XianzheFan Jan 19, 2025
2759cf2
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
309 changes: 283 additions & 26 deletions examples/experimental/minecraft_agents/group_discussion_agents.py

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -6,38 +6,65 @@ node_name = "Jack"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 5
query_interval = 15
output_channel = "Jack"
input_text_channels = ["Jane", "John"]
input_tick_channel = "tick/secs/1"
goal = "want to craft a Stone Pickaxe with your friends. The materials needed include various types of Logs (which need to be turned into Planks and a Crafting Table) and different kinds of Stone."
model_name = "gpt-4o-mini"
goal = """
You and your friends need to craft 2 "chest" and place them. Please discuss the task division quickly and start working.

The complete process for crafting a "chest" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log", or "birch_log", with at least 3 pieces.
2. Convert the logs into planks ("birch_planks", "spruce_planks", or "oak_planks"). Each log can produce 4 planks.
3. Use !craftRecipe("crafting_table", 1) to craft a "crafting_table" (requires 4 planks).
4. After crafting a "crafting_table" (Ensure the inventory shows crafting_table: 1, not just having previously outputted !craftRecipe("crafting_table", 1)), use the command !placeHere("crafting_table").
5. After crafting or finding a "crafting_table" (You can use !goToBlock("crafting_table", 20, 50) to locate a nearby "crafting_table"), use !craftRecipe("chest", 1) to craft a "chest" (requires 8 planks).
6. Use the command !placeHere("chest") to place the chest."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jack"

[[nodes]]
node_name = "Jane"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 7
query_interval = 19
output_channel = "Jane"
input_text_channels = ["Jack", "John"]
input_tick_channel = "tick/secs/1"
goal = "want to craft a Stone Pickaxe with your friends. The materials needed include various types of Logs (which need to be turned into Planks and a Crafting Table) and different kinds of Stone."
model_name = "gpt-4o-mini"
goal = """
You and your friends need to craft 2 "chest" and place them. Please discuss the task division quickly and start working.

The complete process for crafting a "chest" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log", or "birch_log", with at least 3 pieces.
2. Convert the logs into planks ("birch_planks", "spruce_planks", or "oak_planks"). Each log can produce 4 planks.
3. Use !craftRecipe("crafting_table", 1) to craft a "crafting_table" (requires 4 planks).
4. After crafting a "crafting_table" (Ensure the inventory shows crafting_table: 1, not just having previously outputted !craftRecipe("crafting_table", 1)), use the command !placeHere("crafting_table").
5. After crafting or finding a "crafting_table" (You can use !goToBlock("crafting_table", 20, 50) to locate a nearby "crafting_table"), use !craftRecipe("chest", 1) to craft a "chest" (requires 8 planks).
6. Use the command !placeHere("chest") to place the chest."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jane"

[[nodes]]
node_name = "John"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 10
query_interval = 23
output_channel = "John"
input_text_channels = ["Jack", "Jane"]
input_tick_channel = "tick/secs/1"
goal = "want to craft a Stone Pickaxe with your friends. The materials needed include various types of Logs (which need to be turned into Planks and a Crafting Table) and different kinds of Stone."
model_name = "gpt-4o-mini"
goal = """
You and your friends need to craft 2 "chest" and place them. Please discuss the task division quickly and start working.

The complete process for crafting a "chest" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log", or "birch_log", with at least 3 pieces.
2. Convert the logs into planks ("birch_planks", "spruce_planks", or "oak_planks"). Each log can produce 4 planks.
3. Use !craftRecipe("crafting_table", 1) to craft a "crafting_table" (requires 4 planks).
4. After crafting a "crafting_table" (Ensure the inventory shows crafting_table: 1, not just having previously outputted !craftRecipe("crafting_table", 1)), use the command !placeHere("crafting_table").
5. After crafting or finding a "crafting_table" (You can use !goToBlock("crafting_table", 20, 50) to locate a nearby "crafting_table"), use !craftRecipe("chest", 1) to craft a "chest" (requires 8 planks).
6. Use the command !placeHere("chest") to place the chest."""
model_name = "gpt-4o-2024-11-20"
agent_name = "John"

[[nodes]]
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
redis_url = "redis://localhost:6379/0"
extra_modules = ["examples.experimental.group_discussion_agents.group_discussion_agents"]

[[nodes]]
node_name = "Jack"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 15
output_channel = "Jack"
input_text_channels = ["Jane", "John"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft a "chest" and a "boat". YOUR TASK IS TO CRAFT A "crafting_table".
After crafting a "crafting_table" yourself (If the condition is that the inventory shows crafting_table: 1, rather than just having previously outputted !craftRecipe(\"crafting_table\", 1)), use the command !placeHere(\"crafting_table\").

The complete process for crafting a "crafting_table" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log" or "birch_log" with at least 1 piece.
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. Use 4 planks to craft a "crafting_table"."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jack"

[[nodes]]
node_name = "Jane"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 19
output_channel = "Jane"
input_text_channels = ["Jack", "John"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft a "chest" and a "boat". YOUR TASK IS TO CRAFT ONE "chest".
Once you have crafted at least 8 planks (If the condition is that the inventory shows oak_planks: 8, birch_planks: 8 or spruce_planks: 8), use the command !goToPlayer("Jack", 0) to go to Jack's location.
Then, use the command !goToBlock("crafting_table", 10, 20) to locate a nearby "crafting_table". Next, use !craftRecipe("chest", 1) to craft a chest, then use the command !placeHere(\"chest\") to place it.

The complete process for crafting a "chest" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect at least 2 logs, such as "oak_log", "spruce_log" or "birch_log".
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. After finding Jack and a nearby "crafting_table", use 8 planks to create a "chest"."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jane"

[[nodes]]
node_name = "John"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 23
output_channel = "John"
input_text_channels = ["Jack", "Jane"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft a "chest" and a "boat". YOUR TASK IS TO CRAFT ONE BOAT.
Once you have crafted at least 5 planks (If the condition is that the inventory shows oak_planks: 5, birch_planks: 5 or spruce_planks: 5), use the command !goToPlayer("Jack", 0) to go to Jack's location.
Then, use the command !goToBlock("crafting_table", 10, 20) to locate a nearby "crafting_table". Next, use !craftRecipe("oak_boat", 1), !craftRecipe("birch_boat", 1) or !craftRecipe("spruce_boat", 1) to craft a boat.
After that, use the command !goToBlock("chest", 10, 20) to locate a nearby "chest", and then use !putInChest("oak_boat", 1), !putInChest("birch_boat", 1) or !putInChest("spruce_boat", 1) to place the boat in the chest.

The complete process for crafting a boat in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect at least 2 logs, such as "oak_log", "spruce_log" or "birch_log".
2. Convert the logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. After finding Jack and a nearby "crafting_table", use 5 planks to create a boat."""
model_name = "gpt-4o-2024-11-20"
agent_name = "John"

[[nodes]]
node_name = "record"
node_class = "record"

[nodes.node_args]
jsonl_file_path = "log.jsonl"

[nodes.node_args.record_channel_types]
"Jack" = "agent_action"
"Jane" = "agent_action"
"John" = "agent_action"

[[nodes]]
node_name = "tick"
node_class = "tick"
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
redis_url = "redis://localhost:6379/0"
extra_modules = ["examples.experimental.group_discussion_agents.group_discussion_agents"]

[[nodes]]
node_name = "Jack"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 15
output_channel = "Jack"
input_text_channels = ["Jane", "John"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft 2 "boat". Please discuss the task division quickly and start working.

The complete process for crafting a "boat" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log" or "birch_log" with at least 3 piece.
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. Use !craftRecipe("crafting_table", 1) to craft a "crafting_table" (requires 4 planks).
4. After crafting a "crafting_table" (Ensure the inventory shows crafting_table: 1, not just having previously outputted !craftRecipe("crafting_table", 1)), use the command !placeHere("crafting_table").
5. After crafting or finding a "crafting_table" (You can use !goToBlock("crafting_table", 20, 50) to locate a nearby "crafting_table"), use !craftRecipe("oak_boat", 1), !craftRecipe("birch_boat", 1) or !craftRecipe("spruce_boat", 1) to craft a boat (requires 5 planks).
"""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jack"

[[nodes]]
node_name = "Jane"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 19
output_channel = "Jane"
input_text_channels = ["Jack", "John"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft 2 "boat". Please discuss the task division quickly and start working.

The complete process for crafting a "boat" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log" or "birch_log" with at least 3 piece.
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. Use !craftRecipe("crafting_table", 1) to craft a "crafting_table" (requires 4 planks).
4. After crafting a "crafting_table" (Ensure the inventory shows crafting_table: 1, not just having previously outputted !craftRecipe("crafting_table", 1)), use the command !placeHere("crafting_table").
5. After crafting or finding a "crafting_table" (You can use !goToBlock("crafting_table", 20, 50) to locate a nearby "crafting_table"), use !craftRecipe("oak_boat", 1), !craftRecipe("birch_boat", 1) or !craftRecipe("spruce_boat", 1) to craft a boat (requires 5 planks)."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jane"

[[nodes]]
node_name = "John"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 23
output_channel = "John"
input_text_channels = ["Jack", "Jane"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft 2 "boat". Please discuss the task division quickly and start working.

The complete process for crafting a "boat" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log" or "birch_log" with at least 3 piece.
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. Use !craftRecipe("crafting_table", 1) to craft a "crafting_table" (requires 4 planks).
4. After crafting a "crafting_table" (Ensure the inventory shows crafting_table: 1, not just having previously outputted !craftRecipe("crafting_table", 1)), use the command !placeHere("crafting_table").
5. After crafting or finding a "crafting_table" (You can use !goToBlock("crafting_table", 20, 50) to locate a nearby "crafting_table"), use !craftRecipe("oak_boat", 1), !craftRecipe("birch_boat", 1) or !craftRecipe("spruce_boat", 1) to craft a boat (requires 5 planks)."""
model_name = "gpt-4o-2024-11-20"
agent_name = "John"

[[nodes]]
node_name = "record"
node_class = "record"

[nodes.node_args]
jsonl_file_path = "log.jsonl"

[nodes.node_args.record_channel_types]
"Jack" = "agent_action"
"Jane" = "agent_action"
"John" = "agent_action"

[[nodes]]
node_name = "tick"
node_class = "tick"
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
redis_url = "redis://localhost:6379/0"
extra_modules = ["examples.experimental.group_discussion_agents.group_discussion_agents"]

[[nodes]]
node_name = "Jack"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 15
output_channel = "Jack"
input_text_channels = ["Jane", "John"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft 2 "chest". YOUR TASK IS TO CRAFT A "crafting_table".
After crafting a "crafting_table" yourself (If the condition is that the inventory shows crafting_table: 1, rather than just having previously outputted !craftRecipe(\"crafting_table\", 1)), use the command !placeHere(\"crafting_table\").

The complete process for crafting a "crafting_table" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect logs, such as "oak_log", "spruce_log" or "birch_log" with at least 1 piece.
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. Use 4 planks to craft a "crafting_table"."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jack"

[[nodes]]
node_name = "Jane"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 19
output_channel = "Jane"
input_text_channels = ["Jack", "John"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft 2 "chest". YOUR TASK IS TO CRAFT ONE "chest".
Once you have crafted at least 8 planks (If the condition is that the inventory shows oak_planks: 8, birch_planks: 8 or spruce_planks: 8), use the command !goToPlayer("Jack", 0) to go to Jack's location.
Then, use the command !goToBlock("crafting_table", 10, 20) to locate a nearby "crafting_table". Next, use !craftRecipe("chest", 1) to craft a chest, then use the command !placeHere(\"chest\") to place it.

The complete process for crafting a "chest" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect at least 2 logs, such as "oak_log", "spruce_log" or "birch_log".
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. After finding Jack and a nearby "crafting_table", use 8 planks to create a "chest"."""
model_name = "gpt-4o-2024-11-20"
agent_name = "Jane"

[[nodes]]
node_name = "John"
node_class = "llm_agent"

[nodes.node_args]
query_interval = 23
output_channel = "John"
input_text_channels = ["Jack", "Jane"]
input_tick_channel = "tick/secs/1"
goal = """
You and your friends need to craft 2 "chest". YOUR TASK IS TO CRAFT ONE "chest".
Once you have crafted at least 8 planks (If the condition is that the inventory shows oak_planks: 8, birch_planks: 8 or spruce_planks: 8), use the command !goToPlayer("Jack", 0) to go to Jack's location.
Then, use the command !goToBlock("crafting_table", 10, 20) to locate a nearby "crafting_table". Next, use !craftRecipe("chest", 1) to craft a chest, then use the command !placeHere(\"chest\") to place it.

The complete process for crafting a "chest" in Minecraft is as follows (Please craft or obtain the items strictly in the following order):
1. Collect at least 2 logs, such as "oak_log", "spruce_log" or "birch_log".
2. Convert logs into planks ("birch_planks", "spruce_planks" or "oak_planks"). Each log can produce 4 planks.
3. After finding Jack and a nearby "crafting_table", use 8 planks to create a "chest"."""
model_name = "gpt-4o-2024-11-20"
agent_name = "John"

[[nodes]]
node_name = "record"
node_class = "record"

[nodes.node_args]
jsonl_file_path = "log.jsonl"

[nodes.node_args.record_channel_types]
"Jack" = "agent_action"
"Jane" = "agent_action"
"John" = "agent_action"

[[nodes]]
node_name = "tick"
node_class = "tick"
Loading
Loading