Question about examples/interative_play.py

[amoyplane]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: master(368007d)
Habitat-Sim: v0.2.0

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release versions of Habitat-Lab and Habitat-Sim? Your question may already be addressed in the latest versions. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Master branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-lab/

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/

Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

I meet some questions when I try to run examples/interative_play.py.

1. From Not the same coordinate system? #337 I know that Habitat uses -Y gravity. But when I run interative_play.py using rearrangepick_replica_cad_example_ik.yaml as config file, I got a upsidedown third rgb sensor view and it seems different from the room model I opened in Blender(Maybe it is flipped horizontally?). So is there any place need to be change except the THIRD_RGB_SENSOR entry in the yaml file?

2. I modified obs["rgb"] = obs["robot_head_rgb"] in interactive_play.py for picture from robot view, but I got a pic near the floor. I am wondering that if I want to change the position of camera(e.g. import other robot with different camera position or get view from different part of the robot), which file should I modify.

3. interative_play.py crash when the arm or the robot itself hit wall or furniture. Here is the info I got:

Traceback (most recent call last):
  File "examples/interactive_play.py", line 379, in <module>
    play_env(env, args, config)
  File "examples/interactive_play.py", line 299, in play_env
    env,
  File "examples/interactive_play.py", line 250, in get_input_vel_ctlr
    return step_env(env, name, args, g_args), arm_action
  File "examples/interactive_play.py", line 129, in step_env
    return env.step({"action": action_name, "action_args": action_args})
  File "/habitat-lab/habitat/core/env.py", line 274, in step
    observations=observations,
  File "/habitat-lab/habitat/core/embodied_task.py", line 166, in update_measures
    measure.update_metric(*args, **kwargs)
  File "/habitat-lab/habitat/tasks/rearrange/rearrange_sensors.py", line 400, in update_metric
    cur_coll_info = self._task.get_cur_collision_info()
  File "/habitat-lab/habitat/tasks/rearrange/rearrange_task.py", line 112, in get_cur_collision_info
    self._config.COUNT_OBJ_COLLISIONS,
  File "/habitat-lab/habitat/tasks/rearrange/utils.py", line 112, in rearrange_collision
    colls = list(filter(should_keep, colls))
  File "/habitat-lab/habitat/tasks/rearrange/utils.py", line 99, in should_keep
    if match_link is not None and robot_model.is_base_link(match_link):
AttributeError: 'FetchRobot' object has no attribute 'is_base_link'

4. Is there any way to get observation(rgbd img) from arbitrary view during simulation without registration in config?

Thanks for your help.

[dhruvbatra]

CC: @ASzot

[ASzot]

Thank you for raising these issues. The interactive play script was working locally for me, but upon reinstalling I experienced the same problems.

1. This is a bug. Fixed in this PR: Improved and fixed interactive play script #687.
2. Not sure why this is happening, I think this is a bug, I will fix it and get back to you.
3. You have an incompatible version of Habitat Sim installed, I recommend installing from source on Habitat Sim master if you are using Habitat Lab master. @mathfac , what do you recommend?
4. Yes, you can dynamically change the camera offsets and look at positions. You can do so with sim.robot.params (accessed as env._sim.robot.params in interactive_play.py) and changing these values in the robot parameter data structure: https://github.com/facebookresearch/habitat-sim/blob/334d642f30c53e79305efd0c6f47cfb64c5c9d2f/habitat_sim/robots/mobile_manipulator.py#L68.

[amoyplane]

Thank you for raising these issues. The interactive play script was working locally for me, but upon reinstalling I experienced the same problems.

1. This is a bug. Fixed in this PR: Improved and fixed interactive play script #687.
2. Not sure why this is happening, I think this is a bug, I will fix it and get back to you.
3. You have an incompatible version of Habitat Sim installed, I recommend installing from source on Habitat Sim master if you are using Habitat Lab master. @mathfac , what do you recommend?
4. Yes, you can dynamically change the camera offsets and look at positions. You can do so with sim.robot.params (accessed as env._sim.robot.params in interactive_play.py) and changing these values in the robot parameter data structure: https://github.com/facebookresearch/habitat-sim/blob/334d642f30c53e79305efd0c6f47cfb64c5c9d2f/habitat_sim/robots/mobile_manipulator.py#L68.

Thank you for your reply. I use Habitat-sim mater and #https://github.com/facebookresearch/habitat-lab/tree/interactive_play_fixes , the problem of crash and flipped horizontally solved :).

As for the wrong view from headcamera, I found that https://github.com/facebookresearch/habitat-sim/blob/master/habitat_sim/robots/fetch_robot.py#L22 uses a -Z gravity coordinate system, and modify it with

env._sim.robot.params.head_cam_offset_pos=mn.Vector3(0.17, 1.2, 0.0)
env._sim.robot.params.head_cam_look_pos=mn.Vector3(1, 0.75, 0.75)

will work:

Here comes the next questions that after update to Habitat Sim master, some configs in yaml file become invalid. For example

THIRD_RGB_SENSOR:
        WIDTH: 512
        HEIGHT: 512
        ORIENTATION: [-1.57, 1.57, 0.0]
        POSITION: [0, 2, 0]

would not change the position of third rgb sensor and I have to use code to modify the robot parameter data. And the position is described in robot's frame. It would be a great help if I can access a camera from both world frame and robot's frame. (maybe related with facebookresearch/habitat-sim@7992248)

[amoyplane]

When adapting my work to habitat2.0 I am confused with some questions that:

1. The arm can get through the surface of wall, table and refrigerator where it should be blocked (while the robot itself can't). I can see the number of Robo-Scene Coll increase, so this would be a feature or a bug? And the work of Home Assistant Benchmark (HAB) in paper (like open the drawer) seems have not been released yet?
   

2. For some reason I need some high quality picture at some keypoint during simulation. The paper says habitat2.0 is a lazy simulator, and I want to konw if there is any config to get rendering in high quality. Thanks for your help.

[ASzot]

We improved the lighting in the most recent version and it looks a bit better now. However, we are still working on fixing the lighting.

Also, the hab_suite branch now contains most of the functionality from the paper and we are slowly merging this into main. We made this tutorial to help get started with Habitat 2.0.

Please let me know if you have any more questions!