3D Control Interfaces

[mayacakmak]

I see progress on the 3d-interface branch but haven't been able to test it. @kaviMD what is the status? What remains to be done? How could I help?

[kavidey]

I have integrated the IK into practice.html, and split the camera into four views:

Currently, it can be run locally by creating a python server with python -m http.server.

The biggest challenge right now is having multiple EEs in one webpage. The way the code is written right now, we can only have one per webpage. I think that this is a deeper issue than just editing workspace.js, but that control.js and each of the individual controls will need to be edited to support having multiple interfaces.

In terms of how the webpage is physically setup, there are 3 layers of HTML. The furthest back is the three.js canvas which is split into 4 views. On top of that is one svg element (The svg id of #workspace is hardcoded in many locations, and it seemed simpler to have 3 targets in one svg element rather than modify all of the code to have 3 separate svg elements with 3 separate ids). Because there will be 3 targets in one workspace, we will have to write custom code to limit the possible positions of the EE. I have not implemented this yet, so in the current version of the website, the EE is only meant to control the top left camera view, even though it move across the whole screen.

[mayacakmak]

Thanks, I was able to try it out!

Hmm I think I understand the problem. We might need a deeper refactoring to address. In the new setup the end-effector and the target are now SE3 instead of SE2, i.e. they have 6 degrees of freedom (x, y, z, theta_x, theta_y_theta_z). Each view has a separate control that can change a subset of these V1: x, y, theta_z; V2: x, z, theta_y; V3: y, z, theta_x. So it is not really three different EEs and probably best not to try and patch it by treating it that way. Does that make sense?

Starting some TODOs here:

• Need a way to switch between the three views. Also clear indicator of which view is currently selected.
• Target should be a ghost robot gripper (end effector). Need to indicate a clear point on the hand that corresponds to the center of the control (the dot).
• Control starts at the center of the screen -- should be aligned to the gripper in the initially selected view.
• Each view should be zoomed and repositioned to cover a reasonable workspace within which we will choose targets. Perhaps need to zoom equally such that the gripper is the same size in all three control views.
• One arm should be tucked to the side, out of view and out of the way; we'll always control one of the arms
• Initial pose of the arm should be such that it can easily move in each direction. Gripper should probably be open the whole time.
• There's some flickering with IK, could be resolved if movements are more within range, but might need some smoothing to avoid.

[kavidey]

I think I understand what we need to do. I am working on creating an SE3 class with support for movement and rotation in all 3 axes. I also added some radio buttons at the top of the page to select one of the 3 views (This is probably not the most efficient solution, but it is simple to test with, and it should be easy to replace with something better later).

This is a mockup of how I envision the whole interface looking:

There is one "controller" per view that can be manipulated. Each degree of freedom of that control maps to one degree of freedom of the EE.
When a view is selected the other 3 are darkened and can no longer be interacted with. Because multiple controllers can be attached to the same axis (For example the x axis of the controllers in views 1 and 2 are the same), if view 1 is active and that controller is moved on the x axis, it would automatically move the controller in view 2 also.
We could also add green arrows to help differentiate between the 3 axes of movement.

Let me know if there are any changes that I should make to the interface

[mayacakmak]

@kaviMD nice work at getting to those TODOs! The mock up looks great. I agree the view change control could be more intuitive/natural but this is totally fine as a place holder.

[mayacakmak]

It occurred to me that with the target EE visualization being a 'ghost' gripper, visualization of flex thresholds is going to get complicated. One idea is to change the task slightly, to something that's actually more realistic--instead of telling the user where to put the gripper, we place a 3D object to be grasped (and make it somewhat obvious roughly how it would be grasped) and the user's job is to place the gripper around that object so as to grasp it; if the object is small that corresponds to larger thresh_xy; if the object is round that corresponds to larger thresh_theta, etc. We wouldn't worry about systematically varying those values just having some variations. We would need a simple way to check if the grasp would have been successful (since we can't really simulate it, but there are some heuristics we can use). What do you think?

[mayacakmak]

TODO:

• Decide what interfaces to implement in SE3

Pasting the narrative skeleton I came up with below. Some observations:

• It would be difficult to talk about the work without mentioning accessibility--it is possible but some of our choices will be difficult to justify (e.g. why split into three views, why click and press/release options,..).
• The choice of splitting into three views is particularly tricky to explain.. I realized that conventional interfaces (single moveable view) are not necessarily controlling only SE2 in any particular view; some of them allow control of all DoFs in certain viewpoints. So our starting point for focusing on SE2 first is not super solid. More reason to include accessibility in the framing (previous bullet).
• A comparison to a conventional interface would be nice to address the previous bullet point, but probably too ambitious within out initial paper scope. We should still do it afterwards ;)
• We need a framework to assess accessibility. We have a vague sense of it based on discussions with Kavita but we need to systematize it. Ideally this would include talking to a few more people with mobility impairments. We can also look into the input accessibility literature for something that already exists, that we can either use as is or extend/adapt. Basically something that allows us (expert evaluator) to look at an interface and give it a score. Some example factors to include: is continuous input required, is continuous input while pressing required, granularity/sensitivity of continuous input, number of discrete inputs required (e.g. for someone using a switching interface)..
• Having people with mobility impairments test the SE3 interfaces would be very good to strengthen the paper--we could piggy back on participants in Maru's other study with the Stretch robot.
• What felt natural, in terms of which SE2 interfaces to transfer into SE3, in the narrative was to take the extremes (most efficient and most accessible) and one or two middle ground/compromise interfaces. We sort of know which one is most efficient, but for the others we'll need to first establish the accessibility scores for the interfaces.
• The narrative has several references to related work, other/existing/conventional interfaces -- would be good to do a thorough survey of those and characterize/code them in a few dimensions so it is easy to cite subsets of them to support statements in the narrative.

======

Robot tele-manipulation is important: it can enable applications that are currently not possible with autonomy, and can serve as a fall-back/recovery mechanism for autonomy in the long term. For them to be practical the need to get more efficient than what current interfaces offer. It is also important for them to be accessible to people who can most benefit from them; current interfaces pose several challenges.

Many telemanipulation interfaces are controlled with a cursor. This is desirable because it does not require special devices. It also makes interfaces inherently accessible since many assistive devices convert input to cursor control. At the core of the inefficiency of telemanipulation interfaces is the challenge of controlling a 6 DoF robot end-effector with a cursor that provides 2D input. Most interfaces employ a combination of three strategies:
(1) Viewpoint restriction: Interfaces involve 3D rendering of the robot that can be viewed from different perspectives possibly changeable by the user. The viewpoint constrains the meaning of the mouse inputs.
(2) Mode switching: Temporal event inputs provided by the user through mouse press, release, or clicks can trigger mode switches. Then the mouse input can take a different meaning in each mode.
(3) Screen regions: Interfaces take advantage of the ability to display guiding visualizations on a 2D screen. Screen regions can further refine the meaning of mouse events, e.g. clicking on a rendered arrow to move the end effector along a particular axis vs. clicking on a ring to rotate the end effector.

This the large design space and many variations exist, but it is unclear how choices in the design space impact the efficiency and accessibility of the resulting interface. In this paper we set out to characterize the design space and evaluate the performance of different alternatives in this space. To simplify the analysis, we first focus on control of three degrees of freedom (SE2) with a cursor, assuming a particular viewpoint restriction method. We construct nine different interfaces and evaluate their efficiency in different settings and characterize their dependence on target parameters (Experiment 1). We also present a framework, developed in consultation with N individuals with mobility impairments, for heuristically evaluating accessibility of an interface (or perhaps someone has already done this and we use their framework). We then take a subset of the SE2 interfaces (most accessible, most efficient, two middle grounds) and combine them in a 3-view interface to fully control a robot manipulator. We evaluate the performance of these interfaces (maybe in comparison to a traditional single view interface?) in different manipulation tasks, both with users from the general population and with individuals with motor impairments (Experiment 2).

[mayacakmak]

I just tried the different SE3 interfaces--great progress @kaviMD on implementing all options! They all work quite well and the interface is very intuitive. Two things to address:

Re: Ghost EE---I think the easiest would be to not have a ghost of the gripper but instead have a ghost of the "red indicator"--in turn I think it would be great if we could make the indicator slightly different: (1) make it more elongated so its directionality is more apparent; something similar to the EE in SE2 (with a ball at the center and a stick in the direction away from the wrist; or perhaps additional one or two shorter sticks in orthogonal directions to that) would be great; (2) place the "center" of the indicator at the middle point between the fingers (instead of the wrist) -- this will require slight adjustment to the IK calculations. Then for the targetdrag interfaces, you can have a ghost indicator appear before the pose is finalized and the robot gripper is moved.

Re: Flex targets. Let's keep it simple for now, the target can be a rectangular prism object or cylindrical object with one longer edge (the long edge can be longer than the opening b/w the fingers, while the other edges/2*radius is smaller, such that it is clear the object needs to be grasped along the long edge). An end effector pose "satisfies" a target if closing the gripper would result in grasping the object (we can estimate this by ensuring the line between the two fingers intersects with the object and the fingers don't collide with the object). Flex can be added by making the short edge/radius of the object smaller (i.e. more possible successful grasps).

There is still some flickering of IK solutions in tricky parts of the kinematic space; one thing we can do to avoid is have the initial pose of the robot be closer to the comfortable parts of the space and have the targets also be within that space. Typically with PR2 we raise the torso all the way up and have it manipulate objects on a table (with the torso all the way down that ends up quite close to the robot's base, so if we have a way of raising the torso that would be great). Such tabletop manipulation tasks often involve the robot's elbow being to the sides or even higher (rather than below the arm as it is now).

Minor: Perhaps we can remove the 1 - 2 -3 buttons at the top now? the click-to-activate works very well.

[kavidey]

I finished getting targetdrag working in 3d with the new IK indicator. While testing it, I ran into a bit of a weird behavior with rotation in the 3d environment.
All of the translation is relative to the view of the camera, so even if the EE is rotated 45 degrees, it will move in a straight line parallel to the plane of the camera. The axis of rotation on the other hand is local to the EE and not based on the orientation of the camera. That means that if the EE is rotated in one view, it changes the axis of rotation (relative to the camera) in other views.
I have tried to demonstrate it in this video: https://www.youtube.com/watch?v=5dtq6Mvkzcc
This can create situations where the EE can be rotated at a really weird angle that is very difficult to undo. I'm not sure if there is a solution to this problem (having the axis of rotation be based on the view of the camera creates other weird behavior I think)

I have removed the 1-2-3 buttons at the top and adjusted PR2's torso. I am still adjusting the best camera positions so that they have the best view.

[mayacakmak]

Cool!

Hmm, I think having the axis of rotation relative to the camera view is more intuitive. I assume that's what we're doing for the other interfaces? That's why we have three orthogonal views, each view provides orientation around one axis and the combination is full control on rotation in 3D. Not sure how relative to local EE would work and couldn't access the video (seems to be set to private?)

[kavidey]

Sorry about the video, that should be fixed now (https://www.youtube.com/watch?v=5dtq6Mvkzcc). Right now, the rotation is actually relative to the EE across all of the interfaces.

I can work on changing the rotation axis to be based on the camera.

[kavidey]

I have implemented rotation based on the world axis.

For testing, I added a toggle at the top of the page to easily switch between the world and local axes of rotation (the toggle does not affect translation). @mayacakmak think that you're right that rotating on the world axis is easier to control than rotating on the local axis.

Edit: It seems like there is a problem with some of the math for the rotation, I'm working on fixing it. There was an issue with some quaternion decomposition, it should be fixed with the latest GitHub commit

[mayacakmak]

Kavi's 3D interfaces demo is live here:
mayacakmak.github.io/se2/se3/practice.html

Here are some ideas for improving the interface:

• Improved visualization of the "end effector" I propose using a standard SE3 coordinate frame visualization with three color coded axes and a sphere at the center; here is a prototype. the color coding of the axes could be reflected onto some of the interfaces, as in the arrows example in the prototype.
• A feedback mechanism for when NO IK is found--that is when the EE visualization updates to where you move it but the robot's arm is unable to attain that pose. We could try making the EE visualization gray when there is no IK and the user shouldn't expect to see the arm pose match what with what the EE visualization indicates.
• The target also needs to be visualized better, it's hard to see right now. This could improve with sampling but the fact that the object is black is not helping, could it have reflective material like the robot itself to have better 3D perception of the object? Also pick a color different from the ones used for EE viz, perhaps orange or yellow.
• Zooming in each view could be more even? The target object looks smaller in one of the views.

Regarding sampling of tasks: We discussed having two objects, one standing one lying on a table, possibly two different sizes. Here's a sketch (note that T3-T5 are the same object in the same location just rotated 45 degrees and 90 degrees, shown from different views). If we have two different sizes we would make a thinner bottle and a thinner rectangular prism ('remote'), but perhaps we forget about size for now pick something medium.

• Figure out the right poses for the objects that are within the robot's kinematic reach.
• Figure out the right sizes of the objects to they are 'graspable' but not super small.
• Figure out a good starting pose for the robot arm so the targets can be seen and they can be reached without getting too much into "no IK" regions. Gripper opening facing down (i.e. for top-down table top pick up) is quite common for manipulator robots. We could also have different starting poses for different tasks to make the teleop task easier.
• The robot's gripper should be completely open, currently looks closed or only half-way open.

In addition Tapo brought up last time and I agree the IK solver might need some work--solutions seem to flicker a lot and not find solutions where I'd expect it to have one. I've been trying for a while to make the robot's gripper face down (i.e. as if it were picking objects up from the top off of a table) and just couldn't do it. @kaviMD you know the last wrist joint (which rotates the gripper like a screw driver) is a 360 joint that can keep moving (doesn't have a joint limit) - perhaps that's not represented in the IK solver? I'm trying to think of other possible reasons for non-ideal/missing IK solutions.

@tapomayukh @csemecu Any feedback or thoughts to add before we can start collecting data from this?

[tapomayukh]

Looks great! This is not as easy as I thought and it needs some getting used to :-) I am so used to rotating 3D views using mouse that adapting to this is tricky and requires me to map multiple views continuously. Note, the IK is finicky. I really wish we had some joint control or atleast some elbow control. Minor comments: We could improve the visualization by adding a different color for the arm and the gripper atleast? It sometimes gets hard to see things, especially the gripper and the target object. Can we also make the robot a different color from the background? Finally, can we have a cheat sheet to map mouse actions to interface behaviors for each interface (on the side perhaps)?

[kavidey]

I think that I have completed most of the visual changes (I am still working on updates to the IK and sampling).

One note:
The PR2 model that I am using prevents the gripper from being opened all the way. I edited the file to try and remove the limits, but I was not able to reliably increase the bounds for both fingers (I was able to get one finger further open, but the other one wouldn't move which made the gripper fairly lopsided). Overall, opened both fingers a bit more (from 20 to 30 degrees I think). Opening the gripper even a little bit made it easier to reach targets

[mayacakmak]

Nice work @kaviMD!!! I noticed some of the changes already earlier today when I was taking some place holder screenshots.

Strange issue about the fingers, but opening evenly as much as you can (i.e. w/o making it lopsided) should be fine for now.

[kavidey]

I added code to turn the whole arm dark grey if a bad IK solve is detected (i.e. if the ee gets too far away from the target in orientation or position). When this happens, the IK is also "reset":

• The state of the IK partially dependent on where it was last frame. Resetting it sets the "last frame" to the default resting position of the arm and then runs several rounds of optimization to try and find a better solve.
• This resetting used to take place once every 1500ms, no matter what, which created fairly jerky movement. It is much smoother and more efficient to run it only when a bad solve is detected.

The tolerance for detecting a bad solve and initiating a reset can be set individually for bad rotation or bad position of the EE. I adjusted it to what seemed decent to me, but if it is over or under sensitive, that is really easy to change. https://github.com/mayacakmak/se2/blob/master/se3/scripts/control.js#L106

I also updated the objective function of the optimization to take into account the previous state of the robot more efficiently which should speed things up as well as make them more accurate.

There were joint limits on all of the joints before, thanks for pointing that out @mayacakmak! I removed the limits from the two joints that shouldn't have had them (the wrist and elbow). The arm still jumps around a bit when trying to rotate around in a circle multiple times, but it is much better. With the new system for detecting (and trying to correct) bad IK solves, I found it a bit easier to get the gripper to face down, but it is still not as easy as would probably be ideal.

I tried messing around with starting locations (such as with the elbow starting up). But most performed worse than the starting state that is being used right now. https://github.com/mayacakmak/se2/blob/master/se3/scripts/3d/3d.js#L6

I can try to update the objective function to also take into account the location of the elbow, but I worry a bit about how complicated that would become / if there is enough time to correctly implement it. I think it would probably take me a few days to get it working properly.

I'm not sure that I quite understand what would be necessary for the cheatsheet. Is that something that would be covered by the video?

[tapomayukh]

@kaviMD Great work on using the previous state and removing the soft limits. For the cheat sheet, I just meant the mapping between mouse actions and interface behaviors. If you have a video before every interface test that talks about it, then great but from our previous Stretch study with people with disabilities, we found that they still need a cheat sheet because they tend to forget how to move the robot. Just something to think about.

At this point, I agree it's better not to do heavy development but rather polish things so that it is robust during the study. We don't have time if things break :-)

[kavidey]

I updated test.html to work with SE3 and finished the updates to the sampling algorithm.
There is a little bit of randomization in the size and position of the targets, but not too much that changes the difficulty too much.

Everything is here: mayacakmak.github.io/se2/se3/test.html

With the updated gripper pose and limited sampling positions, 5 targets are really quick to get done (at least to me) it might be worth it to add more targets at the end.

[csemecu]

I tried this last week the first time you sent out an update, and tried it out today. It looks really good and it feels a bit cleaner right now. I agree that this time around feels faster to complete. That might also be a learning effect.

…

On Sun, Oct 25, 2020 at 9:32 AM Kavi Dey ***@***.***> wrote: I updated test.html to work with SE3 and finished the updates to the sampling algorithm. There is a little bit of randomization in the size and position of the targets, but not too much that changes the difficulty too much. Everything is here: mayacakmak.github.io/se2/se3/test.html With the updated gripper pose and limited sampling positions, 5 targets are really quick to get done (at least to me) it might be worth it to add more targets at the end. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXGBLXMD6ARNRRE3ZS7CXLSMRHI7ANCNFSM4RC2F66Q> .

[mayacakmak]

I agree, this is great @kaviMD! the visual interface changes have really helped and the IK seems more stable.

One question still: it seems the robot declares 'no IK' only when the position (x,y,z) cannot be reached; it is okay with the desired rotation (thetaX, thetaY, thetaZ) not being reached. Is that correct? I think that is sort of unintuitive--it can feel like the orientation knobs are not working, even though the robot continues to try to reach the specified orientation as position changes. Let me know what you think about this!

[mayacakmak]

Also I still think we could be operating in a "better" part of the robot's kinematic space, one thought is the following:

.. in our tasks I keep being restricted by the wrist joint limit being on the wrong side; could the initial pose of the robot be reversed. The target objects might also need to be further down.

Also sending along a video of the PR2 to give a better sense of the comfortable kinematic space:
https://www.youtube.com/watch?v=Eo7r0ex3JT0
.. the elbow out pose in these tasks if quite good to table top or side interactions.

[kavidey]

The code should declare 'no IK' when the target orientation cannot be achieved. It looks like the threshold was set much too high. I updated the threshold which combined with the update to the IK starting position makes rotation feel a bit better.

The video about PR2 was super useful for picking a better starting position (and really cool to watch). Based on it I updated the starting position to this:

The IK still takes over a bit and modifies the position of the elbow:

But, I found that the new starting position is a bit better at getting to more difficult angles: https://youtu.be/GEMvvdNmCEM
I also moved all of the targets a bit further down as suggested.

One thing that I find difficult now is to get the arm in a sideways position like this:

One solution might be to make the rectangular targets thinner and more "remote" shaped like @mayacakmak so that they can be grasped from the top. I can also try and update some part of the IK to encourage the elbow to not stick up which might make grasping objects from the side easier.

[mayacakmak]

Excellent! Indeed, makes sense that the side grasp would be harder in that part of the space, and I would suggest exactly what you already suggested--make the object graspable from the top. The top and front grasps come up a lot with table top objects, whereas this type of side grasp would be more relevant for when someone hands over an object to the robot (which is okay for us to exclude as a task).

You could just rotate the object 90degrees around the blue axis; I would also suggest making the smallest dimension of the object slightly bigger (double?) so it makes sense as something standing on a table.

[kavidey]

The size should be changed to be closer to a remote shape:

I also doubled the thickness as you suggested.

[mayacakmak]

Looks like the starting pose of the arm has changed; could we make it like it was in the first image above (where you said you changed it):

[kavidey]

The starting joint angles (before any IK runs) is actually:

The problem is that as soon as the IK starts, it will move the arm to better be closer to the target and change the location of the elbow.

By setting the location and orientation of the target to be the exact same as the EE, I can put the IK in a position where it doesn't need to run (until the user interacts with the interface).
During testing, however, I found that movement was smoother when the elbow was a bit further up. I added a toggle to switch between elbow up and elbow down to test.html
https://mayacakmak.github.io/se2/se3/test.html

(You might need to clear your browser cache for it to update. That may have actually been why the interface wasn't updated with the new IK state earlier. I have found that when a site is updated, Chrome will re-cache the .HTML file(s) but still use your older .js files, so it is possible for some changes to not apply immediately if you visit the site frequently.)

[mayacakmak]

Ah yes, I've noticed the issue with caching. I agree it's still a lot smoother. I think this is a reasonable point to stop changes for the study; I'll make the screen capture videos etc with this version and try to finish videos tomorrow so we can start collecting data!

[mayacakmak]

@kaviMD A few issues and potential bugs I noticed while trying to make videos:

• The arm does not turn green for the two target interfaces (ring-click and target); perhaps the screen moves on before it does? would be good to delay the transition a bit so people can see they succeeded in reaching the "target"

• The "IK found" threshold still seems to be high; in the image below the rotation about the red axis is about 45 degrees off but still 'accepted.'

• There seems to be a bug in the test scene where the object is diagonal (5/5), somehow the axes get misaligned.

• I think the object in the "remote control lying on the table" scenarios (3/5, 4/5, 5/5) is still too high. It has a hard time finding IKs with the gripper facing down. Here's a screenshot of the EE height range that I think has more IK options (even though it prefers putting the arm in reverse instead of what seems more natural in the PR2 videos). I'd recommend moving them down to where they would be reached at about this height.

• With things moving downwards we can't see the end effector in the fourth view and it might get harder to move the EE (esp. with the arrow interface) so it might be good to shift all the views slightly down

Let me know if we can address these. Thanks!!

[mayacakmak]

A couple more:

• The target can get off limits in panel click
• The arm also does not turn green in panel click and panel press/release
• Can we make the cylinder radius slightly larger? it's hard to see sometimes

[kavidey]

Most of those should be fixed! Some notes:

• On top of moving targets further down, I also moved the "remote" targets a bit closer horizontally to the PR2 body to make them easier to grasp (the IK has trouble grasping from above when they are really far away)
• Fitting everything in the bottom right view is difficult to frame, so I also zoomed it out a bit
• I was not able to recreate the arm not turning green in panel click or press/release (Was this specifically on the test/practice interface or with box/cylinder targets or something like that?)

[mayacakmak]

@kaviMD There seems to be a bug with IK visualization in the target interface, it appears there is no IK solution even though the arm seems to move correctly:
https://www.dropbox.com/s/7lytzvi0nbx2prt/target_intro.mov?dl=0

A similar thing might be happening in targetdrag, but that one still found some IK solutions once other decgress started moving, still suspicious in comparison to the non-target interfaces that seem to have a good range of IK options around the start pose that I'm trying to get to here:
https://www.dropbox.com/s/9mqaun9ub6nqmml/targetdrag_intro.mov?dl=0

Let me know what you think. Also if it is easy, I think this might be useful:

• Add a button somewhere (perhaps below near the 'Next' button), for 'Reset arm' which basically places the arm back to the initial pose. Sometimes it gets into strange configurations that are hard to get out of using the interfaces. Make sure the use of the button is logged (would give us an idea of which interfaces get into more trouble).

[mayacakmak]

One more detail about the bug with the 'target' interface: changing the orientation slightly seems to fix the visualization issue... so basically every time only position is changed (through a click in free space) the IK/no-IK viz doesn't seem to be updating.

[mayacakmak]

Different possible bug:

• In the targetdrag interface, when the pose is specified in the top-left corner view, the height of the gripper (which shouldn't be controllable from that view) suddenly jumps to a much higher value (observable in the top right view). Here's a video: https://www.dropbox.com/s/tff7l0eqxy9wgyw/targetdrag_task1.mov?dl=0
  @kaviMD see if you can find anything that could be causing that or let me know if there is another explanation.

[kavidey]

The problem with the gray color for the target interfaces is due to how the arm color is updated in code. At the moment, the color of the arm is only updated when the user interacts with the interface (not when a new frame is drawn), so it is possible that when the first frame is set (after only 1 round of IK), the arm is in a "bad" state and grey, however over a few frames, it converges on a more accurate position, but the color of the arm is never updated. This should be fixed now (Updating only on interactions was originally done for the sake of speed, and works well on interfaces with continuous input, but not so well on the target ones).

I think that the problem finding IK solutions has to do with the fact that on the target and targetdrag interfaces, the target makes a really large jump in one frame. One of the things that the optimization takes into account in the objective function, is the difference in position between this frame and the last. This really helps create smooth movement on interfaces with continuous input, but can cause some problems on the target and ring-click interfaces that have big changes in the ee pose in a single frame. That combined with the fact that, like the arm color code, the bad-solve correction code also only used to run when the user interacted with the interface meant that those interfaces also had a harder time finding good solves.
I have removed the comparison to the previous frame for the target and target drag interfaces which to me feels a bit more snappy. I also updated the solve correction code to run on every frame for the target and targetdrag interfaces and tuned the number of iterations for the optimization on those interfaces to improve quality.

The button to reset the arm IK pose should also be added and working with logging (an event is added to the list of events for that cycle, and the variable reset_ik_num is updated under the properties of that cycle).

The problem with the ghost ik target jumping around up is a bit weird. Basically, the ghost ik target did not have the same starting position as the visible ik target (the ghost was roughly 5 units higher). This meant that when the ghost appeared it was in the wrong position (the top left view only changes 2 axes of the target, x and z, but the y axis was the one with the problem), so it appeared much higher than it should have and threw everything off. That should be fixed now.

Sorry for the delay in getting these fixes out, let me know if you find any other bugs!

[mayacakmak]

Awesome job hunting down the bugs @kaviMD! and thanks for explaining what they were ;) I'll remake a couple clips, almost done with videos.

[mayacakmak]

@kaviMD The 'Reset Arm Pose' button was not working for me, here's what I did--in manualResetIK() I replaced:
resetIK();
with:
setEEPoseAtCenter(); // This calls resetIK() Control.initialize(ee.pose);
Seems to be working from testing, but just noting down here in case this messes up something else.

[kavidey]

@mayacakmak That makes sense, I originally misread what the 'Reset Arm Pose' button was supposed to do. I thought that it was just supposed to recalculate the IK without moving the IK target back to its original position.

The one change that I made to the code was replacing Control.initialize(ee.pose) with Control.updateControlPositions(). In addition to moving the svg based UI elements to match up with the 3d interface, Control.initialize(ee.pose) also registers event handlers for the UI each time it is called which might create problems on certain interfaces.

Control.updateControlPositions() does basically the same thing as Control.initialize(ee.pose) without registering new event handlers, and with the addition of coloring the arm/trying to fix bad solves.

[mayacakmak]

Aha, exactly the type of thing I was worried about with calling an 'initialize' function :) Thanks for fixing that!

[mayacakmak]

@kaviMD Just to check: are the "view changes" getting logged? i.e. could we recreate how long each participant spent in each view?

This thread is getting too long :-)

[kavidey]

@mayacakmak Yes, the view changes are getting logged. As of right now, they are logged to the action list under se3/cycles/{cycle_id}/ along with resetting the arm position, any click interactions with the UI itself, and a log of the position of the EE every (I think) 750 ms.