PROJECT CONCEPT
Mano Italiano

I am Italian and I have a tendency to talk with my hands. A lot. And because of this, I have knocked over my fair share of glassware and liquids that get in the way of my physical talking. No discussion ends dry, shatter-less or unstained. I want to create an interactive piece with unity where one can talk and see just how destructive their hand gestures can be.

TOOLS:
Unity
Leap Motion

STEP 1:
Download the SDK for Leap Motion to Unity.

STEP 2:
Explore the steps needed to set up the connection. Unity has a launchpad for this.

STEP 3:
Get assets. The assets provided are for using unity with Oculus. I don’t want to make this a VR experience. I want this to be based in reality. So I had to search for another way to go about this. Basically every documentation I find is referring to the necessity of an oculus.

Not having much experience knowing what an SDK is and how it works, caused a bit of a slowdown in up-loading it into unity. But I did find this lovely tutorial. It took a bit to realize that the assets and the SDK are not related.

Assets installed and opened in a new unity project – CHECK.

STEP 4:
Get something running.
The next part of this tutorial calls for:

Every development and client computer must also install the Leap Motion service software (which runs automatically after it is installed).

At this point I have downloaded so many different things to my computer for the leap motion, I have no idea if I have the right software. And there is no link for it on the tutorial. It might be this. Turns out that this is something that I have already downloaded and it is maybe referring to the SDK. Again…have no idea what or how to apply this package and why it is necessary. I searched in my files “Leap” and got this list. I opened the software with the icon of the leap motion next to it, however, there was no feedback as to whether or not the software was running, (bad UX), although it does appear in my Activity Monitor as active.

I went back into unity  and added “prefab –> handmodels –> rigid hand round”

I am seeing hands in the preview, but despite how much i move the camera and where I move the object, nothing appears when I press play.

I did some troubleshooting by opening up the Leap Motion Visualizer to check to see that the hands were getting detected, and they were. The issue lies with the plug-in for the Unity software, which isn’t surprising, being that the documentation and tutorial are a bit unclear as to exactly what has to be pulled in (learning the importance of good documentation with this).

I checked the console and found this error:

I am guessing this means that there is no detection of hands.

I went back through the trouble shooting options:
I have pulled in basically every hand control option. This may be an old tutorial on the Leap Motion site, as that exact item title is not in any asset folder.

I started a new scene and imported the hand controller and the hand options again:
LeapHandController
RigidRoundHand_L
RigidRoundHand_R

I then brought in a cube because I know how to work with the cube in the camera view. I tried to line up the hands, and that did not work. NEXT, I played with the scale of the hands and it still did not work.

STEP 5:
Sniffing through the SDK
I found the following – start-from-scratch tutorial:

I brought in the LeapHandController, set the scale to 5 and placed it where the cube was, so I know exactly where it would be in front of the camera (and then moved the cube behind it.

After deciphering #5, I added the RigidRoundHand_L.prefab and RigidRoundHand_R.prefab to the LeapHandController’s Hand Pool (Script) settings in its inspector.
When adding, i kept getting this pop up message, but since i was doing exactly what the tutorial called for, and don’t know what this could possibly mean, I ignored it, assuming this is what was supposed to happen, because there was no mention of it in the set up.

However, when I went to play the program, the error came back up.

I moved the Rigid bits into the folder of LeapHandController in the hierarchy. Because that is what I think making it a child means. That didn’t work.

I thought maybe these models belonged in the physics section, so I moved them. I got no pop ups, but this didn’t work.

Finally, I played with scale, and tried the trouble shoot of pausing and searching in the scene, and the hands were nowhere to be found.

I checked the console and the latest message is:

Trying to trouble shoot this, I looked at the script attached to the LeapHandControl and didn’t see anything related. And the internet searches led to not much help.

I went back to the tutorial and something seems a bit funny:

Set the Left and Right Hand Graphics Model properties of the HandController to the desired prefab from the HandModelsHuman or HandModelsNonHuman prefab folders. (There is generally no need to change the physics model from RigidHand.)

There was no physics model on there, and no “RigidHand” in the assets package. This may be the problem.

I unplugged the leap motion and no matter what I do, it no longer tracks within its own program.  Great.

STEP 6: Start New Project

-import assets
-add LeapHandControl
-Child the hands
-add the hand prefabs to the LeapHandControl Graphic options
-Play with the scale and sizing

Not working.

I don’t understand why I can see a cube in the camera before, but the hands just don’t show up after playing with everything and changing the scale.

Also, although the graphic hand prefabs are child-ed to the LeapHandControl, they are not moving in relationship to each other and the x, y, zs are all different.

5 hours into this and nothing is working.

Also, all of the prefabs that the package and tutorials say should be in the assets folder are not there.  For example, there is no “HandsModelsHumans” which i thought was the alternate option for “HandsModelsNonHuman” which I was using as the graphic. Perhaps this was my issue.  So next I tried adding the CapsuleHand_L and CapsuleHand_R to the graphics models and used the RigidRoundHand_L and RigidRoundHand_R in the Physics models.

I might be closer.

And then…mystery solved! I was told there was a newer assets package than the one I was originally directed to (multiple times) by all of the Leap Motion tutorials that I was following. I should have stuck with the original tutorial that I found instead of going down the worm hole of the internet looking through Leap Motion links to tutorials. And lesson learned: The Unity site has the most up-to-date information, not the Leap Motion site.

STEP 7:

I imported the new assets package.
Lather rinse repeat.
It worked in 2 minutes.

Well that was a fun adventure.

STEP 8:

The challenge at this point was to align the camera settings with where I want the hands to be and to play around with scale. I played around with settings then gaged my scene settings with the scenes of hands that came with the assets package, to get the alignment correct. I had hands waving in space at one point. Need to work on perspective.

I chose the hand set that I wanted to use (Not quite as dainty as I had hoped, but we can all use some man hands now and then. I feel like this game is turning into an episode of Seinfeld. )

(I never had to use the SDK, and I don’t know why.)

STEP 9: Scene Dev
I have found the assets that I would like to use to set up the scene:

Breakable Glassware
dishes
table and chairs

STEP 10: Test Interaction
Before loading in and creating the actual stage, I wanted to test out the hands with interaction. I added a cylinder. Since having gravity on it caused it to fall right through the scene, I took away the gravity, adjusted the mass a bit, and moved it with the hands. It worked!

 

Notes:
Kinematic allows the hands to go through the object

STEP 11: Integrating new elements

I understand that the HandControlSandbox offers this great arena to play in with cubes and hands, however, I want to try to create my own set.
-Added a “plane” 3d object with rigid body.
-changed the cylinder 3d object to have gravity so that it would fall and hit the plane and not go through or float into space.
-pulled in a jpg of a flooring for temporary use as the material for the plane. (added it to mesh renderer – element by drag/dropping onto the plane in the scene window view)

STEP 12: Constructing the walls
I am not sure what is best to use for the walls of the scene…so I tried adding a plane as the wall with a box collider component. Just to check, the scenes in the assets folder have similar composition. I dug into the HandControllerSandbox feature and saw that this, too was made up of planes, creating walls with box colliders. Great. Sticking with it and creating the base of the scene, playing with scale for the interactions at the moment.

The walls (planes) are all set up. The question I have now is, why is the one wall showing up as gray, and the others as cream? They all have the same physical features at this point. I played with the lighting as well, as I thought it might be the lighting, and that had no effect on it.

For the sake of prototype, I decided to expand the size of my walls. I made them 50 in scale for x, y & z.

STEP 13: Tweaking hands for now
I wanted to try to rescale the hands and found this reference page Leap Motion. Transform Scale – changes size of hands
Hand Movement Scale – increases range of motion without changing the size of the hands.

STEP 14: Import Furniture
Although the warning said that the assets were designed for an older version of unity, I wanted to try my luck and import this furniture pack.

TABLE:
I imported the table, added material to it from the assets package, rescaled the item and placed it in the proper place in the scene. I readjusted the camera for the scene view and moved the HandController to the the location where it can detect where the hands are. I added a box collider.
(I also changed the material of the floor plane from the materials in the demo.

At this point I found the right settings for the hand controller movement scale and the transform scale, to interact with the cylinder on the table.

CHAIR:
I added a chair, adjusted the scales. I added a box collider. It was too big. So I adjusted the scale on that. It didn’t seem to be too effective so I removed the box collider. But it worked better when I had it on.

SET DESIGN:
I started playing with adding elements to a wall. To add a picture frame, I put a 3D cube, changed the scale, added a box collider, threw it up on the back wall, and added an image of my Grandmother. Time to make this place more paesano.

I imported these plate assets to set the table. I added two dishes, resized the scale and added the box collider.

I also noticed that the cylinder kept colluding with the table, so I adjusted the size of the box collider.

TROUBLESHOOTING:

There do remain some items at this point that I have to figure out.
-The cube item went shooting right through the chair in a place where the box collider was not covering. How can a box collider cover only the dimensions of an oddly shaped object? OR do I need to create multiple ones to cover it?
-I also noticed that from the perspective of the camera, the Y & Z axes. Could that be – and is there a way to fix it? (highlight all and change rotation?)

*Leap Motion Complaint: It is not compatible with paws. This is an atrocity.