I am not familiar with exporting on Maya side, but you could try directly importing the maya(.mb/.ma( file into unity. If you haven't seen this already http://unity3d.com/support/documentation/Manual/HOWTO-ImportObjectMaya.html
answered Jul 06 '10 at 06:01 PM
I haven't tried exporting a character with it's control rig because I didn't want to throw all kinds of node garbage to Unity, so instead I used a salve rig that only has the deformable joints.
In my scene with the rig, the actual geometry being shown and deformed is binded to the slave rig to ensure I'm not using a control that doesn't do anything to my character. I then bake the animation to the slave rig, create a character node and export the animation clip.
Then in a GameSkeleton scene which only has the geometry and deformation joints (slave rig), you can import all your animations. You can either put them all in one timeline or as I now prefer, as separate files with the @ naming convention, which Unity reads and splits automatically.
Unity seems to have trouble reading maya versions above 2009, specially with the fbx, you'll want to stick to the 2009 version, which you can choose in the export dialog.
EXTRA TIP: Try using as few joints as you can on your slave rig, though your animation rig can have and probably should have more joints. (for example. you can eliminate the tip joints unless you'll be using locomotion). I also tend to get rid of other root joints like clavicles and sometimes hips on multilegged characters. You can still use them as controls, affecting your slave joints, but without the extra cost.
Hope that helps!
answered Jul 13 '10 at 06:26 PM
Oh and also, Maya's fbx is faulty in that if you want to correctly transfer a character, even to motion builder, you need to have your character bound to two root joints, even if the second one isn't weighted.
You can tell in Unity if you select your mesh and if it shows the preview or not. When using two roots, the preview should be there and everything should behave as expected. If not you'll end up having to assign shaders and such to your root joint in the hierarchy which is kind of awkward.
answered Jul 13 '10 at 06:32 PM
I'll throw my hat into the ring aswell--
It looks to me like your error doesn't have much to do with your rig, but the mesh itself "Unsupported Operation on Mesh Vertices"
I only dabble in Maya during my TA duties, and am a Max user personally, but perhaps if you flatten your modifier/operation stack as much as possible you'll be able to eliminate the issue.
As a tip, I found that letting Unity do it's thing and import the .ma/.mb itself gave exponentially better results than doing manual exports. (This was from Maya 8.5)
answered Jul 13 '10 at 06:42 PM
Not sure if this helps, but straight from the Unity3d Documentation:
Unity currently imports from Maya
Unity does not import blend shapes. Use Bone-based animations instead. Unity automatically triangulates polygonal meshes when importing, thus there is no need to do this manually in Maya.
If you are using IK to animate characters you have to select the imported .mb file in Project View and choose Bake IK & Simulation in the Import Settings dialog in the Inspector.
In order to import Maya .mb and .ma files, you need to have Maya installed on the machine you are using Unity to import the .mb/.ma file. Maya 8.0 and up is supported.
answered Jul 13 '10 at 10:48 PM