Don't use physics for everything; hack it. We came up with a great solution for Castle Smash (http://apps.facebook.com/castlesmash)

basically when you spawn a rock on a slingshot, it spawns it at a certain point (referred to from now on as the origin) and then you drag that rock back. There are some functions that use a bit of trigonometry to get the diagonal length of the pull-back. When the user lets go, the object points towards the origin, and then the physics engine activates, and a force is added to the object, proportional to how far back the user pulled.

woiks like magic!

It won't work remarkably well no matter what you do. I'm (once again) only guessing, but I imagine from the complete absence of fluid physics and the poor cloth simulation, even Unity 3.1 is using a comparatively ancient release of PhysX. I wish someone would help me verify that though, so I know in the future whether they're using old PhysX or just using new PhysX badly (because cloth issues and no fluid is the least of the odd problems, like extremely poor performance compared to PhysX tech demos). :)

The only setting that comes to mind is the cloth setting, I forget what it's named, but in the tooltip it explains that it applies force back on anything that collides with it. I think toying with that setting is your best bet if you really insist on using PhysX cloth as a slingshot.

If you can settle for an alternative, you may want to try stringing together multiple rigidbodies in a chain with elastic-y joints (by playing with the spring and spring damping values). It wouldn't look all pretty and deformative like a single piece of cloth, though. At best, more like a necklace. :P