Painting on a texture without crossing mesh triangle.

I am painting onto a mesh with the mouse. Wherever the user clicks, the current colour will be painted in a circular shape of about 32 pixels, centered around the click position. To find the position I have been using a Raycase:

RaycastHit hitInfo;
Ray ray = Camera.main.ScreenPointToRay (inputPos);
if (Physics.Raycast (ray, out hitInfo, Mathf.Infinity, planeMask)) {
    Vector2 hit = hitInfo.textureCoord;

which can be combined with `texture.SetPixels()` to change the texture to the correct colour.

My problem is this:

I want to only change the colour for the pixels which lie on the triangle I have hit by the raycast. I realize I can get the triangle index from the `RaycastHit` using `hitInfo.triangleIndex` which in turn can be used to calculate the pixel positions of each of the vertices in the triangle. But how do I restrict which pixels I want to update and send back to 'texture.SetPixels()'?

I just wrote my own barycentric calculation routine. It's not optimised and don't have error checking/handling, but it works great in my tests.

Vector3 GetBarycentric (Vector2 v1,Vector2 v2,Vector2 v3,Vector2 p)
{
    Vector3 B = new Vector3();
    B.x = ((v2.y - v3.y)*(p.x-v3.x) + (v3.x - v2.x)*(p.y - v3.y)) /
        ((v2.y-v3.y)*(v1.x-v3.x) + (v3.x-v2.x)*(v1.y -v3.y));
    B.y = ((v3.y - v1.y)*(p.x-v3.x) + (v1.x - v3.x)*(p.y - v3.y)) /
        ((v3.y-v1.y)*(v2.x-v3.x) + (v1.x-v3.x)*(v2.y -v3.y));
    B.z = 1 - B.x - B.y;
    return B;
}

bool InTriangle(Vector3 barycentric)
{
    return (barycentric.x >= 0.0f) && (barycentric.x <= 1.0f)
        && (barycentric.y >= 0.0f) && (barycentric.y <= 1.0f)
        && (barycentric.z >= 0.0f); //(barycentric.z <= 1.0f)
}

GetBarycentric just returns the same value you get from hit.barycentricCoordinate, but for any point you like. The point you offer have to be in uv coordinates. Just check every "pixel" you want to draw. If your "circular shape" is generated in pixels i guess that should work too but you have to use the pixelcoordinates instead of uvs for the 3 vertex positions.

Mesh m = hit.collider.GetComponent<MeshFilter>().mesh;
int index = hit.triangleIndex*3;            
Vector2 uv1 = m.uv[m.triangles[index + 0]];
Vector2 uv2 = m.uv[m.triangles[index + 1]];
Vector2 uv3 = m.uv[m.triangles[index + 2]];

Vector2 P = hit.textureCoord;

Vector3 B = GetBarycentric(uv1,uv2,uv3,P);

Debug.Log("hit:" + hit.barycentricCoordinate);
Debug.Log("calculated:" + B);
Debug.Log("inside triangle:" + InTriangle(B));

InTriangle is not tested, but it should work. z don't have to be tested for lower 1 since it's calculated out of x and y.

I hope that helps a bit ;) It's a pity that Unity don't offers a barycentric conversion routine. I just adapted the wikipedia formular