Try this. It just uses the minimum of the two alpha values:

dest[ptr].a = Mathf.Min(mask[ptr].a, dest[ptr].a);

If I understand you correctly, you want to apply a cutout texture, but with edges that are not 100% sharp. So you can use the alpha from the mask to control the strength with which it should be applied.

For each pixel you can apply the mask this way:

float maskAlpha = mask[ptr].a;

// Calculate the solid mask color (the mask color with full alpha)
Color maskSolidColor = mask[ptr];
maskSolidColor.a = 1;

// Now apply the solid mask color with the strength
// controlled by the mask alpha
dest[ptr] = Color.Lerp(dest[ptr], maskSolidColor, maskAlpha);

This way should even work for dest pixels that are not necessarily solid.

Optional: If you want to be able to control how sharp the cutout's edges should be, you need variables to control the sharpness (contrast) and where the threshold should be (usually at 0.5):

float threshold = 0.5f;
float contrast = 1.0f;

Then you can calculate the maskAlpha this way instead of the way above:

// Calculate a mask alpha value with increased contrast
//float maskAlpha = mask[ptr].a;
float maskAlpha = Mathf.Clamp01((mask[ptr].a - threshold) * contrast + threshold);

Hope that helps!