Create a custom Collider?

Hmmm. Kind of an odd question. Perhaps it might be enlightening to have a better understanding what it is exactly you want from Unity's physics engine.

Unity's Physics engine has different collider types which implement known intersection algorithms for simple primitives Sphere, Box, Capsule and Plane. Mesh colliders are be more expensive as they would have to perform in the worst case per-triangle intersections. I believe that most if not all of this is done through PhysX, but that really doesn't matter since we cannot really change how these colliders and their collisions are implemented under the hood.

I'm sorry I don't have a better answer, but because of this closed implementation, I do not believe that you can easily override the base collider if at all. I am afraid you will likely have to make do with the MeshCollider. If you are trying to do something as you describe in the linked blog post (tessellate based on proximity to the camera or something according to some noise?) you would indeed likely have to modify the mesh where you are changing it, using the renderer's mesh interface. You wouldn't have to replace the entire mesh exactly, but you would have to find, change/add the appropriate vertices, triangles, uvs and normals and because of the automated recalculations when this happens, you had best store the arrays in temporary memory, change them and then re-assign them, meaning that for all that it matters, you really are replacing the mesh.

I've implemented a system that seems to be working fairly well. It's very similar to the method I implemented in Bullet that I described in my blog. The one major difference is that the triangle processing in Bullet is double sided (collision is in both directions) whereas the MeshCollider in Unity is only single sided (collision is only in 1 direction). That required me to reverse some of my triangles, but that wasn't hard.

Here is my solution: I set up a trigger volume (a sphere) surrounding my spherical terrain, and then I overrode the trigger Enter, Stay, and Exit methods. The Enter and Stay methods are the same. They calculate the 8 corners of the bounding box and then project them down to the terrain. If any of the corners collide with the terrain, then a MeshCollider is created/updated using the 3 closest sides projected to the terrain (only 6 triangles are in each MeshCollider). So, all of the collision response is still handled by Unity/PhysX.

It is optimized slightly by storing the MeshCollider objects in a Dictionary that is keyed off of the original GameObject. That way, only the mesh needs to be updated, instead of destroying and recreating a new GameObject every frame. The Exit method is used to remove the entries from the Dictionary.

I want to add a further optimization where stationary GameObjects (ones just sitting on the terrain) don't have their meshes updated. This will require another Dictionary storing the previous positions, but that should be fairly simple.

As I said it works fairly well. I haven't seen any objects fall through the terrain, and it has no noticeable slow down with 6 objects colliding with the terrain at the same time. I'll plan on doing a test with a lot more objects to see what sort of performance hit there is.

I would post the code here, but it's kind of lengthy (over 150 lines). If anybody is interested in certain parts, I can post those.

But is there no way how to inherit from collider? For example i'd like to have a mesh collider that behaves normally in physics but i'd add an extra condition to the raycast method (for example ignoring the mesh if there is zero alpha texture mapped) so it would work with unity raycasts globally.