Layer masks are bit masks.

Consider that data in binary computers are represented by 1's and 0's.

The left-bit shift operator << takes as arguments the value to shift and the number of positions. (...00100000000 is ...00000000001 shifted 8 positions to the left). You seem to have gotten your mask example from the docs on layers, but you didn't read the comments about casting only against layer 8, not everything up to layer 8.

If you want a mask that masks a bunch of layers you could do something like:

1 + 1<<1 + 1<<2 + ...

To simplify this method for some range, you could use:

var mask : int = 0;
for(var i : int = start; i < end+1; i++) mask += 1<<i;

As described in the docs on layers, Bitwise negation turns all 1's to 0's and all 0's to 1's, thus creating the inverted mask.

var mask : int = ~(1<<8); //everything but 8

If we assume the binary representation of numbers in use, you can generate custom masks by understanding that everything is in base 2.

...0000 is 0, ...0001 is 1, ...0010 is 2, ...0011 is 3, ...0100 is 4, ...0101 is 5, etc.

A mask that is layers 0, 1, and 2 is 7, and a mask that is layers 0 and 2 is 5. The bit representing the layer is equivalent to 2^layerNumber. To get all layers from 0-12, would be 2^13 - 1 = 8191. To get all layers from 1-12, you would subtract layer 0 (which is 1<<0 or simply 1), so your mask is 2"13 - 2 = 8190.

camera.cullingMask = 8190;
//should be equivalent to
//camera.cullingMask = 0;
//for(var i : int = 1; i < 13; i++) camera.cullingMask += 1<<i;

Here is what I do for render 2 layer :

camera.cullingMask = 1 << 8 | 1 << 12;