Yes, there are only two bits left for z, assuming it uses 32 bit integers. (Can't run on jsfiddle. Also i don't know if all this is guaranteed to work at all with a language that is not type save.)
But the point is, you actually waste 8 bits here:
mask|=(lod & 3); // using only 2 bits
mask|=(x & 1023) << 10; // but then you shift the next number by 10 bits, leaving bit 2-10 unused.
If lod==3 and x==1023 the resulting bits would be 0b111111111100000011, with the zeros showing the unused bits.
Fixing this you get 10bits also for z with a 32bit number and it should work:
mask|=(lod & 3);
mask|=(x & 1023) << 2;
mask|=(y & 1023) << 12;
mask|=(z & 1023) << 22;
There is one more catch to know if you use signed integers for packing. Example in C:
int32_t r = 10, g = 20, b = 80, a = 130;
int32_t packed = r | (g<<8) | (b<<16) | (a<<24);
int32_t ra = packed >> 24; // beacasue the type is signed, the shift op also becomes signed
Here ra would become a negative number with all leading buts set to one, which is not what we want.
This can be avoided either using unsigned types so leading bits become zero, or using the masking we already did before.