I mean for the 1:5:12 format, the sign bit is shifted by 11 to account for the unused bits so I'd expect same for 1:5:10 and also 0xFFFD0 doesn’t seem to discard last 5 bits, so maybe it should be 0xFFFE0? or am I missing something?
I think it's actually the 1:5:12 compression which shifted incorrectly. I think you want to shift the sign bit by the number of bits that are ignored in total, not just the number of bits that are ignored from the left. If that understanding is correct, the 1:5:10 compression was correct to do ">> 16". But the 1:5:12 example would be incorrect. It should've used ">> 14" instead of ">> 11"
I agree that 0xfffd0 must've been a typo. It should've been 0xfffe0 like you said
I might have misunderstood the lossy compression function, but shouldn’t it be smth like this?
uint32_t compress(uint32_t d)
{
return ((d & 0x80000000) >> 11) | ((d & 0xFFFE0) >> 5);
}
I mean for the 1:5:12 format, the sign bit is shifted by 11 to account for the unused bits so I'd expect same for 1:5:10 and also 0xFFFD0 doesn’t seem to discard last 5 bits, so maybe it should be 0xFFFE0? or am I missing something?
I think it's actually the 1:5:12 compression which shifted incorrectly. I think you want to shift the sign bit by the number of bits that are ignored in total, not just the number of bits that are ignored from the left. If that understanding is correct, the 1:5:10 compression was correct to do ">> 16". But the 1:5:12 example would be incorrect. It should've used ">> 14" instead of ">> 11"
I agree that 0xfffd0 must've been a typo. It should've been 0xfffe0 like you said