Hmm. I must have messed up someting in my earlier tests (maybe they ran without BUILD_TIMELESS=1?). You're right; the resulting firmware images do not depend on the build path, so the extra CFLAGS are not necessary.
CBFS binaries (SELFs) cannot contain debug symbols, so -fdebug-prefix-map can't really affect them. If you want intermediate build artifacts (e.g. bootblock.elf and friends) to be reproducible, you'd have to add some flags like you suggested. If you're fine with just the final images it should already work out of the box.