Why does Halide perform better than the handwritten code here? I can understand the small performance boost from assembly to Halide, given the complexity, but why are there such large gains in the Bilateral filter? Was the starting code simply not extensively optimized, or was it just the best that engineers could come up with at that level of abstraction?

I would guess it's because Halide can come up with a schedule with the best tradeoff between being cache friendly and arithmetically intensive while programmers are not able to ensure the same hand written code can run with best performance on different platforms.