I understand the weight issue, which is why I suggested the M54 and N54. M54 you would have to turbo yourself, N54 is already boosted from the factory (which I think would help the chances of a reliable swap), since both are aluminum block, and not too heavy as far as straight sixes go. If I were going to swap in a 4cyl, I think it would be a Honda K24. It wouldn't be a torque monster, but they're great motors. With the N54, you could convert to single turbo to drop some weight.
S54 is a heavy motor, which is why I didn't bring it up.