You should, the original outer metal piece of the stock bushing has to come out to fit either Prothane or Energy poly bushings. I've done several in the shop - it has to come out. I installed Energy poly in my personal 260Z (Goldie) nearly 10 years ago (and, yes, I cut that barrel out - with a sawzall (but, I'm careful) and, I daily drive my car)).
Please don't hit that with a hammer anymore.