Dan - Yup you got it buddy - the engine is out a early '90s FWD Corolla turned around and mated to my stock tranny to make it RWD. No mods required except shaving the right engine mount a bit with an angle grinder to clear the supercharger.
My car is a little heavier as its fully optioned (electric sunroof etc etc) and I have some pretty comfy but heavy Supra seats in it. The car with a full tank of gas weighs in at 2340lbs, which is not bad. I hope to remove the heavy North American front bumper (with the big re-bar) and replace it with the feather weight Jap Spec version. So that should drop another 30lbs or so.
Yes the car was one of the most popular cars in Japan for circuit racing and drifting (still is) but the S13-S15 Silvias are now the king as they have much more powerful engines with the SR20DET and an awesome suspension and factory LSD.
Overal like you say, the potential is huge. These cars are becoming more rare to come across now, as most have fallen victim to the rust bug, but the odd time you will see one for sale and they are really good buys with Toyota reliabbility and that sweet reving 4AG engine mated to an a great transmission and a factory 4.3 rear end.
Here are some sweet jap spec versions
Yasin