If this question is targeted toward S30 Z, you can find your answer in search. The cheapest upgrade is to get rid of unnecessary weight in the car like seats, interior trims, speakers, radio, air condition, spare tire and tools, power window motors, power lock....etc. A friend of mine stripped his Honda S2000 clean and removed 200 lbs of weight, but that figure is obtained after he replaced the convertable mech with a hardtop and stock hood with a CF one.
Then come the upgrades that costs some money like underdrive pulley, K&N filter, air intake and exhaust upgrades.
Up from there is ignition upgrades and cam. After that, it's no longer "relatvely inexpensive."