The way you describe the problem, I think there is corrosion on the contact points of the electrical system.
Clean all the electrical connections you touched and put some di-electric grease (like $5 at napa) on the connection.
Oh yeah, be sure to disconnect the battery when cleaning, especial around the fuse box.