You're both right.
The initial temperature and rate of cooling will determine the micro-crystal structure of the metal, and the change in volume as the metal cools can result in internal stresses in the material, or in the part "pulling" as it cools. This movement can work-harden the metal in the weld area.
Some metal alloys (like tool steels) can go through many phase changes as they cool from the liquid state, and a fast quench can lock in harder structures (e.g. martensite) as the metal crystals are denied the time required to reorganize into softer structures. Some of these harder structures can also be brittle, and there may be an advantage in heat treating the material and allowing other crystal structures to form.
Work hardening occurs when the ordered crystals within the metal are shattered during deformation (e.g. bending soft copper tubing, or rolling out a metal plate). These shattered crystals then resist movement better than larger crystals because the slip-planes within them are no longer continuous, and the atomic bonds are under internal stresses. Reheating the metal to a high temperature and allowing the metal crystals to reorganize will ease these internal stresses and soften the metal again and is called annealing.
Some alloys used for rolled stock get their hardness from being cold formed during manufacture, but they become locally annealed by the heat of welding with resulting weakness in the heat affected zone. With some base metal/filler metal alloy combinations, part of the heat affected zone is immediately work-hardened as the filler metal cools, deforming the surrounding base metal and strengthening the finished weld. In the extreme, the movement of the cooling filler can overstress the joint and crack the weld.