Slow rate of power going down.

So I made a regulated linear power supply with minimum components, transformer-caps-resistor- lm317-çeramic cap- resistor- pot and last a cap. It works but when I turn down the power it goes Really slow. I havent testetd with a lod, only multimeter could that be the problem?