You can hold him as much as you and he want. Your body will keep him warm. The only REALLY important time not to take him out from under his lamp is the couple of hours after eating so his digestion doesn't get upset. I typically hold my kids for several hours and none of them have ever been any the worse for wear because of it. I like to jack up the heat before taking them out, but that's as much for me as them. However, if he wanders off and falls asleep in a cool corner, it's time to go back in the tank to warm up again.
I handle all of my beardies every day. If its chilly, I wrap them in the beardie burrito (small fleece blanket wrapped around). They love it. All of my beardies enjoy being handled and would miss it if they weren't handled.