I've only had a beardie for a short time, but I've had other animals my whole life. I would say you have two options.
1) Give him a break for a week. When he shows that he is scared, back off. Build up some trust between the two of you that you will respect him and not make him feel scared. Be near the tank, but not reight next to it after you feed him so he doesn't think he's competing with you for food. (My baby won't eat if I'm within 5 or 6 feed of her tank - I think she worries that I am going to want those crickets and chase her off) Once he has settled down, maybe try spending a couple days gently touching his back when he's basking and calm. Once that goes well, then try picking him up once he's sleeping/going to sleep for the night. My baby is much happier being picked up if I wait for her to fall asleep on her plant. Then she will hang out with me for quite awhile.
2) Take all the things out of his tank so he cant hide and just go for it. I don't know if I would recommend this since you don't have a relationship with him yet.