If he's relaxed with your hand being in his enclosure, are you patting him then too ?
That would be next step - gently stroke him on his head and back and tickle him under the chin while quietly and calmly talking to him.
When he's allowing you to do that , you can try gently wrapping your hand around him and lifting him off his basking spot or where he is sitting and bring him out and place him on your tummy / chest with you hand still gently restraining him (good support for his full body is key here), I make sure I have hold (gentle but firm enough so he can't get away) of his tail (just placing it under your wrist should be enough) , more gentle patting , more quiet calm talking to him, ticking, maybe handfeed a yummy bug or bit of favoured green leafy stuff.
Growing lizards are VERY FOOD ORIENTATED !!!
Short nurses and cuddles to start (maybe 5 - 10min per day) and build up, I think it helps to give a food treat with all initial nurses , this will help with bonding and will help him learn you are the food giver and wont harm him ,and he'll come to associate nurses with food treats, then later he'll discover you are like a hot water bottle and that's nice (to a beardie too).
Soon, maybe a month or 2 of this and you'll have a beardie who will be crawling and climbing all over you and who will really enjoy his time out on you and you can let him "play with you" and explore you ... you aren't a real beardie person until you've had a beardie climb ontop your head (just because he can).
Tip. Make sure you get him out while in a room where he can't hide (under/behind/inside the furnature) or get hurt, or go missing. A bedroom is a good option , so is a lounge room (if you can block off any access to under stuff, you don't want to risk hurting a little lizard while moving heavy furnature around to find / get hold of a scared little lizard).
Tip2. Take your time.