With one that young and small, you don't want to rush it. It's good that he's not scared of you -- you're off to a good start. You could try handling him for a few minutes each day, and when you go to pick him up make sure you approach from the front or the side (not from behind or above as a predator would do), and try to support as much of him as possible when you lift him -- they get nervous when they don't have all 4 of their feet on something solid -- then settle him on your chest and keep your hands over him so he doesn't fall. If he gets wiggly and doesn't seem to want to be held, best to put him back in his enclosure and don't force it. Bedtime when he's sleepy is also a good time for cuddles and "bonding".
Hopefully you've got good heat and light for him, including UV -- that's all important for a growing beardie, and you don't want to have him out of his enclosure (and out from under his lights) too much because he will need to spend enough time basking and making Vitamin D.
Hope this helps at least a little.
