I would actually offer advice in the opposite direction...
I don't mean to sound, well, mean, but if he's acting afraid and puffing at your hand, removing it is only teaching him that this works to make it go away. If he's going to learn the hand isn't to be feared (beyond food), carefully scoop him into your hand and hold him/cup him if he's fidgety. Hold him for a short moment of time and put him back. Continue this and gradually increase the amount of time you hold him.
If you can, try handling him more at night when he's less energetic and ready to sleep. Considering he is fearful, I would not recommend waking him to hold him. Even now, I will pet my dragons after lights out to basically say "it's only me" before taking them out, just so they don't see shadow masses coming for them.