Is it just your brother he's scared of, or is your beardie scared of everyone other than you? Has your brother ever done any teasing to him, or ever been rough while handling him? It can also sometimes be caused by Cologne, a hat he wears, sunglasses, etc.
I agree, have your brother hand-feed your beardie his favorite live insect or favorite veggie or fruit (usually insect treats work best, like live wax worms or superworms) every day. Have your brother sit on the couch with the TV on and a blanket on his lap, put your beardie on his lap on the blanket, and have him hand-feed your beardie the live insects and/or veggies until he doesn't want anymore, then try to see if your beardie will sit on the couch wrapped up in the blanket with him. If he likes to be stroked or petted he can try that after he gets more and more comfortable with him. Do it every single day and quickly he'll associate your brother with good things...