I don't think it's anything to worry about either, unless it gets larger/starts to spread, then you'll want to get a Skin-Scraping done to make sure it's not a fungal infection or anything else harmful...
Just to be sure, lol, have you tried to get it to come off while in a warm bath with a soft toothbrush? They do get all kinds of junk stuck to them...