Silicone won't hold, you need something that dries much more rigid. I have used loctite power grab, and like it very much. It dries in 24hrs, and is water based...so its pretty inert. No harsh fumes either, and the stuff has the strongest initial tack of any construction adhesive.
That said, there are stronger adhesives built for the task. The important things to remember are "low VOC", and Non-solvent based (latex based is what you want). You want something that doesn't have solvents, because the ones that do will smell for days. Low VOC is a good indicator of lower odor...so low voc items will typically have fewer fumes that aren't as harsh. Latex based (water based) glues are a good place to start. Liquid nails brand has lots of specialty adhesives, many of which are low VOC. Take a gander at home depot, and see if you can find their masonry adhesive (should provide a strong rigid bond for slate).
Loctite also makes a heavy duty version of power grab, which might be a better choice. Unfortunately, the loctite specifically mentioned for this kind of application is solvent based (though its listed as low VOC). I wouldn't trust it to cure within 24hrs...but if you can give it more time, it might be an option.
http://www.loctiteproducts.com/p/12/29/pl_ca_400_voc/overview/Loctite-PL-400-VOC-SubFloor-&-Deck-Adhesive.htm
Hope that helps.