I tried silkworms and Alvin loved them. However, they made his poo really stink and I didn't love that too much. You can buy them small and they won't be too big for your baby. However, they are definitely harder to take care of and more expensive then crickets. If they don't stay warm enough or condensation gets into the container or you touch them or their area is not clean, they will die.
I don't think I will be going back to silkworms. When Alvin gets big enough, I am going to try Superworms and in the meantime, I am sticking with crickets. He loves them and I ordered 1000 for $20 which even if a bunch die, it is still way cheaper than getting them at the pet shop.