Should I be worried about stone wool dust?

No! The World Health Organization has confirmed it to be bio-soluble and safe.