Ultrafast all-optical switching by single photons