I'm at this very point and have the same problem. You can't go back to Kakariko because the bridge is out.