Grounding Language with Visual Affordances over Unstructured Data