The Jolly Roger episode

Hook tells Ariel that Emma broke his heart. I have racked my brain trying to think of how. Was it when she dicho she still had feelings for Neal? Cause he didn't really give up much after that. o was it when she crossed the town line, that she chose Henry instead of staying with him, cause that seems a little selfish even for him. He knew Henry would've been alone with no memories of Storybrooke. So what did Emma do that broke Hook's heart?