I think he loved the IDEA he had in his head of her, but he didn’t actually SEE her and love her for the real her. Once she felt safer and could let her real self show more he wasn’t ready for that/wasn’t ready to grow with her or just let her grow.