I dislike baby as a pet name in general, so regardless of genre it pulls me out of the story. I did read one high fantasy where the pet name was spitfire and that was far more jarring to come across than baby would have been though.