Why did Harvey Milk and Scott Smith break up?

From what I got from the movie, Scott didn't like that Harvey was completely absorbed in his work and had little to no time for them. While Scott was all for changing attitudes and policies and supported Harvey running for office, it seemed like he just felt that he didn't want to be on the front lines of it like Harvey did. That's how I remember it, I could be wrong. I've only seen the movie once and that's all I have to go off of. I don't know what was changed or not changed for storyline concerns.