Actually he was a maverick who spoke his mind and did what he thought was the right thing, notwithstanding party opposition, which is why he didn't get re-elected.

No one considers him the worst president ever. That position would be taken from among Andrew Johnson, Richard Nixon, or Donald Trump.