After this year's election result, many believe the party has not only lost the support of swinging voters, but it's base too.