Marriage is not a privilege conferred by voters or legislatures. It is "one of the basic civil rights of man", as the US supreme court wrote in Loving v Virginia, the 1967 case that struck down miscegenation laws, and civil rights are not at the disposal of voters.

We don't get to say who has them and who doesn't; we all have them. We call this equal protection – never a favorite constitutional clause among today's tricorn-wearing brigade – and the approval of a majority of voters on a given November day is not required...

...while marriage is a state affair, equality is a federal one. A class that faces discrimination enjoys constitutional protections that no state, and no state voters, can override.