Well, the Divine Comedy (particularly the first part, Inferno) is rather dystopian considering that it has a large cast of people, some of them named, who are going to continue to be tortured until literally the end of time...
Similar remarks probably apply to Paradise Lost. Of course, neither of these are exactly SF by modern definitions. And, technically, both being poems neither of them is a novel either - although they are of novel length.
Plato's Republic and More's Utopia are rather dystopian to modern eyes, too; both of them could be called early SF if one is prepared to stretch the definition somewhat.