I think ambiguity can be really powerful, and the mind can usually imagine something worse than the explanation provided. Lovecraft is interesting in this respect: sometimes his ambiguity is really effective (the phonograph recording in “The Whisperer in Darkness”, say), but other times he just sounds like a parody of himself. Likewise David Lynch is usually pretty good but I just found
Inland Empire to be nonsense (although the "Rabbits" short film is quite unsettling).
Actually, “Rabbits” is an interesting example of this. It basically makes no sense and has no explanation, but it’s disturbing for a number of reasons. The dialogue is ambiguous and meaningless, but threatening (“I will find out one day”). Elements that should be comforting are used out of place, making them confusing: the characters are dressed as children's characters, and there’s a laugh track suitable for a sitcom. The music sounds like something from a film noir: I don’t know much about this, but I suspect that it uses minor chords to put the listener on edge. And every so often, red lights come on and a character speaks in a distorted voice, as if possessed. The problem with “Rabbits” is that it has no real meaning, and appears to have been created purely to unsettle. The ambiguity makes you want to know the truth, but there is no truth. So I wonder if ambiguity is most effective when it is genuinely hiding something.
I think a lot of ambiguity derives strength from the idea that you are seeing a key to accessing a terrible truth, or a form of that truth that is bearable, where it would be too awful if properly revealed. Basil Valentine’s alchemical drawings have this effect on me. The instructions to summoning a demon are often more sinister than the demon itself.
That said, there can be something unsettling about a complete lack of ambiguity. The Terminator – more specifically, the robot skeleton – is frightening because it’s so blatant and relentless. There it is: a great bit metal Grim Reaper, and it won’t stop. Likewise the villain from
Marathon Man, who is obviously a vicious Nazi from the word go and never shows any hint of humanity. But this is rarer than ambiguity and perhaps harder to pull off.