I also struggle with this. (For background, it might even be more difficult for me in that I am in that 10-20% of readers who are visual instead of auditory—that is, I don't typically "hear" the text in my mind as I read. Many people don't even know this distinction exists and assume everyone hears books in their head while they read the same way they do.)
Anyway, when deciding whether or not to use the comma, I have to force myself to either read it out loud or to imagine someone speaking it so that I hear it in my head. I do this because, I believe (rightly or wrongly) the answer depends on how much of a pause you want the reader to take at that point. Same thing goes for sentence fragments, which are almost always grammatically incorrect, but sometimes are the right story-telling choice (say, for a noir vibe). I use those for even more of a pause.
It's easy to forget that the punctuation 'rules' are there for reasons and not to be mechanically applied just because.
The counterpoint, of course, is that there are those readers who own internalizations of grammar interfere with their enjoyment of works that don't adhere to strict grammatical structures. Folks like that always remind me of the joke: 'This is the type of pedantry up with which I will not put!'