As I understand it, the term 'grammar' denotes the conventions of word order, grouping, and alteration, to indicate the ways the words in any given utterance or piece of writing relate to each other, and facilitate comprehension, thereby. Insofar as apostrophe and the so-called possessive ess are concerned, punctuation may be involved in grammar, but only in writing, as you rightly observed.
However, you can hear punctuation and do hear punctuation, every time you speak or listen. A lot of people seem to think that it's purely decorative and therefore unnecessary; but it's actually meant to be a written representation of a spoken language's prosody: the characteristic ways in which pauses, stresses, and tones are used to group words into easily-understood groups, instead of being delivered in a headlong burst, like suppressing fire. Good punctuation is essential to natural-sounding dialogue, inasmuch as it represents prosody.
And spelling is spelling.