So a few years back, I bought an old scifi book about a planet with one dying sun and a smaller sun. The main character was a male and he had kind of adopted one of the females of his species that was underdeveloped and everyone else in his village thought it was a bad idea and thought she should be left to die. They all had a generational memory that for some reason she didn't have and they like... Emerged from the sea instead of being born. The main character was like the one who built the houses. They all had to serve a purpose. I'm trying to figure out what book it was. It wasn't 1990's or newer so I'm guessing 70's/80's. Don't remember the author. Any help greatly appreciated.