- The respiratory systems of merpeople don't have to mirror those of known mammals and fish. They could have an organ that draws oxygen from either water or air, somehow compensating for the vast difference in oxygen concentration. Salamanders and lungfish, for example, can breathe both air and water, though they prefer one environment to the other. Selkie uses such an organ as a plot point.
- In some stories, they are mammals. They could be the fishy version of a monotreme: mammals with physiology reminiscent of another phylum. Monotremes have features from both mammals and birds (they lay eggs, but suckle young). Why couldn't there be a fishy equivalent (has partial scales and a tail instead of legs, maybe spawns instead of giving live birth, but still suckles young)? Also consider that merpeople move their tails up and down, much like cetaceans (whales, dolphins, and porpoises) and like the manatees that inspired mermaid stories in the first place. And like dolphins, mammalian merpeople would have to return to the surface occasionally to take a breath. This might be associated with living near beaches, like seals or manatees, more than with some sort of The Little Mermaid-style undersea kingdom. This also handily solves the sex problems.
Analysis / Our Mermaids Are Different
Lovemaking isn't the only Mermaid Problem that writers have to solve when tweaking the physiology of merpeople to fit a story. The gills often aren't drawn, which would at first glance leave the mermaid unable to live in a deep water civilization without drowning. So an artist who has done the research has to somehow hide the gills so as not to make the mermaid unattractive to the man.