SFWG Study Hall

In my books (at least ones I'm currently working on), most of my main characters are not human, but of a race I created - Damon. However, I didn't want to throw lots of terms and words for my readers to memorize so as to pick through my stories. I wanted them to be able to read comfortably so I used common references like "woman" for female Damons and "man" for male Damons. At first, no one questioned it or commented on it, but lately I've had like two people question it and basically say that I should probably create a term for a female Damon and male Damon etc. But then you also have fine lines of children, babies etc. Plus I do have elves and humans running around even though they are not focal.

Thoughts? Opinions? Advice? Warnings?

-Renegade