In my language - Swedish - we use two genders called neutral and real. Like German and Latin languages (and unlike English) our genders have a large effect on the forms all nouns take: neutral has different indefinite articles than real has, the definite forms (a Scandinavian peculiarity) work differently depending on gender and so on.
However, we also have words that correspond directly to e.g. the English actress/actor (skådespelerska/skådespelare), and we have separate personal pronouns for he/she/it.
Still I think in general we consider our language to have two genders, since neutral and real is about how articles and rules apply to all nouns. A word like "actress" isn't seen as having a female form, rather it is a separate word in its own right, with the meaning "female actor".
You can look up "actress" in a dictionary, and it will have it's own entry, right? A word form dictated by gender would rather not have its own entry, but be understood as a different form of a base word which you could find in the dictionary.
Also I think, if a language has gender, and a word has a particular form in that gender, the general rule would be that that word should always, without exception, take that form for that gender. If a person said it otherwise, people would assume that that person didn't know the language properly. I.e., if someone called Elizabeth Taylor an "actor", it would just sound weird. (maybe that's the case? I'm not a native English-speaker)
Edit: forgot to add my conclusion - I'd say English has a single gender