Recently out gay guy here. I don't know why this freaks me out so much, but I've been thinking lately that it's mostly just impossible to figure out who's gay unless they tell you. I mean, aside from very stereotypical gay men. Every time I look at a guy, I wonder about him. I mean, not just co-workers and people at the store, but relatives, friends, even married guys with kids as you've mentioned. Anybody can be gay. Maybe someone more experienced like you can tell, but how can I tell who's gay? It's like we're invisible or something. J.H.
And that's the part that probably "freaks you out." Yes, until everyone feels free enough to come out of the closet, we are largely an invisible community -- except in certain urban areas, during Gay pride marches and festivals, at GLBT centers, and so on. What makes it even more confusing is that sometimes men who seem gay aren't. Or at least don't identify as such. [Adding to the confusion, of course, is the number of homosexuals who are not only in the closet, but are in serious denial as to their sexual orientation.]
It's possible that as a "more experienced" person I can sometimes pick out who's gay easier than somebody else, but it's never been an exact science. Like you said, aside from the cases when someone is very flamboyant and obvious, or totally out of the closet, it can be difficult to tell. Sometimes there are some subtle, almost indefinable giveaways. In my younger days I could often tell a guy was gay because I could see that he was checking me out. It still happens even today from time to time, thank goodness! There's just a certain way a man will look at another man if he's into guys ...
It helps to find out who's gay if you're out of the closet yourself. If another person, say a co-worker, is gay they'll be much more likely to come out to you if they know you're gay too and won't judge them.
But this very "invisibility" you mention is a troubling aspect of our very diverse community. Most gays men and lesbians do not conform to stereotypes. All we can hope for is that more and more people will feel comfortable coming out of the closet. The more people who do, the more comfortable the world at large will become with us. And that can only work in our favor.