This is just life. Live your life and deal with this shit-it will not go away. Being yourself does not mean you have to broadcast anything, but it also means not having to worry about shit like this-it will happen anyway regardless of what you do or don't do-you won't please everyone. Do you and let the chips fall where they may. Often when things go bad at one place it is for the best-you just never know with life. It is not worth the thought or worry-that shit ages you inside and out. JMHO.
Concentrate on doing a good job and the decent people will judge you on that, not on your personal life. The rest, fuck them. Again, JMHO.
*also, if you know someone already has 'suspicions', then that one, and the ones who will judge you negatively are already gossiping and judging you now behind your back-you can almost be sure of that. That is how those kind are. You probably won't lose much. And the worry about what these people think, them 'distancing' themselves from you, is causing you to 'distance' yourself from other gay guys who are not your enemy. Is that what you really want? Just something to think about.