Absolutely not. We all have to live within our own comfort zones and build trust in a way that works for us as part of a couple. It is great that you are able to visit Escorts with a clear conscience and not be hiding what you do.
I personally feel comfortable in a monogamous relationship, so deviating from that is very difficult, but I'm in no way expecting anyone else to do the same or feel the same as me, as I am unique. It also doesn't mean that I keep within my comfort zone, as sometimes you have to bite the bullet and look beyond the here and now to gain a future.
I wouldn't question it as long as you are both happy.