When to Stand Firm and When to Let Go

There's an element of my story that many people in one of my writing groups are pretty consistently not liking. Well, it's not a 'not like' sort of thing. It's more of a don't understand. Their suggestion is to change it and make it more conventional and expected and just like it's always been done. 

Yet what's the point in that?

None of them give me solid reasons why changing it would benefit my story. They say that . And I got a 'if you keep it this way, then maybe your outstanding writing would save it.'  Huh? Shouldn't my writing be top notch no matter what? Moreover, no matter what state my writing is in, I doubt that it would win over that particular person. I made a mental note to not trust that person's words. I don't waste my time on close-minded people. The rest will see what I'm going for when they see the first pages.

How do I know when I'm wrong, and everyone's right? Or when I'm right and everyone's wrong? I know in most cases that the majority rules, and I bend to that. 

However, in this case, this doesn't feel right. Why should I change something to fit someone else's experiences when they go against mine? Even when that hurts the story. I thought I was writing what I know?