The Whore of Mensa

Have you read Woody Allen's "Without Feathers"?
It is such a fantastic read, you'd be surprised. He had me in stitches.
One of the mini sections of the book was entitled, "The Whore(s?) of Mensa".
It is a story about how men go out to look for women prostituting, not their bodies, and certainly not for sex, but their minds, for intellectual stimulation.
Which brings me to a question for discussion: at which point is 'talking' considered an act of infidelity?
My cousin enlightens us with her opinion when she says, and I quote:
"Infidelity is committed when there're feelings involved. It doesn't matter whether these feelings are sexual or not, as long as the attraction is there..."
Is this true?
But then, isn't all attraction based on sexual feelings?