In this case a reply informing you about the missunderstanding would also work. My hope is that one of the people downvoting will take the time to do that when there is no downvote button.

I’m torn on the whole no downvote button thing (I missed that when I signed up), but you can be still nice while having a discussion and disagreeing with people.

I would argue that the downvote button can lead to exactly what you are describing though, no disagreement or conflict.
Someone posts a unpopular opinion, a bunch of users downvote it to hell and poof!, no discussion or exchange of ideas. Just out of mind, out of sight.