Even the best of churches cannot meet all the needs of everyone coming through the doors. There is room for diversity within the body of Christ and there is nothing inherently wrong with choosing a church based on preferences like style of preaching or style of worship.
I attend a great church but some people visit and determine that it is not the right church for them for a variety of reasons. I’m OK with that and I know that the leadership of my church is OK with that also.
Some attend a particular church for a while and get plugged into various aspects of church life only to find out that the church does not fit what they want in a church. Or, sometimes, the needs of a family change and another church may be better suited to minister to that family in their new stage in life. People leave churches for various legitimate reasons.
But, if the leader or leadership speaks ill of people who leave, then that church is becoming cultish no matter how orthodox the teaching and worship might be. No church leader or denomination has a monopoly on the truth. We all have equal access to Scripture and the Holy Spirit.
When a leader behaves in a way that indicates that those who left the church were wrong for doing so, then those that remain should seriously question whether that is a leader God would have them follow. To stay in that environment so that you can avoid the criticism if you leave is only delaying the inevitable.
When the leader is behaving in a manor contrary to the example we have in Jesus, then that leader disqualifies himself from leadership. It would be instructive at this point to review the qualifications that the Apostle Paul gives for an elder (or pastor).
“An overseer, then, must be above reproach, the husband of one wife, temperate, prudent, respectable, hospitable, able to teach, not addicted to wine or pugnacious, but gentle, peaceable, free from the love of money. He must be one who manages his own household well, keeping his children under control with all dignity (but if a man does not know how to manage his own household, how will he take care of the church of God?), and not a new convert, so that he will not become conceited and fall into the condemnation incurred by the devil. And he must have a good reputation with those outside the church, so that he will not fall into reproach and the snare of the devil.” (1 Timothy 3:2–7, NASB)
I would draw your attention to the command to not be pugnacious. In other words, the leader should not be looking for a fight. He should not fight with those who stay and he certainly should not fight with those who leave.
I am reminded of how Jesus dealt with Judas. Jesus knew that Judas was the one who would betray him, yet he did not treat Judas differently than the other disciples. Notice that when Jesus announced his upcoming betrayal, none of the disciples had a clue as to who it might be (John 13:25).
If you are a leader who is dealing with people who have left your church, wish them well. If you are tempted to criticize them, bite your tongue.
If you are in a church where there is pressure to stay, do not stay to avoid criticism. Stay only if that is where Jesus is calling you to worship and minister. My experience is that in such an environment, you will not have the freedom to minister as God would have you to do. But, you have to make the determination for yourself as to whether you stay or go.