The 'nay-sayers' on backing tracks typically opine that backing tracks detract from the 'live' performance. Likening backing tracks to 'karaoke' seems a bit harsh, but I've heard it said many times.
On the other side of the 'tracks' are those that hail backing tracks for their ability to create a full-band sound without having to enlist all of the usual suspects, thus delivering 'more bang for the buck.'
Here's how I see it...
If you're playing a solo-gig and you know that people are going to require 'dance music,' backing tracks are a good way to go. I think that for 'dance songs' it is very important that bass/drums be present. It doesn't mean that you can't pull it off without tracks, it's just better if you use them.
I would suggest staying away from backing tracks on ballads and non-dance tunes in most situations.
The most important thing is to 'read the room.' Figure what kind of crowds like/dislike the use of backing tracks.
I favor using 'authentic' bass/drum tracks, rather than midi-generated. It just sounds better.
Get a few tracks together and make sure that you're comfortable with them at home. Try it out at your next gig and see what works for you.
If you're using backing tracks, feel free to ignore the one musician-moron that gives you grief. He doesn't matter!