Try testing it out during off-hours at the gym, when there are less people around. Maybe you get up extra early or go in for a late-night session. You won't have to wait around for the machine to open up, and you're free to test and tweak your form and weight management without a bro hovering over you for his set.
Find a way to challenge yourself in other ways. If you're nursing an injury or still recovering from last week's yoga sculpt, take your time! Maybe you can't deadlift that well, but you can shoulder press like a boss, so own that in the meantime. And if all else fails, sit in the back of the class. Everyone's paying attention to their own bodies, anyway -- they don't have time to critique you.
Try out the exercise in a mirror at home. Watch yourself, watch your form, and once you feel comfortable that you're appropriately completing the movement, then take it to the gym. Even if it's the most ridiculous-looking set (dive-bomber push-ups, anyone?), you'll know you're doing it right, which takes a lot of the embarrassment out of it.
Bring a friend! Going to the gym is one thing, but staying at the gym to finish your set is entirely another. Time and time again I make it halfway through my abs workout just to get bored and go home instead of struggling through 30 more sit-ups. With a gym buddy, though, you're accountable to someone else, which is total motivation-booster.
Work on your form and reps. There's no shame in improving your movements while you build muscle. Slowly adding pounds to your routine is the only way to build strength, but if you overdo it too early, you risk an injury that will just set you back, anyway. Always challenge yourself and understand that you have no idea how long Sally Squats-a-Lot over there has been training for.
Use it as your own fuel. If you're a competitive person, maybe you can actually gain something from the idea that those around you may be peeking at your workout. Show them how a proper renegade row is done, and then show them that you can double your reps if you really set your mind to it.
Have a back-up plan. We understand all too well that the slightest change in your workout routine can sabotage your entire gym session. Maybe you have to get your cardio in before doing anything else, but the StairMasters are packed. Don't panic, and don't abandon your workout. Swap in burpees for cardio machines or plop an aerobic step in the middle of the floor.
Look around! Your fellow gym-goers are the perfect kind of inspiration. I know I just said that no one's watching you at the gym, but take a few minutes to stretch out and subtly take in your surroundings. Someone could be using a machine in a whole new way or trying out a plyometric routine you hadn't thought of. Incorporate it into your own workout and see how it feels!