It's true that my wallet cringes at the prices for decent produce at the supermarket, especially in the middle of winter (even if eating produce still is cheaper than eating gluten-free specialty products). But this is where some time spent planning ahead really pays off: you can shop farmers' markets and farm stands in season to buy that naturally gluten-free produce in bulk. With your careful planning, some of it even will last through the winter until the next growing season.
As an added bonus, if you're particularly sensitive to gluten cross-contamination, the chance of problems with farm stand products is far lower than the (already admittedly low) chance of problems with supermarket produce (I'm speaking as someone who has had problems with pre-packaged supermarket produce, only to find out that the produce was packaged on equipment that also processes wheat).
If you can eat nightshades, the tomatoes and peppers you'll find at your local farm stand are far superior to those you'll find at the supermarket, and cheaper, too. I buy packing crates full of tomatoes and peppers for $20 a crate in August; we eat some right away, and I use my dehydrator to dry some for use over the winter (those "sundried" tomatoes make a great sauce, and rehydrated peppers work really well in stir-fry dishes).
I've also bought sweet potatoes in bulk in the fall for less than 75 cents a pound — kept in a cool place, they'll last five to six months. Regular potatoes (usually around 33 cents a pound or less when you buy a crate of them) will begin to sprout more quickly, but I've kept boxes of those for several months as well. If you have refrigerator space for them, regular potatoes might last quite a while without sprouting.
Unlike supermarket squash, farm stand squash isn't normally coated with wax to keep it fresh longer. However, I've had good luck keeping winter squash for several months even unwaxed ... and you may be able to negotiate a deal on a bulk buy. Pumpkins are a bargain if you buy them right after Halloween — you can dry the seeds for trail mix and use the flesh for all kinds of recipes (see my article on Gluten-Free Pumpkin Recipes for ideas).
Greens are incredibly cheap at the farmers' market, too — kale, mustard greens and collards are 75 cents a (generous) pound in season. I wash the greens and freeze them in large zip-lock baggies (no need to blanch them or otherwise process them), and pull them out when I need them.
It's certainly possible to find overpriced produce at farmers' markets — some of the particularly trendy organic farms place a pretty high premium on their goods, I've found. But you should be able to find vendors with very reasonably-priced fruits and vegetables if you look around.