Columbus is a great place to get married in! Some friends of mine had their ceremony and reception in the room upstairs at the north market.
It was nice.
You can do free walk up weddings at Inniswoods if I recall correctly. No chairs or anything allowed.
Wedding Venues In Columbus
Depending on the size of the wedding, the MetroParks have some nice rentable lodges and shelters that aren't terribly expensive. Scioto-Darby park is particularly good, or the one at Sharon Woods is lovely. I think rentals generally run between $100-400.
For cost savings, book your wedding/reception on any day other than a Saturday.
Do yourself a favor and get a wedding budget setup and get a wedding worksheet. It will make planning a lot easier to know your actual numbers and where your priorities are in costs.
Call the folks at the Columbus Athenaeum. Prices are very reasonable. My cheap might not be the same as your cheap but their prices were significantly lower than other reception halls in the city.
They are very flexible and can help you work within your budget.
I personally wanted something real small and personal.
We only wanted immediate family involved. They have a nice small classic aged feel at their worthington chapel. You can have it inside or outside in their rose garden.
As for cheap I don't know if you'll find much better than their elopement package. Plus they take care of a lot of hassles including invitations and rsvp's. We had a dry reception, and later that night we met up with the couple that made upour wedding party and a few friends down at Double Happiness for booze and yakitori.