I like to study arguments on both sides of the God question. The idea behind “I just believe in one less god than you do” is that humanity has invented thousands of gods throughout history. The atheist and the Christian both reject these many gods, so rejecting gods by the truckload can’t be something the Christian would object to. Why then would the Christian criticize when the atheist rejects that one final god?
Some Christians aren’t impressed. To make sure atheists use only arguments that can be supported, let’s look at rebuttals from Christian authors.
1. You’re misusing the word “atheist.”
The first concern is that the word “atheist” is used too broadly. One form of the argument is, “We are all atheists with respect to other religions, and we atheists just take it one step further.”

The core problem is calling Christians (and others) atheists! …
It would be absurd for a Christian to see a Muslim and say “Oh, they’re an atheist!” For the Muslim is clearly not an atheist, rather, he/she is a theist! (Source)

Does this author propose to now turn on his heel and walk away, confident that he’s slapped down the challenge? Surely the point is obvious, and if it is stated incorrectly, he should correct it. To do otherwise is cowardly.
I’ve written about a similar problem, the (supposedly) self-defeating statement. I’m frustrated at apologists who point out a logical or technical error in a statement and then sit back, as if that were a proper rebuttal. Yes, sometimes it is, but it is at least uncharitable to not consider ways to correct it.
Let’s hope the arguments get stronger.
1b. There’s a big difference between an atheist and a theist
For a related angle, start with this well-known version of the argument from Stephen Roberts: “I contend that we are both atheists. I just believe in one fewer god than you do.” Greg Koukl twisted it by imagining saying to a married man: “I contend that we are both bachelors. I am just married to one less woman than you.”
Someone is a “married man” whether he’s married to one woman or dozens. One’s bachelor status requires him to be married to zero, just like one’s atheist status requires one to accept zero gods.
Like argument #1, this is just a semantic argument. I agree with Koukl that the focus here is on the one (or few) woman you are married to, not the billions you’re not married to. His parallel to our statement is clear: the theist’s focus is on the one or more gods you believe in, not the myriad you don’t.
This handwaving does nothing to respond to the point that the Christian, like the atheist, happily denies the existence of all gods … but the Christian makes one exception. Why is the exception justified rather than special pleading? Doesn’t the Christian have the burden of proof to justify their position?
2. But pretty much everyone throughout history has believed in God!
Yeah, and pretty much everyone throughout history has believed in superstition. They might’ve seen evil spirits behind a death or a famine; they might’ve seen good spirits behind a military victory or good harvest. The longevity of those superstitions doesn’t mean that they’re correct or that we should privilege them in any way. And when we evaluate them, they don’t hold up. Does Christianity hold up any better?

While they differ on the specifics of what God is like … if everyone has some idea of God, then our question should be what is God really like, not does God exist. (Source)

In the first place, belief in god(s) has been widespread, not belief in God (Yahweh). And in the second, different religions can’t even agree on how many god(s) there are, what their names are, or how to placate them. The amount of harmony here is meager. Sorry—if you can’t agree on the fundamentals, that looks like just another superstition.
3. God is like a person about whom we have differing input. That’s no reason to doubt her existence.
This argument compares the world’s many competing religious views to the many competing views of a particular woman. Person A is married to her. Person B is her subordinate at work. Person C is an ex-boyfriend. Person D is a pen pal and has never met her. These and many other people know her in different ways and would have differing and perhaps even contradicting stories to tell, but an outsider wouldn’t conclude that she doesn’t exist.
This is true, but the analogy is poor. This is all commonplace. We know about people and about the many ways we can incompletely understand them. A claim about the supernatural (which has yet to be demonstrated) is a very different thing. “Suppose that a woman exists; now suppose that a god exists” doesn’t work.
A story about a woman that claims to be factual? The default is that that refers to an actual person. When that assumption turns out to be wrong, we have a man-bites-dog story, like the mysterious girlfriend of Notre Dame football player Manti Te’o. That is a bit closer to the god claim, but even there we’ve got a long way to go.
4. Christianity is real while the other religions from history were mere social conventions.

People did not really believe in Shu, Nut, Hercules, Baal, Wearisomu, Enki, Utu, Diana, and the like in the same way that people believe in Yahweh. Their belief was more of a social convention which included all the pressures that such a system demanded. Their gods were more “faddish” than anything else. (Source)

Remember how the god of the Old Testament changes. God was just a dude who walked in the Garden in the cool of the evening to chat with Adam and Eve, but later he said, “No one may see me and live.” God had to send out spies to get intelligence about the status of Sodom and Gomorrah, but now he’s omniscient. Who’s faddish now?

Their existence was rather fluid, changing and even morphing into other gods.

That reminds me of how the omniscient and unchanging Yahweh demanded rituals for the Hebrews to follow. Later, he threw out that rulebook and said that belief in the saving sacrifice of Jesus was the way to please him. You’d think he’d get it right the first time.
Continue with part 2.

“I don’t understand how you don’t believe in God.”
Well, you know how you don’t believe in Zeus?
Like that.
— Ricky Gervais

Image credit: Chris, flickr, CC