This is the Cosmological Argument. I've answered it most thoroughly here, but also to some extent here, here and in a comment here. Jake and Carl Sagan have their own answers here. Look through them and choose your own solutions from that lot to either accept or argue.
I'll just add that simply because the Big Bang seems to imply something which seems nonsensical to you doesn't necessarily mean that it is actually nonsensical, let alone impossible, or that it even implies what you think it does.
- SmartLX
There's no standard atheist answer to these questions. There's really no standard atheist answer to anything. There's no atheist dogma or doctrine, no supposedly inerrant textbook, no creed to recite. Atheism is just a single conclusion. You may draw other conclusions from that one, but they're your own.
And here's what so many people don't consider: there may not be an available answer to every question. We may never know it all. What more people do realise is that just because someone has an answer doesn't mean that it's correct. I'm glad to see that you doubt the god answer even without a satisfying alternative.
The Big Bang was what happened when all the matter in the universe was compressed to one point and then started expanding outwards. It's still happening, and it's even speeding up. We've worked that much out by watching some of that matter whizz apart. Before that point, we don't know what all the matter and energy was doing.
Firstly there may not even be a proper "before" if the Big Bang started time as we know it. Causality gets a bit wobbly when chronological order and displacement are not reliable.
Leaving that aside, the creation of matter or energy breaks the law of conservation. You could make an exception and say that matter can be created by some unknown process and is then permanent, I suppose. If the law is 100% true, though, then all matter and energy has existed forever. I'm fine with that.
Before the Big Bang theory the leading concept was an eternal "steady state" universe. Perhaps there's another universe like that which is the source of all the matter in this universe. Perhaps the pre-Big Bang singularity was a discrete packet of matter ejected from that other universe. Free of the confines of its origin, it relaxed and expanded.
If an eternal universe doesn't appeal to you, why not an eternal series of finite, sequential universes? As each one "dies", the matter somehow collects and starts over. Not necessarily by a Big Crunch, as the discovery of accelerating expansion put paid to that idea, but perhaps by everything draining out when some barrier or membrane finally breaks.
We can carry on this line of thinking if you like (the next step would be to discuss how the above could satisfy the second law of thermodynamics) but my point is that there are plenty of other theories for the origin of matter and energy, one being that it's eternal and needs no origin. This is no less plausible than an intelligent superbeing with the same quality.
I hope having a few more possible answers will make the questions a bit more pleasant as they run around in your head. It won't stop them from running though, because they run through the head of everybody who doesn't utterly accept some religious answer.
Contrary to what anyone tells you, we do not know our ultimate origin, or whether we have one. This doesn't stop us from wondering, observing and theorising. Which is great, because doing all that is fun.
- SmartLX
As an atheist, I look to science to answer the questions of the universe. Do I believe in evolution ? Of course I do. That's like asking someone if they believe in gravity.
However I can't say that it's what all atheists believe. The only thing that an atheist has to have in common with another atheist is a lack of belief in a god or gods. What they think and feel aside from that is as diverse as you can get.