To a large extent this was inspired by Yannik's answer; but I hope that I elaborate on points differently enough to not be redundant.
As far as I can tell, atheism (or secularism or materialism, considered separately) is a component of an over-arching worldview not the totality of the worldview itself. For example Secular Humanism is a worldview whose breadth and scope is comparable to most religions, and atheism, secularism and physicalism are just a components therein.
I'm inclined to point out how this is similar to the mono/poly theistic division: there are worldviews with many gods, one god, no god but a host or more limited spiritual agents, no god and a rejection of all spiritualism. Worldviews in the first two categories are theistic, and what Westerners usually refer to as religions. Obviously, worldviews that fall in the third or fourth categories have the feature of being atheistic. To me saying "polytheism is a religion" doesn't fit right, I'd say "This particular religion is polytheistic." In this way of looking at things asserting "atheism is a religion" is a category error.
Similar considerations hold for secularism and materialism: these are (or can be) components of any set of worldviews, but aren't themselves in the same category as religions. As mentioned in the first, summary, paragraph Secular Humanism is a world view, comparable to a religion, that does exhibit all of these features.