I would like to point out that there is, indeed, a limited amount of detail you can put into the world based on what you are going to use. Would we, for instance, map out every town and city in a nation on the other side of the world from where the first story actually takes place? Of course not.
However, you do want to go into a great amount of detail for background purposes. One reason is to answer the questions that invariably come up from the other writers -- because these characters are going to talk about the world, they're going to have backgrounds and relate to what's happening the same way we do. The second reason is because if that detail isn't there, it will show. If I write a story about some legendary sword being needed to destroy some legendary shield and all I write about the would is what's important to just that story -- you'll very quickly get the sense that that is indeed all there is to the setting. The minutiae of the world may be small, but it adds up. Though, yes, the really gross details are only going to be done for those areas the story actually pertains to. Other stuff will get fleshed out later. Most worlds, like even the Forgotten Realms, build up their scads and scads of lore over time.