It is possible to do configuration management right but it is also possible to do it wrong. I'm going to explain things in the context of chef but there is nothing here specific to chef. The other configuration management systems have the same analogs.
One way to do it wrong is to move the provisioning and deployment logic into various cookbooks and isolate them from the applications that depend on them. The reason this is wrong is because to understand how the application works in production you now have to chase down dependencies somewhere else and once you get there the cookbooks will not make much sense because they will make assumptions about the application. This is all a long-winded way of saying you are keeping track of two contexts when in reality there is really just one context, the application and its associated infrastructure logic as a single unit.
Once you view things in one unified context you realize that the separation doesn’t make much sense because neither the cookbook nor the application can stand alone. This means that your deployable artifact must either include the cookbooks or it must depend on something that does. The simple solution is to bundle everything into one artifact by including the cookbooks alongside the application. In the past I’ve used Librarian-Chef but I hear good things about Berkshelf so either one should be fine. The whole thing is tied together with either a cheffile or a berksfile and once the deployable artifact is unpacked on a production box you just run the cookbooks with chef-solo and an associated configuration file (
solo.rb) during the post-install process. Everything lives in one repository or at least there is enough meta-data to make it seem like everything lives in one repository and the context for everything is obvious because you go to just one place to figure out how everything fits together. There are many benefits of this approach not the least of which is allowing one to trivially set up local development environments because everything is localized and does not require a central orchestration server.
In conclusion, do the right thing and keep the application context with the application instead of the chef/puppet master server. This breaks down when working with distributed systems but that's a post for another day.