Here are a few problems that you might be having:
1 - Your production configuration may not be correct. This is particularly likely if you started out with an early 3.1 release candidate, and have been updating along the way. The suggested options for production.rb changed quite a bit between rc4 and the 3.1.0 release.
Make sure that your production.rb settings include:
# Disable Rails's static asset server (Apache or nginx will already do this)
config.serve_static_assets = false
# Don't fallback to assets pipeline if a precompiled asset is missed
config.assets.compile = false
# Generate digests for assets URLs
config.assets.digest = true
2 - You may have forgotten to precompile your assets
RAILS_ENV=production rake assets:precompile
3 - You may have forgotten to restart your web server to pick up the changes in production.rb.