I love fedoras. Several years ago, I used to wear them all the time. Then I came to a realization: I looked like an idiot. There is a time for everything, and the time for the fedora is not all the time.
Times when it is acceptable to wear a felt fedora: You are in a suit or otherwise formally dressed. It is preferably raining, so you have an actual reason to be wearing the hat. Preferably you are old and your fedora was purchased in the 20's, back when people actually made decent hats. Most made today are crap. A good fedora should be able to be folded up completely and shoved in your back pocket when you walk in doors. When you head back outside, it should pop right back into shape.
There's a bit more flexibility if the hat is made out of straw and it's a very hot day. Kind of. Due to modern sensibilities, you'll still look somewhat silly, but at least the sun will be out of your eyes.
I'll leave you all with this link:
http://putthison.com/post/365446401/hatsdouchesanddorksPTO is a great site in general, I recommend it. But in general, if you want to wear a fedora, rock it. Wear it with shorts, tube socks and a tank top (Note: don't do this). Just don't be surprised if people think you lack any fashion sense.