Next thing I figured out- To remove the fender flares to check for rust...
Preface: During the past week I've been watching videos from this guy;
http://www.youtube.com/watch?v=0W0a1sNq ... hfmFxz0%3D
He's works repairing really rusty vehicles and one thing he often finds is rust behind plastic moldings...
So today I took off my fender flares.
Dirty, but ok
Each rear flare was filled with a few pound of dirt... Front were dusty, but that was it.
3 fenders were not bad, but the front passenger side was quite bad
It seems that the fender-flare gaskets rub through the paint...
I used a dremel to grind it down to metal and used rust converter, primer and paint.
Since the gasket edge was the problem, I had to grind/paint a bit farther (so it would be a bit visible)
This is how it turned out

Its so far down, you can't really see it.
I'm really glad I caught it when I did.
On the other fenders, I washed, clayed and waxed them to prevent rusting and rubbing.
Since I'm from rust-free Vancouver and have a bit of rust, I would advise everyone with flares to check and fix the problem before it spreads to visible areas.