I had the same problem. Once I put my headers on, it went away. It not only improved he sound, but the rattling inside the car disappeared, too. It was shaking so much, that I would think my phone was going off in my pocket, but it was just the exhaust making it feel like it was.
Very annoying.