Nope, it's not just you. I had a rattle last year that was driving me nuts, one of the inserts was loose and moving around. I caught it before it came out. Easy enough to fix, just ground the carbon off, drilled some holes, epoxied and added some carbon.
Every hull will have it's issues...