How do you reduce bass guitar buzzing?

I started learning bass just a little while back, I know how to play and all, I just have issues every so often. I tuned my E string to B for a specific song I have. Every time I try playing on the first fret, I always get buzzing. I will say that I sometimes miss the fret, but even when I stop to play that one note, it buzzes, no matter where my finger is.
What exactly is the problem? The string, the bass or is it just common?

Tagged as: fret