iphone volume doesn't go low enough?

I turned my soundcheck off but my volume only goes so low before it just goes to off. Is there a way to fix this? With soundcheck on, it's even worse as the lowest part of the volume range doesn't work (probably by design). I have really good ear buds that block out a lot of ambient noise and I don't need them that loud. Any ideas?