Why do we need the whole Hilbert space?

This came up in our mathematical physics class. It is often said that quantum mechanical states are described by vectors in Hilbert space. This, however, is slightly inaccurate, since any scalar multiple of a vector describes the same state. Therefore, strictly speaking, quantum mechanical states are actually described by “rays” in Hilbert space. Despite this, it seems that one still needs the whole Hilbert space to do quantum mechanics — one cannot do it by using “rays” alone. In Geroch’s words, we have to “drag the vectors through the mud” in order to do quantum mechanics, and it is not clear why this is the case.