Because something else is already there waiting to gobble them up: dust mites. And the longer you wait between washes, the more food these critters will have and the more they'll procreate and ...
These micro-organisms thrive in the warm, cosy conditions of your bedding, where they feed on sweat, saliva, dead skin cells, ...