Would you knowingly put on a wet pair of undershorts? When I used a badger brush in the past I always made sure it was a couple of days drying before using it again as I live in a very hot & humid climate where mold is rampant. I am now a fanatical synthetic brush convert. YMMV.
I have a few brushes and normally rotate them. I also keep wet brushes out of the bathroom and in a brush stand, I'm not sure this is necessary but they dry quicker. If I had just the one brush, I would rinse and towel dry it and leave it to dry out of the damp environment of the bathroom.