My personal preference is for Badger, I have tried (allegedly) the best boars but not horse. I find none of them can hold the lather as well as a good badger brush and none of them feel as good on the face but my statement about the brush getting better is equally applicable to both badger; yes you are correct in that bar or horse will improve more dramatically.
Boar and horse will take longer to break and to be honest the only tangible benefit when fully broken I found with them was they tend to stay slightly stiffer so cheap boar or horse compared to cheap badger I would take the boar or horse. For a more expensive brush it has to be badger for me.