I find that I have a hard time holding a romantic/sexual interest in more than one person at a time.

A harem in the I am a wealthy mofo with a lot of willing companions kinda way is attractive but really just for sexual use and entertainment purposes. I think it is possible to have some kind of polyamorous or polygamous relationship but I think that there are always going to be favorites. Just look at the Taj Mahal, he didn't build that for every wife just his favorite. Can I care about more than one woman at a time? Sure. Can I have fun with more than one sub at a time? Hell yeah, but one of them is always going to be my first choice and the other is going to be a friend with benefits or just for entertainment. I think having a large harem pretty much eliminates much emotional connection to any of the submissives. If I had a dozen or so women competing for my attention at any given time I would definitely treat the vast majority of them badly because they would be essentially disposable.