No, it is not out of line to be uneasy with your dominant having multiple partners. Talk with your dominant. Let him know you aren't comfortable. If he insists on forcing you into a poly relationship, you can always walk away from the relationship.
The number of partners in a relationship is something that should be negotiated at the start. And if he takes on more submissives after agreeing there will be just the two of you, consider it as him breaking trust - because that's exactly what it is.