This behaviour is by design. The Pager deducts the space needed to show the PagingSummary and renders the buttons centered with respect to the space remaining. Here is an image showing how the items are rendered in the pager:
A solution would be to have the PagingSummary absolute positioned. Have in mind that in this case you might end up with overlapping items, especially if the pager is justified, instead of centered, or if there is not enough screen space: