Combine milk with rosemary and bring to simmer over low heat. Let simmer 5 minutes, then remove from heat. Reserve.
Rinse cubed potatoes, then place in large pot. Cover with cold water and place over medium-high heat. Bring water to boil, then reduce heat to simmer. Cook until potatoes are fork tender, approximately 20 minutes. Drain and rinse. Let rest in colander for 2–3 minutes to dry.
Remove rosemary from milk.
Place food mill over pot and, working in batches, run potatoes through mill. Alternatively, place potatoes back in pot and use potato masher to mash potatoes until all lumps have been removed.
Use wooden spoon or rubber spatula to stir butter into potatoes until combined.
Pour ¾ of warm milk mixture into pot and stir to combine. Add more milk if necessary to reach desired consistency. Season with salt and pepper and serve.