Beat the egg yolks and sugar together until thick. Pour milk into saucepan and put on medium-low heat. Stir slowly and gently scald the milk (bring up just below a boil). Take the milk off the heat and temper it with the egg and sugar mixture (add the mixture slowly to the hot liquid, stirring constantly).
Slowly stir in the brandy, rum, and cream.
Refrigerate until thoroughly chilled and pour into a punch bowl.