skeleton.SetSlotsToSetupPose
resets slot colors.
That's because a slot's color is animatable and does have a setup pose that it can reset to.
The way you're doing it is fine.
In fact, your code should really hold that color information itself rather than applying it on the slot and then forgetting it.
But if you want a version of SetSlotsToSetupPose that doesn't reset colors, it would look like this:
foreach (var slot in skeleton.Slots) {
var slotData = slot.Data;
slot.Attachment = slotData.AttachmentName == null ? null : skeleton.GetAttachment(slotData.Index, slotData.AttachmentName);
}