Normally I only remove the shell but not the "hairy skin". I will rub off the hair as much as possible.
If u do want to remove the "hairy skin", put the chestnut around the edge of a dish and put it in microwave for 2 min. Cool it down and rub the skin off.