not really. I have started driving since 18 yrs in Canada, and drove to school everyday at that time.
After coming back to Hong Kong, I didn't drive for around 5 yrs. The main problem is I didn't know the road well, I always worried that if I miss a turn or got to the wrong lane, I will go to another place.
Starting from 2 yrs ago, I slowly pick up because I need to drive my son to school. I live in CWB and my son attends nursery and K1 in Tai Tam. In fact, many people said the route is not easy because it's narrow, full of ''S" sharp coners...anyway, I get used to it already, and not difficult at all.
As long as I know the road, route, which corner should I turn....I can drive to Taipo, Sai Kung, Stanley...now.
don't worry, practice makes things perfect. drive more and gain more experience, you shouldn't have problem.