For me April is the best time to ride this area. Just after the mad tourist season and just before the rains come - plus you usually get a good bit of Indian Summer. And not too much wind.
Beautiful ocean views on the way down to the Cape of Good Hope. Don leave any items unattended as the area is inhabited by curious baboon troops!
The scenery is tops, the asphalt is super and the roads are winding and hilly.
The weather was brilliant when I was there end November 2004 (blue skies and ~ 25 degrees...!).
Recommendation: Rent a large trailie so that you can go down one of the many gravel or dirt roads to the beaches if you like.