Friend has just got back from Arugam Bay, this is what he said:
" I travelled to Arugam on 28th Feb which is a long journey, in my mind well worth the trip, the town is a little pricey for my taste, locals was great. The main beach is secluded, but the waves are very strong even after the off-season according to locals."
To travel back to Colombo you can catch one of the three overnight buses that leave Potuvil and takes around 7 hours.
You could try Bus Booking