This all depends on how much luggage you have. Little or no luggage best way is via bus.
I too would do it via Weerawila which is a more efficient route.
Tissa-> Weerawila Junction-> Moneragala (in a Matara-Moneragala Bus) ->Catch a Colombo-Pottuvil bus from Moneragala (or there are also some Moneragala-Potuvil or Moneragala-Panama buses)
Arugambay is just in the outskirts of Pottuwil town. (About 5km from the town centre) If you catch a Pottuvil bound bus, get off and take a Tuk Tuk from there. Alternatively, if you get a bus to Panama (not regular as I know) you can get off in Arugam Bay.