Were there any areas on the runway allowing for you to move over while you were backtracking? If so I think that would be the correct thing to do, however I know that a lot of runways are simply a straight strip!
From my understanding the aircraft landing always has right of way (unless you'd just landed and were backtracking to clear the runway in which case they cannot force you off) but this chap sounds like he was wasn't yet on final. If he called a short final in the knowledge you were backtracking with nowhere to sensibly vacate, it does seem a bit silly.
I found this article which suggests that even if the in-air aircraft has called a short final, they still have right of way
http://flighttraining.aopa.org/magazine/2008/November/200811_Features_Who_Has_the_Right_of_Way.html
However despite the rules. I think that we should be applying common sense to situations as well! If I was the chap downwind I'd have probably decreased the speed/weaved/taken a slightly wider circuit to allow for you to take off.