You are making the assumption that there is a past to which you could travel.

There is no evidence that time is linear, or that there is even a past to travel back towards.