It's not completely crazy to discuss time-travel in modern theoretical physics. For example, the Kerr solution to Einstein's field equations (representing a rotating black hole with no charge) admits closed timelike curves. Meaning that an observer in this spacetime could meet themselves in their own past!
There's a beautiful theorem of mathematical physics that tells us that any spacetime that is compact necessarily admits closed time like curves. In non-maths speak: if the spacetime describing our universe is bounded* then time-travel is possible, and in some sense inevitable.
* and contains its limit points