poll-loop: Track the poll timeout as an absolute, not a relative time.