Though you can get a lot of that context from looking at the git history and discussions (and revisions) on past PRs and perhaps access to the history of team chat channels.
You are right that there's a difference. But you can tell the agent to ingest the whole thing once, and distill an approach in a much smaller (and human readable and reviewable) description perhaps? And then that can serve as the basis for the agent going forward.
So you don't have to retrieve the whole history all the time.
It's similar to telling your new hire to catch up on _all_ the slack history. Only that the agent will actually do so.