Skip to content

[WIP] Checkpointing, part 1 - #1167

Draft
dwhswenson wants to merge 17 commits into
openpathsampling:dev-2.0from
dwhswenson:checkpointing-v2
Draft

dwhswenson wants to merge 17 commits into
openpathsampling:dev-2.0from
dwhswenson:checkpointing-v2

Conversation

@dwhswenson

Copy link
Copy Markdown
Member

I've made a couple attempts at checkpointing for OPS. It seems that it requires an API break, because OPS will need to pass a checkpoint variable into movers and then down into engines. This would break any external movers that don't support that variable, so this PR is pointed at the dev-2.0 branch.

This is a top-down implementation: it includes work for checkpointing movers, but not within engines. Of course, the whole point is to also checkpoint within engines, but that will come in a second PR that builds on this one. I don't plan to merge this PR until the second also exists as proof of principle.

General usage:

  1. Movers that don't support checkpointing just don't support checkpointing. Not supporting checkpointing doesn't break anything. It is just an inconvenience. This allows us to skip checkpointing support on some mover types for now.
  2. Movers that do support checkpointing do so by receiving a checkpoint object. They can get the checkpointed data using checkpoint.load_checkpoint(), which returns a tuple of data and files. They can extract data from the data dict and moves files to the right location based on the files dict.
  3. Functions that add checkpointing information and call another function that uses checkpointing information should call the other function within the with checkpoint.next_context() context manager.

As of posting, this still needs more documentation on usage, but the above is a quick overview.

  • Basic checkpointing of movers
  • Full test coverage of changes
  • Documentation of how to use checkpointing when writing path movers
  • Proof of concept that this approach extends to engines (to implement in another PR, but don't merge this one until that is the done.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant