Benchmarking #1276
Open
Benchmarking #1276
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Experiment is no longer a dataclass
Directly uses the builder and an argument-parser friendly parametrization
MischaPanch
force-pushed
the
benchmarking
branch
from
b391d88 to
56ef25e
Compare
…lti-experiment option
Apart from general improvements, added the possibility to persist as json, to be used in benchmarking and visualization
MischaPanch
force-pushed
the
benchmarking
branch
from
f28c819 to
91029ad
Compare
Conflicts: poetry.lock pyproject.toml tianshou/highlevel/experiment.py
opcode81
force-pushed
the
benchmarking
branch
from
c791897 to
7b4588d
Compare
This form is expected by the visualization script
We will include a separate example in a readme, no need for a script
MischaPanch
force-pushed
the
benchmarking
branch
from
9f4b8c9 to
8cf6576
Compare
The bottleneck for off-policy algos is not rollout but training, so it makes sense to parallelize experiments by default
MischaPanch
force-pushed
the
benchmarking
branch
from
fedf01f to
c3cf417
Compare
MischaPanch
force-pushed
the
benchmarking
branch
from
c3cf417 to
fcd48fc
Compare
In particular, changed save_interval default to None so we don't save the policy after each epoch
In particular, changed save_interval default to None so we don't save the policy after each epoch
MischaPanch
force-pushed
the
benchmarking
branch
from
9d3853b to
51493eb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactoring of the scripts, reducing parametrization of HL scripts to the minimum, restored same default config as in v0.5.0 (except for mujoco task versions, which have been bumped from 3 to 4).
Various improvements in the rliable eval code are done.
Also, this PR adds the possibility to run and evaluate multiple experiments directly from an ExperimentBuilder. This possibility is used to establish a benchmarking script that will run multiple scripts in parallel in tmux sessions, evaluate them with rliable and aggregate the stats such that they can be displayed in the benchmarking section in the logs.
This concludes most of the preparations for establishing easily reproducible benchmarking runs.