FIX: refactor Network and DAG SOLVER to fix bad pruning #26
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
were writing in text-mode in PY3. and failing as encoding error.
receiving partial inputs, needed for other operations.
+ The x2 TCs added just before are now passing.
NOTE dict are not deterministic in <PY3.6. So this commit would not improve determinism in those pythons. + build: add `boltons` dependency for ndexedSet. + doc: mark all set usage if affect determinism. + e.g. see reproducibility problem in yahoo#14.
the later described in yahoo#21.
not after compose(). + All TCs pass ok. + NOTE this is not yet what is described in yahoo#21.
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
7e851b1 to
3ee344b
Compare
to pass +TC checking DeleteInst vary when inputs change. - x4 TCs still failing, and need revamp of dag-solution.
+ Read the next doc-only commit to understand changes.
+ Renamed:
+ net.steps --> net.execution_plan.
+ (old)compile() --> _build_execution_plan()
+ _find_necessary_steps() --> (new)compile() + _solve_dag()
compile() became the master function invoking _solve_dag &
_build-execution_plan(), and do the caching.
+ refact(compute()): extract common tasks from sequential/parallel.
+ refact show_layers() to allow full-print, geting also string
(not just printing), and using custom classes for representation.
+ Raise AssertionError when invalid class in plan.
it's a logic error, not a language type-error.
Probaly unreported bug in v1.2.4 for '_neccessary_steps_cache`.
ankostis
force-pushed
the
refact-new_dag_solver
branch
4 times, most recently
from
e67e57c to
31aae2f
Compare
+ Pruning behaves correctly also when outputs given; this happens by breaking incoming provide-links to any given intermedediate inputs. + Unsatisfied detection now includes those without outputs due to broken links (above). + Remove some uneeded "glue" from unsatisfied-detection code, leftover from previous compile() refactoring. + Renamed satisfiable --> satisfied. + Improved unknown output requested raise-message. + x3 TCs PASS, x1 in yahoo#24 and the first x2 in yahoo#25. - 1x TCs in yahoo#25 still FAIL, and need "Pinning" of given-inputs (the operation MUST and MUST NOT run in these cases).
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
31aae2f to
0830b7c
Compare
ankostis
added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 4, 2019
…must run + Insert "PinInstructions" in the execution-plan to avoid overwritting. + Add `_overwrite_collector` in `compose()` to collect re-calculated values. + FIX the last TC in yahoo#25.
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
008d501 to
b870451
Compare
…must run - WIP: PARALLEL execution not adding PINS! + Insert "PinInstructions" in the execution-plan to avoid overwritting. + Add `_overwrite_collector` in `compose()` to collect re-calculated values. + FIX the last TC in yahoo#25.
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
b870451 to
0dc1293
Compare
Codecov Report
@@ Coverage Diff @@
## master #26 +/- ##
==========================================
+ Coverage 77.87% 78.31% +0.44%
==========================================
Files 5 5
Lines 348 392 +44
==========================================
+ Hits 271 307 +36
- Misses 77 85 +8
Continue to review full report at Codecov.
|
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
7c94891 to
91c7b11
Compare
- STILL buggy PIN on PARALLEL, 2 DISABLED TCs FAIL: - test_pruning_with_given_intermediate_and_asked_out() - test_unsatisfied_operations_same_out() + move check if value in asked outputs before cache-evicting it in build-execution-plan time - compute methods don't need outputs anymore. + test: speed up parallel/multihtread TCs by reducing delays & repetitions. + refact: network rightfully adopted stray functions for parallel processing - they all worke on the net.graph, + upd: networkx api by indexing on `dag.nodes` views. + enh: add log message when deleting in parallel (in par with sequential code). + refact: var-renames, if-then-else simplifications, pythonisms. + doc: A lot!
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
91c7b11 to
06f6554
Compare
ankostis
force-pushed
the
refact-new_dag_solver
branch
from
4603bbb to
1cc733e
Compare
ankostis
added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 4, 2019
- WIP: x4 TCs FAIL and still not discovered th bug :-( + BUT ALL+AUGMENTED PARALLEL TCs pass (yahoo#26 were failing some) + refact: net stores also `pruned_dag` (not only `steps`). + refact: _solve_dag() --> _prune_dag(). + doc: +a lot. + TODO: store pruned_dag in own ExePlan class.
Author
|
Can merge now:
|
This was referenced Oct 5, 2019
ankostis
referenced
this pull request
in syamajala/graphkit
Oct 8, 2019
ankostis
added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 8, 2019
many commits ago. Never got it bc TC were not checking merges!
ankostis
added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 8, 2019
many commits ago. Never got it bc TC were not checking merges!
Author
|
3 tasks
Author
- WIP: x4 TCs FAIL and still not discovered th bug :-( + BUT ALL+AUGMENTED PARALLEL TCs pass (yahoo#26 were failing some) + refact: net stores also `pruned_dag` (not only `steps`). + refact: _solve_dag() --> _prune_dag(). + doc: +a lot. + TODO: store pruned_dag in own ExePlan class.
... bugged in the opening commit d403783 of this PR, and discovered 68(!) commits later, and all that time had to live with x4 broken TCs with asked-outputs.
+ Partial fix deterministic results (yahoo#22-2.4.3i).
bc subgraph was taken on plain string outputs. + minor upd err-msg terminology.
due to bad node check, evicting parallels it nevered kicked in.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors the DAG-solver to prune correctly when intermediate data are given as input.
test_pruning_with_given_intermediate_and_asked_out()test_unsatisfied_operations_same_out()ENH: NEW DAG SOLVER
to any given intermedediate inputs and retrofitting satisfaction detector
to include those operations without outputs due to these broken links.
REFACT: Network class COMPILE+COMPUTE:
_build-execution_plan(), and do the caching.
(not just printing).
net.graphattribute; pave the way for a separate class.networkxAPI when indexingdag.nodesviews.it's a logic error, not a language type-error.
I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.