APIs for updating underlying function of a node (2 part question)

[gidhubuser255]

(1) Updating function logic

For reference consider the following DAG

def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D() -> int:
    return 7

dr = driver.Builder().with_modules(__import__('__main__')).build()
dr.execute('A')  # returns 17

If I want to update the logic of B in the DAG with some function alternative_B, the best way I've found to do so is to do the following:

@config.when(use_alternative_b=False)
def B(D: int) -> int:
    return D + 1

@config.when(name='B', use_alternative_b=True)
def alternative_B(C: int, D: int) -> int:
    return C + D

dr = (
    driver.Builder()
    .with_modules(__import__('__main__'))
    .with_config(use_alternative_b=True)
    .build()
)
dr.execute('A')  # returns 25

Which is a bit cumbersome, since it requires modifying source code and adding configuration primitives to enable the ability.

An alternative way to do this is how the fn_graph library handles this (https://fn-graph.readthedocs.io/en/latest/usage.html#building-up-logic), which provides a similar ability to update the implementation of a node by allowing you to make updates on your "driver" (called composer in fn_graph). For example to achieve the same ability in fn_graph you can do this, no decorators or additional config entities needed:

composer = Composer().update(A, B, C, D)  # these are all the same defined functions as above
composer2 = composer.update(B=alternative_B)  # just plain defined function, no decorators
composer3 = composer2.update(B=alternative_B2)

composer2.A()
composer3.A()

Questions:

1. Is there a similar existing mechanism to this in Hamilton?
2. If not could it be added?

(2) Updating function logic in a specific branch

Similar to the above consider when you want to update the implementation for a function, but only when the path from that function to the root executed function passes through another specific node/function.

For example if I want to update D to D__prime but only in the path that includes B, in Hamilton I could achieve that like so:

def A(B: int, C: int) -> int:
    return B + C

def B(D_thru_B: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

@config.when(use_d_prime=False)
def D_thru_B(D: int) -> int:
    return D

@config.when(name='D_thru_B', use_d_prime=True)
def D__prime(D: int) -> int:
    return D * 2

def D() -> int:
    return 7

But this has a couple problems: (a) if I want to later update all references to D with D__prime (not just through B) I'll need to add another D__prime2 with a config.when(name='D', ...) and a new config parameter, which is again pretty cumbersome. And more importantly (b) this method breaks down when the branch specific replacement happens behind a converging point, for example:

def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D(X: int) -> int:
    return X + 1

def X() -> int:
    return 9

If I want to replace X in the path of B with this method I would need to bifurcate all my functions starting at the converging point (D_thru_B, X_thru_B, D_thru_C, X_thru_C, etc), which would be prohibitive for anything more than a very simple example.

Questions:

1. Is there a good existing way to achieve this? I'm thinking there's probably a way using the existing Parallelizable and Collect mechanism which can already dynamically generate new nodes to add to the graph, and which could be used to do the bifurcation I mentioned?
2. Would it be possible to add the ability to do this with something more like the fn_graph style api, something like driver.update(X=X__prime, in_path=['B'])

[jernejfrank]

Hey, cool ideas.

(1) Updating function logic

For reference consider the following DAG

def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D() -> int:
    return 7

dr = driver.Builder().with_modules(__import__('__main__')).build()
dr.execute('A')  # returns 17

Questions:

1. Is there a similar existing mechanism to this in Hamilton?
2. If not could it be added?

At the moment Hamilton does everything via modules (there's things brewing to enable directly passing functions in, but not there yet). There is a way to override B by defining it in a seperate modue and then using .allow_module_overrides() during the building process (checkout this example. In your case:

# new_module.py

@config.when(name='B', use_alternative_b=True)
def B(C: int, D: int) -> int:
    return C + D

and

# main.py

def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D() -> int:
    return 7

if __name__ == "__main__":
    import __main__ as main
    import new_module

    dr = driver.Builder().with_modules(main, new_module).allow_module_overrides().build()
    dr.execute('A')

A word of warning, if you use multiple modules and have in them defined functions with the same name Hamilton will throw an error. However, with the .allow_module_overrides() flag order matters .with_modules(main, new_module) != .with_modules(new_module,main) and the flag is global so in case you use multiple modules you need to be careful not to override something you don't want to.

(2) Updating function logic in a specific branch

Questions:

1. Is there a good existing way to achieve this? I'm thinking there's probably a way using the existing Parallelizable and Collect mechanism which can already dynamically generate new nodes to add to the graph, and which could be used to do the bifurcation I mentioned?
2. Would it be possible to add the ability to do this with something more like the fn_graph style api, something like driver.update(X=X__prime, in_path=['B'])

I am not sure a fully understand the second part, but I can give it a go and let me know if I missed something. Hamilton will create the full graph, but it will only walk / execute nodes that are relevant for a specified output (in your case A). You cannot really control which branch of the graph will execute, only the nodes that are leading to the requested output. In your case, B will always execute because A needs it (same for C) all the way down to X. What you have control over is what the individual nodes are and there's a couple of ways to organise your code to manage multiple potential branches:

1. Define "static" nodes (things that you think will not change) in a seperate module and organise "dynamic" functions into different modules. You can then during the build time choose which module to load that will create the desired nodes. like having a feature catalog.
2. Using subdag decorator to re-use functions example and run custom tags esentially as a node.
3. Similar to subdag for simpler stuff maybe the pipe family might be easier to use example

[skrawcz]

What does "override" refer to here?

Overrides is what we can do to an instantiated driver object. Right now we can provide values that override a node.

What you suggest here would be a form of it:

driver = driver.update(date=new_date, in_path=['shipment_view'])
driver.execute('main_view')

One other way I could see this working which is more consistent with @config.when for node replacement, would be something like @path.passes_through, which could work something like this:

def X() -> int:
return 9

@path.passes_through('B', name='X')
def X_prime() -> int:
return 99

I see. So this feels like you just want to avoid having to update B's parameters for this to work. Since I feel it's not that many more characters to update B's parameter and not have the decorator here.

I think I'd want this instead:

@path.passes_through(B, name=X) # could we do references? or would that be too constraining?
def X_prime() -> int:
     return 99

Edge cases to think about:

1. how would one debug this? What would show up in the "viz" view?
2. Is the main use case to apply updates given a graph? Since I think if it's static code that is checked in and not touched again, I'd just argue that people should make the edits...
3. could a LSP server help stitch together autocomplete / a view of the graph easily? (we haven't put much time into the vscode plugin, but we could...)

I like this:

driver = driver.update(date=new_date, in_path=['shipment_view'])

and/or combining it with the above

# some module
@path.passes_through('B', name='X')
def X_prime() -> int:
     return 99

After all Hamilton is just nodes and edges. So "graph chaining" and this type of "graph overrides/transformation" doesn't seem out of the realm of what Hamilton could support.

[gidhubuser255]

@path.passes_through(B, name=X) # could we do references?

I think using a reference to the function could cause problems because if B is replaced with B_prime under the hood this path decorator would not trigger, when it actually should.

Regarding your edge cases:

(1) I would imagine this would look similar to the viz of config.when in driver.visualize_execution, where there is some suffix on the node name. So something like this:

(2) The way I see it being used is mostly ephemeral replacements for things like my UI stuff I mentioned as well as sensitivity analysis in computational models. Although I have seen static logic replacements like this (not using Hamilton) used in large models where the complexity would have been prohibitive to manually code the unraveled paths (but many of those cases could probably have been considered bad design anyway).

(3) Can't speak to this atm, as I'm unfamiliar with the capabilities and requirements of LSP servers

Personally I do prefer the driver.update method as it makes the ephemeral cases easier to work with, but I realize that is a bit more complicated to implement given the module based architecture. Plus I think I have some dynamic module magic (using types.ModuleType) I can mess with to get it to work closer to driver.update.

[skrawcz]

Personally I do prefer the driver.update method as it makes the ephemeral cases easier to work with, but I realize that is a bit more complicated to implement given the module based architecture. Plus I think I have some dynamic module magic (using types.ModuleType) I can mess with to get it to work closer to driver.update.

Actually once in the driver, we have the nodes & edges data structure, so modules don't play into this anymore -- we already have pointers to the functions.

E.g. think of a driver as an instantiated graph. We can then perform graph transformations on it that result in another static graph.

[gidhubuser255]

so modules don't play into this anymore ... We can then perform graph transformations on it that result in another static graph.

Oh nice, so going back to just part (1) of my question (for non-path-dependent updates), seems like no major architectural changes would be needed to expose some kind of mechanism like driver.update(x=x_prime) (with no in_path argument)?

[gidhubuser255]

@skrawcz assuming my above assumption is correct, is there some process for adding feature requests to a backlog or roadmap, or opening a wider discussion for adoption? Thanks.