Discuss use cases about skipping cells during execution

[willingc]

From #425 by @akx:

Passing --skip-tag (can be passed multiple times) will have the executor entirely skip cells endowed with that tag.

That status is recorded in the result notebook (metadata.papermill.status = "skipped"), as well as the tags that contributed to this skipping (metadata.papermill.skip_tags).

cc @JuhaKiili

I'm opening this issue to gather feedback on real world use cases which may benefit from the ability to skip cells during execution. Let's focus on documenting the use cases before moving toward a technical implementation.

[JuhaKiili]

We are developing a machine learning platform @ valohai.com

One of our features is a Jupyter notebook plug-in, which will allow our customers to run the notebook from start to finish on a cloud instance asynchronously. We use Papermill for the server-side execution.

One of the feature requests from customers is an ability to tag certain cells as "local only", which means they will not be executed on the server. Here is a direct quote from our customer:

Sometimes we have some cell in the notebook that are just used for a quick eda and don't need to be run on a gpu machine. Would it be possible to add some form of tag so that we can choose which cells of the notebook to execute on vh?

We could programmatically remove the tagged cells before we send the notebook to the server, BUT this would ruin the version control side of things. When our customers come back 6 months later and download the resulting notebook, they still want the disabled cells to be there, even if they were not executed.

[willingc]

Thanks @JuhaKiili.

A few thoughts on notebook functionality that appears to address your use case today:

If you look in the keyboard shortcuts for the notebook, you can disable a cell from running by changing it to a raw cell (ESC+r). You can later reactivate it as a code cell with (ESC+y).

Changing a cell to a raw cell provides the same functionality as skipping a cell during execution.

cc/ @MSeal

[JuhaKiili]

Changing a cell to a raw cell provides the same functionality as skipping a cell during execution.

Didn't know about raw cells. Thanks for the info!

[MSeal]

The raw cells would work. Though I can see that being a bit annoying to manage.

I also noticed the cell that you likely want skipped in that gif is a series of magics which would make conditional execution of the logic within the cell irritating to manage by the user.

While we're exploring options. A way to manage how a notebook executes more precisely can be to use a registered engine: https://papermill.readthedocs.io/en/latest/extending-entry-points.html#developing-a-new-engine.
This would let you modify cells or choose how to (or not) execute them within the papermill context. You could change cells to raw there or skip and add some output message indicating the cell wasn't executed perhaps.

[akx]

@MSeal Engines can't register new parameters during the CLI parsing phase though, so having a custom engine for the #425 use case would also require some other method of passing in the regex/pattern/list-of-names, which also sounds clunky. 😕

[MSeal]

We discussed the topic again today in our weekly meetup for nteract and are still of the opinion that it doesn't fit the project to support as a built-in mechanism, especially with some known work-around patterns and the ability to extend papermill for more custom execution patterns. Adding this feature would increase the complexity of describing how papermill operates and why the outcome ran the way it did. It would also break the construct we rely on heavily that rerunning a papermill output in any notebook interface will produce the same result as when it ran in papermill.

Of the solutions available, converting cells to raw that you don't want to have executed if probably the best option. Though if people wish to go the engines route we can help improve the engines abstraction where there are gaps to using the plugin system. e.g. @akx If we need to make a way for plugins to be able to accept plugin kwargs from CLI calls we could open a separate issue on that topic. I think that'd benefit customization in general.

Other maintainers please feel free to add comments or suggestions to this thread as it goes on, since I'm echoing a wider conversation that we've had over the past few months.

[casperdcl]

Adding this feature would increase the complexity of describing how papermill operates

could you specify an explicit word/character limit for describing new features? We can then see if it's possible to concisely describe.

why the outcome ran the way it did [...] we rely on heavily that rerunning a papermill output in any notebook interface will produce the same result as when it ran in papermill.

#403 fixes this by removing cells altogether hence producing notebooks that are runnable in any interface with the same results

Of the solutions available, converting cells to raw that you don't want to have executed if probably the best option.

not sure

we can help improve the engines abstraction where there are gaps to using the plugin system.

sounds complicated but still welcome

[MSeal]

Sorry for not being highly responsive on this thread, it's been a busy couple weeks. I'm going to try to post the nteract groups response here, though it's flavored a bit in my fashion of wording. I asked the group to also comment on this thread so it's not just myself speaking for the group as a whole.

could you specify an explicit word/character limit for describing new features? We can then see if it's possible to concisely describe.

The complexity of describing is not in the number of words, it's in the fact that we no longer have a linear execution model. There's a lot of users of papermill that this would confuse. Many users it wouldn't confuse at all but in the past when we've had to describe non-linear behavior in other jupyter tooling it put a burden on the maintainers and community to constantly explain why execution happened the way it did. We'd like to not have base papermill have the same explanation and support burden.

#403 fixes this by removing cells altogether hence producing notebooks that are runnable in any interface with the same results

That's fair, though then we have to explain to users that they need to flip cell types when they want to turn them on / off. The current UIs don't lend themselves well to this effort, as it won't tell you how to execute a raw cell, it'll just do nothing when you execute it. For a specialized usecase this is probably fine, but there's many users that don't understand the cell type concept so the change has questionable value to them. If there were a more first-class "disabled" concept that guided the users in the interface I think my objection here would be mute for a general purpose tool.

sounds complicated but still welcome

Agreed, just trying to make extensibility for other solutions to problems such as these not be explicitly blocked by the tooling in place.

[willingc]

Adding an additional point that I find important for the linear execution model: reproducibility of research notebooks. A non-linear execution model with more variability makes it more difficult to account for which cells were run or should be run to reproduce.

I agree @MSeal that a future jupyter/nbformat spec would be a better place for a first-class "disabled" concept.

[rgbkrk]

The issue here is that it makes the job of maintainability really difficult for us in terms of support costs. Not the literal code, but the expectation that we have to respond to issues related to this not just in papermill but also in the classic notebook UI, jupyterlab, etc. This creates notebooks that run differently in different systems, which goes against our core tenant of reproducibility.