Experiments in moving from a Notebook to a service
We explored how to take a concept in a Notebook to a service that can be more widely shared. These experiments focused on seamlessly moving from a Notebook to a deployed Bokeh app. However, in the end we decided that the dream of moving seamlessly from prototype to service was both an impossibility and undesirable.
Notebooks are a excellent way of exploring and developing ideas and by adding widgets these ideas can become interactive demos. However, when it comes to sharing these demos, and exposing them as a service that you want to offer more widely, there doesn’t seem to be a clear path. We wanted to explore how an idea developed in a Notebook might become a service that could be offered more widely.
Our first experience with this was with a project called Forest, the initial ideas and components were developed in the Notebook environment but we found that when we wished to share it as a service, we were a little stuck. The solution we turned to was to rewrite the Notebooks to Bokeh apps which we served using Bokeh Server. This had an advantage in that much of the code could stay the same, but we could share the app without people needing their own notebook environment, or needing access to ours.
Inspired by this, we explored a few experimental approaches in order to see whether this process of moving from notebook to deployed Bokeh Server app could be automated and therefore become a route from idea to service.
JuBo (a portmanteau of Jupyter and Bokeh) was an attempt to create a Jupyter Lab plugin that could convert Notebooks to Bokeh apps and an accompanying service that would deploy and serve them. Here is a very simple demonstration.
Jubo Widget Wrap was a slightly different approach that tried to tackle the interactive elements of notebooks by providing a wrapper around Bokeh widgets to make them look like IPython Widgets. Jubo Widget Wrap also hooks in to the IPython display system to try emulate some of display behaviour in Notebooks.
These experiments were interesting, however led to some conclusions that essentially suggested they may be dead ends.
Firstly, bokeh server and Jupyter notebooks are quite different paradigms, which makes converting between the two challenging. Display was one big area of discrepancy, just one example of which is how in Jupyter Notebooks standard out and error form part of what is displayed, however in Bokeh server this information is just part of the logs. Another big difference of approach is illustrated by the way in which, within each instance of a Jupyter Notebooks running in its own kernel (Python instance), the kernel stores the state of the session. By contrast, in Bokeh Server multiple sessions run under the same Python instance, Bokeh has the concept of a document (one per session) in which state is stored. These are just two of the issues encountered, however they illustrate how any conversion between Notebooks and Bokeh is likely to be fragile and hamstring the best qualities of each approach.
A wider conclusion however, was that a smooth path way from prototype to semi-production ready services is likely to be both impossible and undesirable. Both exploratory and prototyping work almost always produce code that isn’t fit for supporting a service that is expected to be in anyway robust or reliable. The free and ill-defined nature of prototype work is an anathema to production code. We think it’s important that people can quickly and easily share their prototypes, but at the point that it’s being considered a service it may be time to take away what’s been learnt and start again. We think that tools such as Binder could possibly fulfill the need to share, however making it too easy may actually create an environment where poor quality services flourish at the expense of quality ones.