What’s your alternative to jupyter notebooks in ML R&D? (here’s mine:)

[limpid raven]

I do a lot of data transformations or simple model fine tuning in Jupyter notebooks. After listening to a few MLOps podcasts, it seems like that’s generally advised against.

I guess because you want reproducibility and more modular, more manageable code.

The good thing about notebooks for me is the UI where you can inspect Python variables and debug functions quickly.

So, my approach has been to develop in notebooks, write functions for everything and once I stop touching certain functions I separate them into a “lib” type of repo and continue importing them from there.

What’s your take on this? What are the pros and cons of notebooks for you and how do you mitigate the cons?

[dull oyster]

Since I manage environment with poetry, I typically have a /notebooks directory that I put in .gitignore, so that whatever I develop in the notebook is using the same environment with /src or other directories, where I actaully write the packaged solutions, but notebooks still wouldn't be comitted. That also means as soon as finetuning code works, I also put them in the packaged solutions and run pipeline with cloud platforms