My workflow for a long time involved me switching back/forth between vscode and browser/databricks ui. I like to write my "production code" in normal python, but notebooks are great for exploration, spikes, visualization, triage etc.
I could write a small dissertation but for various reasons I don't really like jupyter, and databricks notebooks have their own problems with commented magic commands etc.
This led me to check out marimo, and wow, these are so cool. Code that runs in normal python, merges cleanly, has visualizations, widgets, the the app runs locally and doesn't glitch out, and even the vscode extension works nicely. The problem was, the databricks support wasn't great. It just felt a bit dated. It required a warehouse for sql, doesn't seem to really support serverless, and there were just so many oppurtunities to plug databricks into Marimo.
This led me to create marimo-databricks-connect pypi
I tried to plug in "all the things" databricks into the place where they go in Marimo. I'm pretty happy with the result.
- Connect to databricks using databricks-connect & spark (not sql warehouse)
- Authenticate/configure spark using the default databricks-connect process (env vars, .databrickscfg etc), no additional auth config.
- Execution of both python & sql cells
- Autocomplete Catalog/Schema/Table/Column Names
- Browsing of catalogs/schemas/tables/columns in the marimo data sources view
- Browsing of external locations, volumes, dbfs, workspace in the marimo storage browser Notebook widgets to monitor and control of specific instances of databricks capabilities (clusters, workflows, vector search, apps etc)
- Widgets to browse & explore databricks capabilities (compute, workflows, unity catalog)
- Works in local marimo marimo edit notebook.py, in the vscode extension
- Deploy as a databricks app to provide an alternative web based marimo UI.
I'm working on adding serving endpoints as AI providers to the notebooks too.
In particular what I like to use this for is creating "command center" notebooks for given processes that can include some normal pyspark/sql code to query/triage, widgets to monitor/control various databricks resources, visualizations to monitor dq etc.
I just wanted to share and see what the community thinks, would you use it? contributions are welcome.
throwaway account because i'm doxing myself via gh repo.
