pyspark show dataframe as table with horizontal scroll in ipython notebook

I'm not sure if anyone's still facing the issue. But it could be resolved by tweaking some website settings using developer tools.

WHen you do

Open developer setting (F12). and then inspect element (ctrl+shift+c) and click on the output. and uncheck whitespace attribute (see snapshot below)

You just need to do this setting once. (unless you refresh the page)

This will show you the exact data natively as is. No need to convert to pandas.

Just add (and execute)

from IPython.core.display import HTML
display(HTML("<style>pre { white-space: pre !important; }</style>"))

And you'll get the df.show() with the scrollbar

this is a workaround

spark_df.limit(5).toPandas().head()

although, I do not know the computational burden of this query. I am thinking limit() is not expensive. corrections welcome.

Just edit the css file and you are good to go.

1. Open the jupyter notebook ../site-packages/notebook/static/style/style.min.css file.

2. Search for white-space: pre-wrap;, and remove it.

3. Save the file and restart jupyter-notebook.

Problem fixed. :)