How to configure Jupyter notebook using pyspark ?

Step 1: First you need to install Jupyter notebook using below this command:

pip install jupyter
pip install "ipython[notebook]"

Step 2: Then you need to setup your environment variable in ~/.bashrc

export SPARK_HOME="/usr/hdp/current/spark-client"
export PYSPARK_SUBMIT_ARGS="--master local[2]"

Step 3: Then type in your terminal below this command

jupyter notebook --generate-config

Step 4: Now you need to open this file using vim or text editor

                   /root/.jupyter/jupyter_notebook_config.py

                   and paste this below lines
                   c.NotebookApp.ip = '*'
                   c.NotebookApp.open_browser = False
                   c.NotebookApp.port = 8889
                   c.NotebookApp.notebook_dir = u'/usr/hdp/current/spark-client/'

Step 5: And then create a file name “00-pyspark-setup.py” in “/root/.ipython/profile_default/” location and paste this given text

import os
import sys

spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "/python")

sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.8.2.1-src.zip'))

pyspark_submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "")
if not "pyspark-shell" in pyspark_submit_args: pyspark_submit_args += " pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = pyspark_submit_args

execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))

Step 6: Now type in your terminal “jupyter notebook” command

And enjoy jupyter notebook using pyspark

4 thoughts on “How to configure Jupyter notebook using pyspark ?”

    1. First of all you need to install python-pip using “sudo apt-get install python-pip” then try to install jupyter notebook using pip command using your terminal

  1. Hi! I could have sworn I’ve been to your blog before but after going through a few of the posts I realized it’s new to me.
    Anyways, I’m certainly pleased I found it
    and I’ll be bookmarking it and checking back often!

  2. hello there andd thank you for your information –
    I have definitely pcked up anything new from
    right here. I did however expertise some technical issues uing thi site,
    since I experienced to reload the site lots off times previous
    to I could get it to load properly. I had been wondering if your
    web osting is OK? Not that I am complaining, but sluggish loading instances
    times will very frequently affect your placement in google and coulkd damage your high quality score if advertising
    and marketing with Adwords. Anytway I’m adding this RSS to
    my email and can look out ffor a lot more of your resppective exciting content.
    Ensure that you update this again very soon.
    5 langkah prosedur penerimaan siswa baru

Leave a Reply