spark-euler

Spark Demo 1 - A simple Euler sum computation¶

This is a very trivial demo of spark. If you download and install spark from spark-2.0.2-bin-hadoop2.7 then cd to that directory and do the following

$ export PYSPARK_DRIVER_PYTHON=Jupyter
$ export PYSPARK_DRIVER_PYTHON_OPTS=notebook
$bin/pyspark 

Jupyter will come up in that directory with the spark context "sc" already loaded. The usual way to get tthe spark context is

sc = pyspark.SparkContext('local[*]')

which tell spark to use all available cores on the local machine. For other clusters you may need something else.