Using Scala IDE with Sparklet(Zeppelin on Windows)

Tweet about this on TwitterShare on LinkedInShare on Google+Share on Facebook

Zeppelin comes with built in Apache Spark. Below are the steps to use Scala IDE to run and debug code for Apache Spark.

Below are the links to download to Spark and Zeppeline on Windows env
Sparklet (Apache Spark & Zeppelin Installer for Windows 64 bit)
Scala IDE

  • Install and make sure Zeppelin and Spark running properly
  • Run Scala IDE and select a Workspace

    Workspace Launcher

  • Go to File -> New -> Scala Project and enter a project name and click Finish.

    Create a Scala Project Window

  • Right click on the Project folder and go to Build Path -> Add External Archive

    add external archive

  • In Jar Selection window go to the Sparklet installed folder. Then go to interpreter -> spark -> dep folder and select zeppelin-spark-dependencies-0.5.6-incubating.jar file. Click on Open to add it to the project.

    jar selection window

  • Right click on the src folder and select New -> Scala object

    new scala object

  • Give a name and click Finish.

    create new file wizard

  • Type the below code. The program counts the occurrence of each word and prints the top 15 words from readme.md file.
    import org.apache.spark.SparkContext
    import org.apache.spark.SparkConf
    object WordCount {
       def main(args: Array[String]) {
         
         val conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]")
         val sc = new SparkContext(conf)     
         val textfile=sc.textFile("c:/Sparklet/readme.md")
         val counts = textfile.flatMap(line => line.split(" "))
                     .map(word => (word, 1))
                     .reduceByKey(_ + _)
         val top15=counts.sortBy(_._2, false)take(15)
         top15.foreach(println)
       }
      
    }
    
  • Before running the code you need to set Scala Installation project to Scala 2.10. To do so right click on project folder then select Scala -> Set the Scala Installation

    set scala installation

     

  • In Scala Installation Choice window select Fixed Scala Installation: 2.10.6(bundled)

    scala installation choice

  • To run the program you can click on the run button or right click on the code window then select Run As-> 1 Scala Application

    run as scala application

  • If the code runs successfully you will see the output as below.

    scala output window

Leave a Reply

Your email address will not be published. Required fields are marked *


6 × four =