Using Scala IDE with Sparklet(Zeppelin on Windows)

Zeppelin comes with built in Apache Spark. Below are the steps to use Scala IDE to run and debug code for Apache Spark.

Below are the links to download to Spark and Zeppeline on Windows env
Sparklet (Apache Spark & Zeppelin Installer for Windows 64 bit)
Scala IDE

  • Install and make sure Zeppelin and Spark running properly
  • Run Scala IDE and select a WorkspaceWorkspace Launcher
  • Go to File -> New -> Scala Project and enter a project name and click Finish.Create a Scala Project Window
  • Right click on the Project folder and go to Build Path -> Add External Archiveadd external archive
  • In Jar Selection window go to the Sparklet installed folder. Then go to interpreter -> spark -> dep folder and select zeppelin-spark-dependencies-0.5.6-incubating.jar file. Click on Open to add it to the project.jar selection window
  • Right click on the src folder and select New -> Scala objectnew scala object
  • Give a name and click Finish.create new file wizard
  • Type the below code. The program counts the occurrence of each word and prints the top 15 words from readme.md file.[code language=”scala”]
    import org.apache.spark.SparkContext
    import org.apache.spark.SparkConf
    object WordCount {
    def main(args: Array[String]) {

    val conf = new SparkConf().setAppName(“Simple Application”).setMaster(“local[*]”)
    val sc = new SparkContext(conf)
    val textfile=sc.textFile(“c:/Sparklet/readme.md”)
    val counts = textfile.flatMap(line => line.split(” “))
    .map(word => (word, 1))
    .reduceByKey(_ + _)
    val top15=counts.sortBy(_._2, false)take(15)
    top15.foreach(println)
    }

    }
    [/code]

  • Before running the code you need to set Scala Installation project to Scala 2.10. To do so right click on project folder then select Scala -> Set the Scala Installationset scala installation

     

  • In Scala Installation Choice window select Fixed Scala Installation: 2.10.6(bundled)scala installation choice
  • To run the program you can click on the run button or right click on the code window then select Run As-> 1 Scala Applicationrun as scala application
  • If the code runs successfully you will see the output as below.scala output window

Leave a Reply

Close Menu