Spark 1.6.2, Java code (sorry), this will change every instance of Tesla to S for the entire dataframe without passing through an RDD: dataframe.withColumn(“make”, when(col(“make”).equalTo(“Tesla”), “S”) .otherwise(col(“make”) ); Edited to add @marshall245 “otherwise” to ensure non-Tesla columns aren’t converted to NULL.
Instead of playing whac-a-mole by listing specific tasks to disable (publish, publish-local, publish-signed, etc), another option is to turn off artifact publishing at the source. publishArtifact := false Even though there’s no publishing happening, I also found I needed to supply a publishTo value to make sbt-pgp’s publish-signed task happy. It needs this value, even … Read more
If you have x as Option[Long], x.get will give you Long.
What you observe is expected behaviour – yet of course it’s very bad. Good that known solutions and best practices exist to guard against it. In this answer I’d like to spend some time to explain the issue short, long, and then in depth – enjoy the read! Short answer: “don’t block the routing infrastructure!“, … Read more
I will try to answer the Scala Future to JS Promise part of the question. As you haven’t provided an example. I will provide one here with the conversion. If we say that we have a Future implemented in Scala this way: val f: Future = Future { session.getX() } f onComplete { case Success(data) … Read more
First there is a plethora of HTML scraping libs in JVM all you need to do is pimp one of them (pimp my library pattern). The four I have used are: HtmlUnit – Will emulate the browser and even run Javascript Jericho – Preserves formatting and ideal if you want to edit the scraped HTML … Read more
The problem with the Application trait is actually described in its documentation: (1) Threaded code that references the object will block until static initialization is complete. However, because the entire execution of an object extending Application takes place during static initialization, concurrent code will always deadlock if it must synchronize with the enclosing object. This … Read more
+: works with any kind of collection, while :: is specific implementation for List. If you look at the source for +: closely, you will notice that it actually calls :: when the expected return type is List. That is because :: is implemented more efficiently for the List case: it simply connects the new … Read more
Extending the answer By @Rohan Aletty. Here is a sample code of a BroadcastWrapper that refresh broadcast variable based on some ttl public class BroadcastWrapper { private Broadcast<ReferenceData> broadcastVar; private Date lastUpdatedAt = Calendar.getInstance().getTime(); private static BroadcastWrapper obj = new BroadcastWrapper(); private BroadcastWrapper(){} public static BroadcastWrapper getInstance() { return obj; } public JavaSparkContext getSparkContext(SparkContext sc) … Read more
But it seems Scala Arrays & Lists doesn’t provide any methods for adding items dynamically due to the immutable nature. Well, no. Scala Arrays are just Java arrays, so they are mutable: val arr = Array(1,2) arr(0) = 3 // arr == Array(3, 2) But just as Java (and C/C++/C#/etc.) arrays, you can’t change the … Read more