Filter spark DataFrame on string contains

by

You can use contains (this works with an arbitrary sequence):

df.filter($"foo".contains("bar"))

like (SQL like with SQL simple regular expression whith _ matching an arbitrary character and % matching an arbitrary sequence):

df.filter($"foo".like("bar"))

or rlike (like with Java regular expressions):

df.filter($"foo".rlike("bar"))

depending on your requirements. LIKE and RLIKE should work with SQL expressions as well.

Related Contents:

  1. How to define partitioning of DataFrame?
  2. Renaming column names of a DataFrame in Spark Scala
  3. Provide schema while reading csv file as a dataframe in Scala Spark
  4. DataFrame equality in Apache Spark
  5. How do I detect if a Spark DataFrame has a column
  6. How do I check for equality using Spark Dataframe without SQL Query?
  7. Joining Spark dataframes on the key
  8. Derive multiple columns from a single column in a Spark DataFrame
  9. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  10. How to change a column position in a spark dataframe?
  11. Aggregating multiple columns with custom function in Spark
  12. Create new Dataframe with empty/null field values
  13. Spark dataframe get column value into a string variable
  14. How to select the first row of each group?
  15. How can I change column types in Spark SQL’s DataFrame?
  16. How to sort by column in descending order in Spark SQL?
  17. Spark – load CSV file as DataFrame?
  18. How to convert rdd object to dataframe in spark
  19. How to create an empty DataFrame with a specified schema?
  20. Extract column values of Dataframe as List in Apache Spark
  21. How to save DataFrame directly to Hive?
  22. Get current number of partitions of a DataFrame
  23. How to write unit tests in Spark 2.0+?
  24. how to filter out a null value from spark dataframe
  25. How to use Column.isin with list?
  26. Querying Spark SQL DataFrame with complex types
  27. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  28. How to avoid duplicate columns after join?
  29. How to aggregate values into collection after groupBy?
  30. Spark unionAll multiple dataframes
  31. Automatically and Elegantly flatten DataFrame in Spark SQL
  32. Fetching distinct values on a column using Spark DataFrame
  33. What are the various join types in Spark?
  34. How to convert Row of a Scala DataFrame into case class most efficiently?
  35. dataframe: how to groupBy/count then filter on count in Scala
  36. Append a column to Data Frame in Apache Spark 1.3
  37. Get the size/length of an array column
  38. Spark – extracting single value from DataFrame
  39. Spark extracting values from a Row
  40. Flattening Rows in Spark
  41. Filter Spark DataFrame by checking if value is in a list, with other criteria
  42. Reading DataFrame from partitioned parquet file
  43. Change nullable property of column in spark dataframe
  44. How to convert DataFrame to RDD in Scala?
  45. Encoder error while trying to map dataframe row to updated row
  46. Spark: Add column to dataframe conditionally
  47. Create new column with function in Spark Dataframe
  48. How to define and use a User-Defined Aggregate Function in Spark SQL?
  49. Select Specific Columns from Spark DataFrame
  50. “value $ is not a member of StringContext” – Missing Scala plugin?

Leave a Comment