Apache Spark 3.3.0 breaks on Java 17 with “cannot access class sun.nio.ch.DirectBuffer”

by

Solution

A similar question was asked at Running unit tests with Spark 3.3.0 on Java 17 fails with IllegalAccessError: class StorageUtils cannot access class sun.nio.ch.DirectBuffer, but that question (and solution) was only about unit tests. For me Spark is breaking actually running the program.

Please, consider adding the appropriate Java Virtual Machine command-line options.
The exact way to add them depends on how you run the program: by using a command line, an IDE, etc.

Examples

The command-line options have been taken from the JavaModuleOptions class: spark/JavaModuleOptions.java at v3.3.0 · apache/spark.

Command line

For example, to run the program (the .jar file) by using the command line:

java \
    --add-opens=java.base/java.lang=ALL-UNNAMED \
    --add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
    --add-opens=java.base/java.io=ALL-UNNAMED \
    --add-opens=java.base/java.net=ALL-UNNAMED \
    --add-opens=java.base/java.nio=ALL-UNNAMED \
    --add-opens=java.base/java.util=ALL-UNNAMED \
    --add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
    --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED \
    --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
    --add-opens=java.base/sun.nio.cs=ALL-UNNAMED \
    --add-opens=java.base/sun.security.action=ALL-UNNAMED \
    --add-opens=java.base/sun.util.calendar=ALL-UNNAMED \
    --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED \
    -jar <JAR_FILE_PATH>

IDE: IntelliJ IDEA

References:

References

Related Contents:

  1. How do I install Java on Mac OSX allowing version switching?
  2. Add JAR files to a Spark job – spark-submit
  3. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  4. Spark Error – Unsupported class file major version
  5. IntelliJ – Invalid source release: 17
  6. How to flatten a struct in a Spark dataframe?
  7. PySpark: java.lang.OutofMemoryError: Java heap space
  8. Difference in Used, Committed and Max Heap Memory
  9. OpenJDK Java 17 docker image
  10. aggregate function Count usage with groupBy in Spark
  11. Spark sql how to explode without losing null values
  12. What are sealed classes in Java 17?
  13. Spark read file from S3 using sc.textFile (“s3n://…)
  14. Apache Spark – foreach Vs foreachPartition When to use What?
  15. Is gzip format supported in Spark?
  16. How to run a Spark Java program
  17. How to set amount of Spark executors?
  18. What are the Spark transformations that causes a Shuffle?
  19. How to run Spark code in Airflow?
  20. Resolving dependency problems in Apache Spark
  21. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  22. How do I create a file and write to it?
  23. NullPointerException in Java with no StackTrace
  24. What is an uber jar?
  25. JAX-RS / Jersey how to customize error handling?
  26. Unable to locate tools.jar
  27. What is a good use case for static import of methods?
  28. Default fetch type for one-to-one, many-to-one and one-to-many in Hibernate
  29. A Java API to generate Java source files [closed]
  30. Best approach to converting Boolean object to string in java
  31. What exactly is a Context in Java? [duplicate]
  32. ExecutorService that interrupts tasks after a timeout
  33. can someone please explain me @MapsId in hibernate?
  34. Spring Data JPA difference between findBy / findAllBy
  35. Apostrophe doesn’t get translated properly when placed in a resource bundle
  36. Lambda that does absolutely nothing
  37. How to get the fields in an Object via reflection?
  38. Any simple way to test null before convert an object to string
  39. High performance serialization: Java vs Google Protocol Buffers vs …?
  40. Why does java.util.Properties implement Map and not Map
  41. Create a mutable java.lang.String
  42. Spring Webflux : Webclient : Get body on error
  43. Java Queues – why “poll” and “offer”?
  44. What are helper objects in java?
  45. How to select an item from a dropdown list using Selenium WebDriver with java?
  46. Using Java with Microsoft Visual Studio 2012
  47. Android programmatically include layout (i.e. without XML)
  48. When to use intern() on String literals
  49. What is the use of Map.ofEntries() instead of Map.of()
  50. Specifying JVM arguments when calling a jar file?

Leave a Comment