Available options in the spark.read.option()

by

Annoyingly, the documentation for the option method is in the docs for the json method. The docs on that method say the options are as follows (key — value — description):

  • primitivesAsString — true/false (default false) — infers all primitive values as a string type

  • prefersDecimal — true/false (default false) — infers all floating-point values as a decimal type. If the values do not fit in decimal, then it infers them as doubles.

  • allowComments — true/false (default false) — ignores Java/C++ style comment in JSON records

  • allowUnquotedFieldNames — true/false (default false) — allows unquoted JSON field names

  • allowSingleQuotes — true/false (default true) — allows single quotes in addition to double quotes

  • allowNumericLeadingZeros — true/false (default false) — allows leading zeros in numbers (e.g. 00012)

  • allowBackslashEscapingAnyCharacter — true/false (default false) — allows accepting quoting of all character using backslash quoting mechanism

  • allowUnquotedControlChars — true/false (default false) — allows JSON Strings to contain unquoted control characters (ASCII characters with value less than 32, including tab and line feed characters) or not.

  • mode — PERMISSIVE/DROPMALFORMED/FAILFAST (default PERMISSIVE) — allows a mode for dealing with corrupt records during parsing.

    • PERMISSIVE : when it meets a corrupted record, puts the malformed
      string into a field configured by columnNameOfCorruptRecord, and sets
      other fields to null. To keep corrupt records, an user can set a
      string type field named columnNameOfCorruptRecord in an user-defined
      schema. If a schema does not have the field, it drops corrupt records
      during parsing. When inferring a schema, it implicitly adds a
      columnNameOfCorruptRecord field in an output schema.
    • DROPMALFORMED : ignores the whole corrupted records.
    • FAILFAST : throws an exception when it meets corrupted records.

Related Contents:

  1. Apache Spark: How to use pyspark with Python 3
  2. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  3. How to convert ‘binary string’ to normal string in Python3?
  4. “for line in…” results in UnicodeDecodeError: ‘utf-8’ codec can’t decode byte
  5. How to delete last item in list?
  6. How can I convert a .py to .exe for Python?
  7. How to set class attribute with await in __init__
  8. Local (?) variable referenced before assignment [duplicate]
  9. Image library for Python 3
  10. Display all dataframe columns in a Jupyter Python Notebook
  11. How do you fix “runtimeError: package fails to pass a sanity check” for numpy and pandas?
  12. RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
  13. Python 3 – Encode/Decode vs Bytes/Str [duplicate]
  14. How to run a http server which serves a specific path?
  15. ValueError: unsupported pickle protocol: 3, python2 pickle can not load the file dumped by python 3 pickle?
  16. Type hints when unpacking a tuple?
  17. SystemError: Parent module ” not loaded, cannot perform relative import [duplicate]
  18. How to measure server response time for Python requests POST-request
  19. Change figure size and figure format in matplotlib [duplicate]
  20. How to join on multiple columns in Pyspark?
  21. Get webpage contents with Python?
  22. Not able to create super user with Django manage.py
  23. What SOAP libraries exist for Python 3.x? [closed]
  24. from __future__ import annotations
  25. How to specify python version used to create Virtual Environment?
  26. What do * (single star) and / (slash) do as independent parameters? [duplicate]
  27. How do I create a Python set with only one element?
  28. Django Admin nested inline
  29. Django ImportError: cannot import name ‘render_to_response’ from ‘django.shortcuts’
  30. Why use contextlib.suppress as opposed to try/except with pass?
  31. Forbidden (403) CSRF verification failed. Request aborted. Reason given for failure: Origin checking failed does not match any trusted origins
  32. How to use type hints in python 3.6?
  33. How can I learn more about Python’s internals? [closed]
  34. Which key failed in Python KeyError?
  35. What does python3 open “x” mode do?
  36. Invalid character in identifier
  37. Seeking from end of file throwing unsupported exception
  38. Wrap an open stream with io.TextIOWrapper
  39. Is there a way to set a default parameter equal to another parameter value?
  40. Find how many lines in string
  41. Python: AttributeError: ‘_io.TextIOWrapper’ object has no attribute ‘split’
  42. datetime range filter in PySpark SQL
  43. Python-3.2 coroutine: AttributeError: ‘generator’ object has no attribute ‘next’ [duplicate]
  44. Should 3.4 enums use UPPER_CASE_WITH_UNDERSCORES?
  45. urllib.request.urlopen(url) with Authentication
  46. ModuleNotFoundError when running script from Terminal
  47. What is a good practice to check if an environment variable exists or not?
  48. How to run headless Firefox with Selenium in Python?
  49. “Unicode Error ‘unicodeescape’ codec can’t decode bytes…” when writing Windows file paths [duplicate]
  50. How do I append a string to a Path?

Leave a Comment