MapType#

class pyspark.sql.types.MapType(keyType, valueType, valueContainsNull=True)[source]#

Map data type.

Parameters:

keyTypeDataType: DataType of the keys in the map.
valueTypeDataType: DataType of the values in the map.
valueContainsNullbool, optional: indicates whether values can contain null (None) values.

Notes

Keys in a map data type are not allowed to be null (None).

Examples

>>> from pyspark.sql.types import IntegerType, FloatType, MapType, StringType

The below example demonstrates how to create class:MapType:

>>> map_type = MapType(StringType(), IntegerType())

The values of the map can contain null (None) values by default:

>>> (MapType(StringType(), IntegerType())
...        == MapType(StringType(), IntegerType(), True))
True
>>> (MapType(StringType(), IntegerType(), False)
...        == MapType(StringType(), FloatType()))
False

Methods

`fromDDL`(ddl)	Creates `DataType` for a given DDL-formatted string.
`fromInternal`(obj)	Converts an internal SQL object into a native Python object.
`fromJson`(json[, fieldPath, collationsMap])
`json`()
`jsonValue`()
`needConversion`()	Does this type needs conversion between Python object and internal SQL object.
`simpleString`()
`toInternal`(obj)	Converts a Python object into an internal SQL object.
`toNullable`()	Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).
`typeName`()

Methods Documentation

classmethod fromDDL(ddl)#

Creates DataType for a given DDL-formatted string.

Added in version 4.0.0.

Parameters:

ddlstr: DDL-formatted string representation of types, e.g. pyspark.sql.types.DataType.simpleString, except that top level struct type can omit the struct<> for the compatibility reason with spark.createDataFrame and Python UDFs.

Returns:

DataType

Examples

Create a StructType by the corresponding DDL formatted string.

>>> from pyspark.sql.types import DataType
>>> DataType.fromDDL("b string, a int")
StructType([StructField('b', StringType(), True), StructField('a', IntegerType(), True)])

Create a single DataType by the corresponding DDL formatted string.

>>> DataType.fromDDL("decimal(10,10)")
DecimalType(10,10)

Create a StructType by the legacy string format.

>>> DataType.fromDDL("b: string, a: int")
StructType([StructField('b', StringType(), True), StructField('a', IntegerType(), True)])

fromInternal(obj)[source]#: Converts an internal SQL object into a native Python object.

classmethod fromJson(json, fieldPath='', collationsMap=None)[source]#

json()#

jsonValue()[source]#

needConversion()[source]#

Does this type needs conversion between Python object and internal SQL object.

This is used to avoid the unnecessary conversion for ArrayType/MapType/StructType.

simpleString()[source]#

toInternal(obj)[source]#: Converts a Python object into an internal SQL object.

toNullable()[source]#

Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).

Added in version 4.0.0.

Returns:

MapType

Examples

Example 1: Simple nullability conversion

>>> MapType(IntegerType(), StringType(), valueContainsNull=False).toNullable()
MapType(IntegerType(), StringType(), True)

Example 2: Nested nullability conversion

>>> MapType(
...     StringType(),
...     MapType(
...         IntegerType(),
...         ArrayType(IntegerType(), containsNull=False),
...         valueContainsNull=False
...     ),
...     valueContainsNull=False
... ).toNullable()
MapType(StringType(), MapType(IntegerType(), ArrayType(IntegerType(), True), True), True)

classmethod typeName()#