pyspark.sql.SQLContext#

class pyspark.sql.SQLContext(sparkContext, sparkSession=None, jsqlContext=None)[source]#

The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.

As of Spark 2.0, this is replaced by SparkSession. However, we are keeping the class here for backward compatibility.

A SQLContext can be used to create DataFrame, register DataFrame as tables, execute SQL over tables, cache tables, and read parquet files.

Deprecated since version 3.0.0: Use SparkSession.builder.getOrCreate() instead.

Parameters:

sparkContextSparkContext: The SparkContext backing this SQLContext.
sparkSessionSparkSession: The SparkSession around which this SQLContext wraps.
jsqlContextoptional: An optional JVM Scala SQLContext. If set, we do not instantiate a new SQLContext in the JVM, instead we make all calls to this object. This is only for internal.

Examples

>>> from datetime import datetime
>>> from pyspark.sql import Row
>>> sqlContext = SQLContext(sc)
>>> allTypes = sc.parallelize([Row(i=1, s="string", d=1.0, l=1,
...     b=True, list=[1, 2, 3], dict={"s": 0}, row=Row(a=1),
...     time=datetime(2014, 8, 1, 14, 1, 5))])
>>> df = allTypes.toDF()
>>> df.createOrReplaceTempView("allTypes")
>>> sqlContext.sql('select i+1, d+1, not b, list[1], dict["s"], time, row.a '
...            'from allTypes where b and i > 0').collect()
[Row((i + 1)=2, (d + 1)=2.0, (NOT b)=False, list[1]=2,         dict[s]=0, time=datetime.datetime(2014, 8, 1, 14, 1, 5), a=1)]
>>> df.rdd.map(lambda x: (x.i, x.s, x.d, x.l, x.b, x.time, x.row.a, x.list)).collect()
[(1, 'string', 1.0, 1, True, datetime.datetime(2014, 8, 1, 14, 1, 5), 1, [1, 2, 3])]

Methods

`cacheTable`(tableName)	Caches the specified table in-memory.
`clearCache`()	Removes all cached tables from the in-memory cache.
`createDataFrame`(data[, schema, ...])	Creates a `DataFrame` from an `RDD`, a list, a `pandas.DataFrame`, or a `pyarrow.Table`.
`createExternalTable`(tableName[, path, ...])	Creates an external table based on the dataset in a data source.
`dropTempTable`(tableName)	Remove the temporary table from catalog.
`getConf`(key[, defaultValue])	Returns the value of Spark SQL configuration property for the given key.
`getOrCreate`([sc])	Get the existing SQLContext or create a new one with given SparkContext.
`newSession`()	Returns a new SQLContext as new session, that has separate SQLConf, registered temporary views and UDFs, but shared SparkContext and table cache.
`range`(start[, end, step, numPartitions])	Create a `DataFrame` with single `pyspark.sql.types.LongType` column named `id`, containing elements in a range from `start` to `end` (exclusive) with step value `step`.
`registerDataFrameAsTable`(df, tableName)	Registers the given `DataFrame` as a temporary table in the catalog.
`registerFunction`(name, f[, returnType])	An alias for `spark.udf.register()`.
`registerJavaFunction`(name, javaClassName[, ...])	An alias for `spark.udf.registerJavaFunction()`.
`setConf`(key, value)	Sets the given Spark SQL configuration property.
`sql`(sqlQuery)	Returns a `DataFrame` representing the result of the given query.
`table`(tableName)	Returns the specified table or view as a `DataFrame`.
`tableNames`([dbName])	Returns a list of names of tables in the database `dbName`.
`tables`([dbName])	Returns a `DataFrame` containing names of tables in the given database.
`uncacheTable`(tableName)	Removes the specified table from the in-memory cache.

Attributes

`read`	Returns a `DataFrameReader` that can be used to read data in as a `DataFrame`.
`readStream`	Returns a `DataStreamReader` that can be used to read data streams as a streaming `DataFrame`.
`streams`	Returns a `StreamingQueryManager` that allows managing all the `StreamingQuery` StreamingQueries active on this context.
`udf`	Returns a `UDFRegistration` for UDF registration.
`udtf`	Returns a `UDTFRegistration` for UDTF registration.