pyspark.sql.Column.outer#

Column.outer()[source]#

Mark this column as an outer column if its expression refers to columns from an outer query.

This is used to trigger lazy analysis of Spark Classic DataFrame, so that we can use it to build subquery expressions. Spark Connect DataFrame is always lazily analyzed and does not need to use this function.

Added in version 4.0.0.

See also

pyspark.sql.DataFrame.scalar

pyspark.sql.DataFrame.exists

Examples

>>> from pyspark.sql import functions as sf
>>> employees = spark.createDataFrame(
...     [
...         (1, "Alice", 45000, 101), (2, "Bob", 54000, 101), (3, "Charlie", 29000, 102),
...         (4, "David", 61000, 102), (5, "Eve", 48000, 101),
...     ],
...     ["id", "name", "salary", "department_id"],
... )
>>> employees.alias("e1").where(
...     sf.col("salary") > employees.alias("e2").where(
...         sf.col("e2.department_id") == sf.col("e1.department_id").outer()
...     ).select(sf.avg("salary")).scalar()
... ).select("name", "salary", "department_id").orderBy("name").show()
+-----+------+-------------+
| name|salary|department_id|
+-----+------+-------------+
|  Bob| 54000|          101|
|David| 61000|          102|
+-----+------+-------------+