Spark Security

• Spark Security: Things You Need To Know
• Spark RPC (Communication protocol between Spark processes)
  • Authentication
    • YARN
    • Kubernetes
• Network Encryption
  • SSL Encryption (Preferred)
  • AES-based Encryption (Legacy)
• Local Storage Encryption
• Web UI
  • Authentication and Authorization
  • Spark History Server ACLs
  • SSL Configuration
  • Preparing the key stores
    • YARN mode
    • Standalone mode
  • HTTP Security Headers
• Configuring Ports for Network Security
  • Standalone mode only
  • All cluster managers
• Kerberos
  • Long-Running Applications
    • Using a Keytab
    • Using a ticket cache
  • Proxy User
  • Secure Interaction with Kubernetes
• OIDC Credential Propagation
  • How it works
  • Security model
  • Custom CredentialProvider
  • Provider-declared configuration and driver-side access
  • Configuration
  • AWS reference provider
    • Enabling the AWS reference provider
    • Using a non-AWS STS-compatible endpoint
    • AWS reference provider configuration
• Event Logging
• Persisting driver logs in client mode

Spark Security: Things You Need To Know

Security features like authentication are not enabled by default. When deploying a cluster that is open to the internet or an untrusted network, it’s important to secure access to the cluster to prevent unauthorized applications from running on the cluster.

Spark supports multiple deployment types and each one supports different levels of security. Not all deployment types will be secure in all environments and none are secure by default. Be sure to evaluate your environment, what Spark supports, and take the appropriate measure to secure your Spark deployment.

There are many different types of security concerns. Spark does not necessarily protect against all things. Listed below are some of the things Spark supports. Also check the deployment documentation for the type of deployment you are using for deployment specific settings. Anything not documented, Spark does not support.

Spark RPC (Communication protocol between Spark processes)

Authentication

Spark currently supports authentication for RPC channels using a shared secret. Authentication can be turned on by setting the spark.authenticate configuration parameter.

The exact mechanism used to generate and distribute the shared secret is deployment-specific. Unless specified below, the secret must be defined by setting the spark.authenticate.secret config option. The same secret is shared by all Spark applications and daemons in that case, which limits the security of these deployments, especially on multi-tenant clusters.

The REST Submission Server supports HTTP Authorization header with a cryptographically signed JSON Web Token via JWSFilter. To enable authorization, Spark Master should have spark.master.rest.filters=org.apache.spark.ui.JWSFilter and spark.org.apache.spark.ui.JWSFilter.param.secretKey=BASE64URL-ENCODED-KEY configurations, and client should provide HTTP Authorization header which contains JSON Web Token signed by the shared secret key.

YARN

For Spark on YARN, Spark will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret. In the case of YARN, this feature relies on YARN RPC encryption being enabled for the distribution of secrets to be secure.

Kubernetes

On Kubernetes, Spark will also automatically generate an authentication secret unique to each application. The secret is propagated to executor pods using environment variables. This means that any user that can list pods in the namespace where the Spark application is running can also see their authentication secret. Access control rules should be properly set up by the Kubernetes admin to ensure that Spark authentication is secure.

Alternatively, one can mount authentication secrets using files and Kubernetes secrets that the user mounts into their pods.

Note that when using files, Spark will not mount these files into the containers for you. It is up you to ensure that the secret files are deployed securely into your containers and that the driver’s secret file agrees with the executors’ secret file.

Network Encryption

Spark supports two mutually exclusive forms of encryption for RPC connections:

The preferred method uses TLS (aka SSL) encryption via Netty’s support for SSL. Enabling SSL requires keys and certificates to be properly configured. SSL is standardized and considered more secure.

The legacy method is an AES-based encryption mechanism relying on a shared secret. This requires RPC authentication to also be enabled. This method uses a bespoke protocol and it is recommended to use SSL instead.

One may prefer to use the SSL based encryption in scenarios where compliance mandates the usage of specific protocols; or to leverage the security of a more standard encryption library. However, the AES based encryption is simpler to configure and may be preferred if the only requirement is that data be encrypted in transit.

If both options are enabled in the configuration, the SSL based RPC encryption takes precedence and the AES based encryption will not be used (and a warning message will be emitted).

SSL Encryption (Preferred)

Spark supports SSL based encryption for RPC connections. Please refer to the SSL Configuration section below to understand how to configure it. The SSL settings are mostly similar across the UI and RPC, however there are a few additional settings which are specific to the RPC implementation. The RPC implementation uses Netty under the hood (while the UI uses Jetty), which supports a different set of options.

Unlike the other SSL settings for the UI, the RPC SSL is not automatically enabled if spark.ssl.enabled is set. It must be explicitly enabled, to ensure a safe migration path for users upgrading Spark versions.

AES-based Encryption (Legacy)

Spark supports AES-based encryption for RPC connections. For encryption to be enabled, RPC authentication must also be enabled and properly configured. AES encryption uses the Apache Commons Crypto library, and Spark’s configuration system allows access to that library’s configuration for advanced users.

This legacy protocol has two mutually incompatible versions. Version 1 omits applying key derivation function (KDF) to the key exchange protocol’s output, while version 2 applies a KDF to ensure that the derived session key is uniformly distributed. Version 1 is default for backward compatibility. It is recommended to use version 2 for better security properties. The version can be configured by setting spark.network.crypto.authEngineVersion to 1 or 2 respectively.

There is also support for SASL-based encryption, although it should be considered deprecated. It is still required when talking to shuffle services from Spark versions older than 2.2.0.

The following table describes the different options available for configuring this feature.

Local Storage Encryption

Spark supports encrypting temporary data written to local disks. This covers shuffle files, shuffle spills and data blocks stored on disk (for both caching and broadcast variables). It does not cover encrypting output data generated by applications with APIs such as saveAsHadoopFile or saveAsTable. It also may not cover temporary files created explicitly by the user.

The following settings cover enabling encryption for data written to disk:

Web UI

Authentication and Authorization

Enabling authentication for the Web UIs is done using jakarta servlet filters. You will need a filter that implements the authentication method you want to deploy. Spark does not provide any built-in authentication filters.

Spark also supports access control to the UI when an authentication filter is present. Each application can be configured with its own separate access control lists (ACLs). Spark differentiates between “view” permissions (who is allowed to see the application’s UI), and “modify” permissions (who can do things like kill jobs in a running application).

ACLs can be configured for either users or groups. Configuration entries accept comma-separated lists as input, meaning multiple users or groups can be given the desired privileges. This can be used if you run on a shared cluster and have a set of administrators or developers who need to monitor applications they may not have started themselves. A wildcard (*) added to specific ACL means that all users will have the respective privilege. By default, only the user submitting the application is added to the ACLs.

Group membership is established by using a configurable group mapping provider. The mapper is configured using the spark.user.groups.mapping config option, described in the table below.

The following options control the authentication of Web UIs:

On YARN, the view and modify ACLs are provided to the YARN service when submitting applications, and control who has the respective privileges via YARN interfaces.

Spark History Server ACLs

Authentication for the SHS Web UI is enabled the same way as for regular applications, using servlet filters.

To enable authorization in the SHS, a few extra options are used:

The SHS uses the same options to configure the group mapping provider as regular applications. In this case, the group mapping provider will apply to all UIs server by the SHS, and individual application configurations will be ignored.

SSL Configuration

Configuration for SSL is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. Note that all settings are inherited this way, except for spark.ssl.rpc.enabled which must be explicitly set.

The following table describes the SSL configuration namespaces:

The full breakdown of available SSL options can be found below. The ${ns} placeholder should be replaced with one of the above namespaces.

Spark also supports retrieving ${ns}.keyPassword, ${ns}.keyStorePassword and ${ns}.trustStorePassword from Hadoop Credential Providers. User could store password into credential file and make it accessible by different components, like:

hadoop credential create spark.ssl.keyPassword -value password \
    -provider jceks://[email protected]:9001/user/backup/ssl.jceks

To configure the location of the credential provider, set the hadoop.security.credential.provider.path config option in the Hadoop configuration used by Spark, like:

  <property>
    <name>hadoop.security.credential.provider.path</name>
    <value>jceks://[email protected]:9001/user/backup/ssl.jceks</value>
  </property>

Or via SparkConf “spark.hadoop.hadoop.security.credential.provider.path=jceks://[email protected]:9001/user/backup/ssl.jceks”.

Preparing the key stores

Key stores can be generated by keytool program. The reference documentation for this tool for Java 17 is here. The most basic steps to configure the key stores and the trust store for a Spark Standalone deployment mode is as follows:

• Generate a key pair for each node
• Export the public key of the key pair to a file on each node
• Import all exported public keys into a single trust store
• Distribute the trust store to the cluster nodes

YARN mode

To provide a local trust store or key store file to drivers running in cluster mode, they can be distributed with the application using the --files command line argument (or the equivalent spark.files configuration). The files will be placed on the driver’s working directory, so the TLS configuration should just reference the file name with no absolute path.

Distributing local key stores this way may require the files to be staged in HDFS (or other similar distributed file system used by the cluster), so it’s recommended that the underlying file system be configured with security in mind (e.g. by enabling authentication and wire encryption).

Standalone mode

The user needs to provide key stores and configuration options for master and workers. They have to be set by attaching appropriate Java system properties in SPARK_MASTER_OPTS and in SPARK_WORKER_OPTS environment variables, or just in SPARK_DAEMON_JAVA_OPTS.

The user may allow the executors to use the SSL settings inherited from the worker process. That can be accomplished by setting spark.ssl.useNodeLocalConf to true. In that case, the settings provided by the user on the client side are not used.

HTTP Security Headers

Apache Spark can be configured to include HTTP headers to aid in preventing Cross Site Scripting (XSS), Cross-Frame Scripting (XFS), MIME-Sniffing, and also to enforce HTTP Strict Transport Security.

Configuring Ports for Network Security

Generally speaking, a Spark cluster and its services are not deployed on the public internet. They are generally private services, and should only be accessible within the network of the organization that deploys Spark. Access to the hosts and ports used by Spark services should be limited to origin hosts that need to access the services.

However, like the REST Submission port, Spark also supports HTTP Authorization header with a cryptographically signed JSON Web Token (JWT) for all UI ports. To use it, a user needs to configure spark.ui.filters=org.apache.spark.ui.JWSFilter and spark.org.apache.spark.ui.JWSFilter.param.secretKey=BASE64URL-ENCODED-KEY.

Below are the primary ports that Spark uses for its communication and how to configure those ports.

Standalone mode only

All cluster managers

Kerberos

Spark supports submitting applications in environments that use Kerberos for authentication. In most cases, Spark relies on the credentials of the current logged in user when authenticating to Kerberos-aware services. Such credentials can be obtained by logging in to the configured KDC with tools like kinit.

When talking to Hadoop-based services, Spark needs to obtain delegation tokens so that processes can authenticate. Spark ships with support for HDFS and other Hadoop file systems, Hive and HBase.

When using a Hadoop filesystem (such HDFS or WebHDFS), Spark will acquire the relevant tokens for the service hosting the user’s home directory.

An HBase token will be obtained if HBase is in the application’s classpath, and the HBase configuration has Kerberos authentication turned (hbase.security.authentication=kerberos).

Similarly, a Hive token will be obtained if Hive is in the classpath, and the configuration includes URIs for remote metastore services (hive.metastore.uris is not empty).

If an application needs to interact with other secure Hadoop filesystems, their URIs need to be explicitly provided to Spark at launch time. This is done by listing them in the spark.kerberos.access.hadoopFileSystems property, described in the configuration section below.

Spark also supports custom delegation token providers using the Java Services mechanism (see java.util.ServiceLoader). Implementations of org.apache.spark.security.HadoopDelegationTokenProvider can be made available to Spark by listing their names in the corresponding file in the jar’s META-INF/services directory.

Delegation token support is currently only supported in Local, YARN and Kubernetes modes. Consult the deployment-specific page for more information.

The following options provides finer-grained control for this feature:

Users can exclude Kerberos delegation token renewal at resource scheduler. Currently it is only supported on YARN. The configuration is covered in the Running Spark on YARN page.

Long-Running Applications

Long-running applications may run into issues if their run time exceeds the maximum delegation token lifetime configured in services it needs to access.

This feature is not available everywhere. In particular, it’s only implemented on YARN and Kubernetes (both client and cluster modes).

Spark supports automatically creating new tokens for these applications. There are two ways to enable this functionality.

Using a Keytab

By providing Spark with a principal and keytab (e.g. using spark-submit with --principal and --keytab parameters), the application will maintain a valid Kerberos login that can be used to retrieve delegation tokens indefinitely.

Note that when using a keytab in cluster mode, it will be copied over to the machine running the Spark driver. In the case of YARN, this means using HDFS as a staging area for the keytab, so it’s strongly recommended that both YARN and HDFS be secured with encryption, at least.

Using a ticket cache

By setting spark.kerberos.renewal.credentials to ccache in Spark’s configuration, the local Kerberos ticket cache will be used for authentication. Spark will keep the ticket renewed during its renewable life, but after it expires a new ticket needs to be acquired (e.g. by running kinit).

It’s up to the user to maintain an updated ticket cache that Spark can use.

The location of the ticket cache can be customized by setting the KRB5CCNAME environment variable.

Proxy User

Spark also provides --proxy-user parameter for spark-submit to enable Hadoop’s Proxy user feature.

If the target cluster, e.g., YARN, HDFS, is running in Secure Mode, the superuser must have a valid Kerberos ticket to log in. The impersonated user (proxy-user) does not need to have a Kerberos ticket. In addition, the superuser must be configured in the cluster to be allowed to impersonate the proxy user.

The authentication happens on the target cluster side, and once the authentication is successful, the cluster will do resource allocation and file system access on behalf of the proxy user.

Note that, depending on Spark’s deployment mode, the proxy user might behave differently. For cluster mode, the JVM running the driver will be started by the proxy user, while for client mode, it will be started by the superuser instead. This is due to the Driver being initialized inside the progress of SparkSubmit client. This makes a difference in file system access for local file permissions. This is not considered a CVE issue, but users should be aware of this difference.

Nowadays, many projects, such as Apache Kyuubi, provide a multi-tenant Spark service with impersonation. To prevent server-side local files from reading and leaking by superuser to other tenants, it is recommended to refer to their documentation to find out instructions on how to ensure cluster mode is used for a more secure purpose.

Secure Interaction with Kubernetes

When talking to Hadoop-based services behind Kerberos, it was noted that Spark needs to obtain delegation tokens so that processes can authenticate. These delegation tokens in Kubernetes are stored in Secrets that are shared by the Driver and its Executors. As such, there are three ways of submitting a Kerberos job:

In all cases you must define the environment variable: HADOOP_CONF_DIR or spark.kubernetes.hadoop.configMapName.

It also important to note that the KDC needs to be visible from inside the containers.

If a user wishes to use a remote HADOOP_CONF directory, that contains the Hadoop configuration files, this could be achieved by setting spark.kubernetes.hadoop.configMapName to a pre-existing ConfigMap.

1. Submitting with a $kinit that stores a TGT in the Local Ticket Cache:

   /usr/bin/kinit -kt <keytab_file> <username>/<krb5 realm>
   /opt/spark/bin/spark-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.HdfsTest \
    --master k8s://<KUBERNETES_MASTER_ENDPOINT> \
    --conf spark.executor.instances=1 \
    --conf spark.app.name=spark-hdfs \
    --conf spark.kubernetes.container.image=spark:latest \
    --conf spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf \
    local:///opt/spark/examples/jars/spark-examples_<VERSION>.jar \
    <HDFS_FILE_LOCATION>
   

2. Submitting with a local Keytab and Principal

   /opt/spark/bin/spark-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.HdfsTest \
    --master k8s://<KUBERNETES_MASTER_ENDPOINT> \
    --conf spark.executor.instances=1 \
    --conf spark.app.name=spark-hdfs \
    --conf spark.kubernetes.container.image=spark:latest \
    --conf spark.kerberos.keytab=<KEYTAB_FILE> \
    --conf spark.kerberos.principal=<PRINCIPAL> \
    --conf spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf \
    local:///opt/spark/examples/jars/spark-examples_<VERSION>.jar \
    <HDFS_FILE_LOCATION>
   

3. Submitting with pre-populated secrets, that contain the Delegation Token, already existing within the namespace

   /opt/spark/bin/spark-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.HdfsTest \
    --master k8s://<KUBERNETES_MASTER_ENDPOINT> \
    --conf spark.executor.instances=1 \
    --conf spark.app.name=spark-hdfs \
    --conf spark.kubernetes.container.image=spark:latest \
    --conf spark.kubernetes.kerberos.tokenSecret.name=<SECRET_TOKEN_NAME> \
    --conf spark.kubernetes.kerberos.tokenSecret.itemKey=<SECRET_ITEM_KEY> \
    --conf spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf \
    local:///opt/spark/examples/jars/spark-examples_<VERSION>.jar \
    <HDFS_FILE_LOCATION>
   

3b. Submitting like in (3) however specifying a pre-created krb5 ConfigMap and pre-created HADOOP_CONF_DIR ConfigMap

/opt/spark/bin/spark-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.HdfsTest \
    --master k8s://<KUBERNETES_MASTER_ENDPOINT> \
    --conf spark.executor.instances=1 \
    --conf spark.app.name=spark-hdfs \
    --conf spark.kubernetes.container.image=spark:latest \
    --conf spark.kubernetes.kerberos.tokenSecret.name=<SECRET_TOKEN_NAME> \
    --conf spark.kubernetes.kerberos.tokenSecret.itemKey=<SECRET_ITEM_KEY> \
    --conf spark.kubernetes.hadoop.configMapName=<HCONF_CONFIG_MAP_NAME> \
    --conf spark.kubernetes.kerberos.krb5.configMapName=<KRB_CONFIG_MAP_NAME> \
    local:///opt/spark/examples/jars/spark-examples_<VERSION>.jar \
    <HDFS_FILE_LOCATION>

OIDC Credential Propagation

Spark supports propagating short-lived, per-workload or per-user credentials to executors in environments that use OIDC / OAuth 2.0 for authentication, such as Spark on Kubernetes accessing cloud object storage. This generalizes the delegation-token propagation model described in the Kerberos section above to OIDC-based systems: instead of obtaining Hadoop delegation tokens from a KDC, the driver reads an OIDC identity token (a JWT), exchanges it for short-lived service credentials, and distributes those credentials to executors, refreshing them automatically for long-running applications.

This feature is disabled by default. It is enabled by setting spark.security.oidc.enabled=true and providing an identity token file via spark.security.oidc.identityToken.file. When disabled, none of the components described below are started and existing applications are unaffected.

As with the Kerberos path, Spark’s role here is limited to propagating delegated credentials, not to authenticating users or enforcing access control. The downstream service (for example, a cloud IAM system or an object store) performs authorization using the credential Spark propagates. The raw identity token is consumed only on the driver; executors receive only the short-lived service credentials derived from it.

How it works

When enabled, credential propagation runs on independent threads alongside the Kerberos delegation token machinery, and the two coexist without interfering with each other. A cluster that accesses HDFS via Kerberos and cloud storage via OIDC at the same time is a supported configuration.

Credential propagation targets Spark on Kubernetes accessing cloud object storage, which is the deployment this feature is designed and documented for. See Running Spark on Kubernetes for complete configuration examples.

1. On the driver, a token ingestor reads the OIDC identity token from the configured file and produces a driver-only user context. Kubernetes projected ServiceAccount tokens (which are rotated automatically by the kubelet) and externally-injected per-user tokens are both supported, because the ingestor detects file rotation and reloads the token.
2. A driver-side credential manager (a sibling of the Kerberos delegation token manager) passes the user context to a CredentialProvider for each configured scheme, obtaining a short-lived service credential.
3. The service credentials are serialized and distributed to all registered executors via an RPC message. Newly-registered executors (for example, those created by dynamic allocation) receive the current credentials as part of their registration response, and each launched task also carries the current credentials in its TaskDescription, so executors have valid credentials before running any task. All of these delivery paths travel over Spark’s RPC channels and are covered by the same RPC encryption.
4. The manager schedules the next renewal ahead of the earliest expiry (the minimum of the identity token expiry and the service credential expiry, minus a configurable safety margin), and retries with exponential backoff on failure.
5. On the executor, connector-specific code reads the current service credential from an in-memory store on each access, so credential refreshes are picked up without restarting tasks or invalidating filesystem caches.

The components mirror the existing Kerberos delegation token path:

Security model

The security model closely follows the Kerberos delegation token model, where executors receive delegation tokens rather than the original ticket-granting ticket.

• The raw identity token never leaves the driver. Executors receive only short-lived, scoped service credentials produced by a CredentialProvider. A service credential is narrower and shorter-lived than the original identity token, but should still be treated as sensitive.
• The driver-side user context that holds the raw token is never serialized or transmitted (it is not Serializable), and it redacts the raw token in its toString() representation, so the token is not written to logs.
• Service credentials are not written to shuffle files, event logs, or checkpoints.
• Credentials are carried over Spark’s RPC channels and therefore rely on Spark’s existing RPC encryption configuration. Operators are strongly encouraged to enable RPC encryption whenever credential propagation is enabled. See Network Encryption for details.

Custom CredentialProvider

The credential exchange step is pluggable. Spark discovers implementations of CredentialProvider using the Java Services mechanism (see java.util.ServiceLoader): an implementation is made available to Spark by listing its fully-qualified class name in the corresponding file in the jar’s META-INF/services directory, exactly as with custom HadoopDelegationTokenProvider implementations for the Kerberos path.

A CredentialProvider declares the URI schemes it supports (for example, s3a) and exchanges the driver-side user context for a ServiceCredential scoped to a target URI. When more than one provider is registered for the same scheme, the provider used for that scheme is selected via spark.security.oidc.provider.<scheme>. The configuration passed to a provider is scoped to keys starting with spark.security.oidc., so unrelated configuration is not exposed to third-party providers.

The CredentialProvider SPI and its related types (UserContext, ServiceCredential, and UserCredentials) are annotated @DeveloperApi and may evolve in minor releases.

Spark ships a reference implementation for AWS S3 / STS-compatible endpoints in an optional module. See AWS reference provider below for how to enable and configure it, and Running Spark on Kubernetes for complete configuration examples using Kubernetes projected ServiceAccount tokens and per-user identity tokens.

Provider-declared configuration and driver-side access

A CredentialProvider may declare additional Spark configuration properties that should be set whenever it is selected (via CredentialProvider.additionalSparkProperties()), for example the Hadoop credentials-provider class for a filesystem scheme. Spark applies these properties on both the driver and the executors – only for keys the user has not already set explicitly – before the components that consume them are initialized. This means that, with the feature enabled and no explicit provider wiring configured, both the driver and the executors access storage using the propagated OIDC credentials.

There are two situations, both specific to accessing storage from the driver very early in startup, where the propagated credentials are not yet available. Both only matter when the driver itself accesses cloud storage (a common case in cluster mode: output-path existence checks, the commit protocol, and schema inference all run on the driver).

• Access during SparkContext construction. The provider wiring is applied early in SparkContext initialization, but the credentials it points at are not resolved until the scheduler backend starts, slightly later in the same initialization. Any driver-side storage access that happens in between runs before the credentials exist and therefore cannot use them. This affects resources fetched or validated during construction from a scheme served by a CredentialProvider (for example s3a://): spark.jars, spark.files, spark.archives, and spark.checkpoint.dir. Note the two failure shapes: spark.files / spark.archives and spark.checkpoint.dir propagate the error and fail SparkContext construction, while spark.jars swallows it (the jar is silently dropped, and tasks later fail with ClassNotFoundException). Prefer local:// for such resources, or stage them through a location that does not require the propagated credentials.

• Hadoop FileSystem cache in Kubernetes cluster mode. In cluster mode on Kubernetes the driver pod runs SparkSubmit in the same JVM as SparkContext. When SparkSubmit downloads the application resource, --jars, or --files from a bucket (for example --jars s3a://BUCKET/app.jar), it does so before provider selection runs, and the resulting FileSystem instance is cached (the cache key is scheme + authority + user). A later driver-side access to the same bucket – such as writing job output to it – reuses that cached FileSystem, which still uses the default credential chain (typically the pod’s service account) rather than the propagated OIDC identity. The job does not fail; instead the driver’s access to that bucket runs under a different identity than the executors, which is easy to overlook. To avoid this, use local:// for --jars / --files, or disable the S3A filesystem cache (for example, spark.hadoop.fs.s3a.impl.disable.cache=true), so the driver builds a fresh FileSystem that picks up the propagated provider. Note that this disables the cache for the entire s3a scheme (not just one bucket), which has a performance cost because a new FileSystem is created per access instead of being reused.

Configuration

The following options control the core (cloud-agnostic) credential propagation framework. Provider-specific options (such as those for the AWS reference provider) are documented alongside each provider.

AWS reference provider

Spark ships a reference CredentialProvider implementation for Amazon S3 and any STS-compatible endpoint (such as MinIO or Ceph) in the optional credential-aws module. It calls sts:AssumeRoleWithWebIdentity with the identity token and a configured IAM role, and returns temporary credentials as the fs.s3a.access.key, fs.s3a.secret.key, and fs.s3a.session.token properties consumed by the S3A connector.

Enabling the AWS reference provider

The reference provider is packaged in the credential-aws module, which is not part of the default Spark distribution (it is kept separate so that the AWS SDK is not added to the core classpath).

The recommended way to make it available (especially for cluster deployments such as Spark on Kubernetes) is to build a custom Spark distribution or container image that includes the module, using the -Pcredential-aws profile. Reading and writing S3 data additionally requires the S3A connector (hadoop-aws), provided by the hadoop-cloud module (the -Phadoop-cloud profile); the two modules are designed to be used together and pin the AWS SDK to the same version, so build with both:

./dev/make-distribution.sh --pip -Pkubernetes -Pcredential-aws -Phadoop-cloud

Baking the module into the image keeps executor startup self-contained and avoids each executor resolving artifacts from a remote repository at launch time, which matters when many executors start (for example, with dynamic allocation).

Alternatively, for interactive use or one-off jobs, the module can be fetched from Maven Central at submit time with --packages (this also works in Kubernetes cluster mode):

--packages org.apache.spark:spark-credential-aws_2.13:In-Progress

Once the module is on the classpath, the provider is discovered automatically via ServiceLoader. Enable credential propagation and point Spark at the identity token file (on Kubernetes this is typically a projected ServiceAccount token), then configure the IAM role to assume:

spark.security.oidc.enabled                 true
spark.security.oidc.identityToken.file      /var/run/secrets/oidc/token
spark.security.oidc.aws.roleArn             arn:aws:iam::123456789012:role/spark-data-access

When credential propagation is enabled and you have not set spark.hadoop.fs.s3a.aws.credentials.provider in your Spark configuration, Spark automatically configures the S3A connector on the driver and executors to read the propagated credentials by setting:

spark.hadoop.fs.s3a.aws.credentials.provider org.apache.spark.security.aws.SparkOidcAwsCredentialsProvider

Only the spark.hadoop.fs.s3a.aws.credentials.provider key in the Spark configuration is considered. If you set it yourself, Spark does not override it. A value set elsewhere – for example fs.s3a.aws.credentials.provider in core-site.xml – is not detected, and the injected spark.hadoop.* value takes precedence over it.

For complete, deployment-ready examples on Kubernetes (including how to mount a projected ServiceAccount token for workload-level identity, or an externally-injected per-user identity token), see OIDC Credential Propagation on the Kubernetes page.

Using a non-AWS STS-compatible endpoint

The provider works with any STS-compatible endpoint by setting spark.security.oidc.aws.stsEndpoint (and, if needed, spark.security.oidc.aws.region). For example, to target a MinIO STS endpoint:

spark.security.oidc.aws.stsEndpoint https://minio.example.com
spark.security.oidc.aws.region      us-east-1

AWS reference provider configuration

These options are read by the AWS reference provider (from the credential-aws module) and are only relevant when it is on the classpath and credential propagation is enabled.

Event Logging

If your applications are using event logging, the directory where the event logs go (spark.eventLog.dir) should be manually created with proper permissions. To secure the log files, the directory permissions should be set to drwxrwxrwxt. The owner and group of the directory should correspond to the super user who is running the Spark History Server.

This will allow all users to write to the directory but will prevent unprivileged users from reading, removing or renaming a file unless they own it. The event log files will be created by Spark with permissions such that only the user and group have read and write access.

Persisting driver logs in client mode

If your applications persist driver logs in client mode by enabling spark.driver.log.persistToDfs.enabled, the directory where the driver logs go (spark.driver.log.dfsDir) should be manually created with proper permissions. To secure the log files, the directory permissions should be set to drwxrwxrwxt. The owner and group of the directory should correspond to the super user who is running the Spark History Server.

This will allow all users to write to the directory but will prevent unprivileged users from reading, removing or renaming a file unless they own it. The driver log files will be created by Spark with permissions such that only the user and group have read and write access.