Pass the Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with ValidTests

To read from a specific Kafka topic using Structured Streaming, the correct syntax is:

python

CopyEdit

option("subscribe", "feed")

This is explicitly defined in the Spark documentation:

"subscribe – The Kafka topic to subscribe to. Only one topic can be specified for this option."

(Source: Apache Spark Structured Streaming + Kafka Integration Guide)

B. "subscribe.topic" is invalid.

C. "kafka.topic" is not a recognized option.

D. "topic" is not valid for Kafka source in Spark.

The default broadcast join threshold in Spark is:

spark.sql.autoBroadcastJoinThreshold = 10MB

Since df2 is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.

From the Spark documentation:

“If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors.”

A is incorrect because Spark does support auto broadcast even with static plans.

B is correct: Spark will automatically broadcast df2.

C and D are incorrect because Spark's default logic handles this optimization.

Final Answer: B

When queries frequently filter on certain columns, partitioning by those columns ensures partition pruning, allowing Spark to scan only relevant directories instead of the entire dataset.

Correct code:

final.write.partitionBy("event_year", "event_month").parquet("events.liveLatest")

This improves read performance dramatically for filters like:

SELECT * FROM events.liveLatest WHERE event_year = 2024 AND event_month = 5;

bucketBy() helps in clustering and joins, not in partition pruning for file-based tables.

Why the other options are incorrect:

B: Bucket count changes parallelism, not query pruning.

C: sortBy organizes data within files, not across partitions.

D: Partitioning by only one column limits pruning benefits.

Spark Connect enables remote execution of Spark jobs by decoupling the client from the driver using the Spark Connect protocol (gRPC).

It allows users to run Spark code from different environments (like notebooks, IDEs, or remote clients) while executing jobs on the cluster.

Key Features:

Enables remote interaction between client and Spark driver.

Supports interactive development and lightweight client sessions.

Improves developer productivity without needing driver resources locally.

Why the other options are incorrect:

A: Spark Connect is not limited to ingestion tasks.

B: It allows multi-language clients (Python, Scala, etc.) but runs via Spark Connect API, not arbitrary remote code.

C: Uses gRPC protocol, not REST.

Pandas API on Spark provides a distributed implementation of the Pandas DataFrame API on top of Apache Spark.

Advantages:

Executes transformations in parallel across all nodes and cores in the cluster.

Maintains Pandas-like syntax, making it easy for Python users to transition.

Enables scaling of existing Pandas code to handle large datasets without memory limits.

Therefore, it combines Pandas usability with Spark’s distributed power, offering both speed and scalability.

Why the other options are incorrect:

B: While it uses Python, that’s not its main advantage.

C: It runs distributed across the cluster, not on a single node.

D: Pandas API on Spark uses lazy evaluation, not eager computation.

The correct answer is B because it uses the new function count_if, introduced in Spark 3.5.0, which simplifies conditional counting within aggregations.

F.count_if(condition) counts the number of rows that meet the specified boolean condition.

In this example, it directly counts how many times spot_price >= min_price evaluates to true, replacing the older verbose combination of when/otherwise and filtering or summing.

Official Spark 3.5.0 documentation notes the addition of count_if to simplify this kind of logic:

“Added count_if aggregate function to count only the rows where a boolean condition holds (SPARK-43773).”

Why other options are incorrect or outdated:

A uses a legacy-style method of adding a flag column (when().otherwise()), which is verbose compared to count_if.

C performs a simple min/max aggregation—useful but unrelated to conditional array operations or the updated functionality.

D incorrectly applies .filter() after .agg() which will cause an error, and misuses string "min_price" rather than the variable.

Therefore, B is the only option leveraging new functionality from Spark 3.5.0 correctly and efficiently.

df.cache() is shorthand for df.persist(StorageLevel.MEMORY_AND_DISK)

df.persist() allows specifying any storage level such as MEMORY_ONLY, DISK_ONLY, MEMORY_AND_DISK_SER, etc.

By default, persist() uses MEMORY_AND_DISK, unless specified otherwise.

The original code has several issues:

It references a variable answer that is undefined.

The function is annotated to return a str, but the logic attempts numeric multiplication.

The UDF return type is declared as T.IntegerType() but the function performs a floating-point operation, which is incompatible.

Option B correctly:

Uses DoubleType to reflect the fact that the multiplication involves a float (3.14159).

Declares the input as float, which aligns with the multiplication.

Returns a float, which matches both the logic and the schema type annotation.

This structure aligns with how PySpark expects User Defined Functions (UDFs) to be declared:

"To define a UDF you must specify a Python function and provide the return type using the relevant Spark SQL type (e.g., DoubleType for float results)."

Example from official documentation:

from pyspark.sql.functions import udf

from pyspark.sql.types import DoubleType

@udf(returnType=DoubleType())

def multiply_by_pi(x: float) -> float:

return x * 3.14159

This makes Option B the syntactically and semantically correct choice.

To exclude sensitive (PII) columns from a DataFrame, the easiest method is to use the .drop() function with the list of column names to remove.

Correct syntax:

df_user_non_pii = df_user.drop("name", "email", "birthdate")

This creates a new DataFrame containing all remaining columns.

Why the other options are incorrect:

B: .dropFields() is not valid for standard DataFrames — it’s used for struct fields only.

C: .select() would keep only PII columns, not remove them.

D: .remove() does not exist in Spark DataFrame API.

The .mode("overwrite") ensures that existing files at the path will be replaced.

partitionBy("country") optimizes queries by writing data into partitioned folders.

Correct syntax:

df.write.mode("overwrite").partitionBy("country").parquet("/data/output")

— Source: Spark SQL, DataFrames and Datasets Guide