Pass the Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with ValidTests

When reading Parquet files, Spark infers a unified schema automatically only if all files share identical structures.

If files have different but compatible schemas, you must enable schema merging by setting the option mergeSchema=True.

Correct syntax:

df = spark.read.option("mergeSchema", True).parquet("/data/parquet/")

This option ensures Spark merges all discovered fields across Parquet files into one unified DataFrame schema.

Why the other options are incorrect:

A: Loads files but ignores extra columns — uses only the first file’s schema.

C: inferSchema applies to CSV/JSON, not Parquet.

D: mergeAllCols is not a valid Spark option.

Option B uses Spark’s built-in SQL function length(), which is efficient and avoids the overhead of a Python UDF:

from pyspark.sql.functions import length, col

df.select(length(col("stringColumn")).alias("length"))

Explanation of other options:

Option A is incorrect syntax; spark.udf is not called this way.

Option C registers a UDF but doesn’t apply it in the DataFrame transformation.

Option D is syntactically valid but uses a Python UDF which is less efficient than built-in functions.

Final Answer: B

Structured Streaming supports three output modes:

Append: Writes only new rows since the last trigger.

Update: Writes only updated rows.

Complete: Writes the entire result table after every trigger execution.

For aggregations like groupBy().count(), only complete mode outputs the entire table each time.

Example:

aggDF.writeStream \

outputMode("complete") \

format("console") \

start()

Why the other options are incorrect:

A: “AGGREGATE” is not a valid output mode.

C: “REPLACE” does not exist.

D: “APPEND” writes only new rows, not the full table.

By default, the spark.read.text() method reads a text file one line per record. This means that each line in a text file becomes one row in the resulting DataFrame.

To read each file as a single record, Apache Spark provides the option wholetext, which, when set to True, causes Spark to treat the entire file contents as one single string per row.

Correct usage:

df = spark.read.option("wholetext", True).text(txt_path)

This way, each record in the DataFrame will contain the full content of one file instead of one line per record.

To also include the file path, the function input_file_name() can be used to create an additional column that stores the complete path of the file being read:

from pyspark.sql.functions import input_file_name

df = spark.read.option("wholetext", True).text(txt_path) \

withColumn("file_path", input_file_name())

This approach satisfies both requirements from the question:

Each record holds the entire contents of a file.

Each record also contains the file path from which the text was read.

Why the other options are incorrect:

B or D (lineSep) – The lineSep option only defines the delimiter between lines. It does not combine the entire file content into a single record.

C (wholetext=False) – This is the default behavior, which still reads one record per line rather than per file.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

PySpark API Reference: DataFrameReader.text — describes the wholetext option.

PySpark Functions: input_file_name() — adds a column with the source file path.

Databricks Certified Associate Developer for Apache Spark Exam Guide (June 2025): Section “Using Spark DataFrame APIs” — covers reading files and handling DataFrames.

The correct way to aggregate information (e.g., max value) from distributed workers back to the driver is using RDD actions such as reduce() or aggregate().

From the documentation:

“To perform global aggregations on distributed data, actions like reduce() are commonly used to collect summaries such as min/max/avg.”

Accumulators (Option B) do not support max operations directly and are not intended for such analytics.

Broadcast (Option C) is used to send data to workers, not collect from them.

Spark UI (Option D) is a monitoring tool — not an analytics collection interface.

Final Answer: A

In Spark’s client mode, the driver runs on the local machine that submitted the job. If that machine is resource-constrained (e.g., low memory), performance degrades.

From the Spark documentation:

"In cluster mode, the driver runs inside the cluster, benefiting from cluster resources and scalability."

Option A is incorrect — executors do not help the driver directly.

Option B might help short-term but does not scale.

Option C is correct — switching to cluster mode moves the driver to the cluster.

Option D (local mode) is for development/testing, not production.

Final Answer: C

The .coalesce(numPartitions) function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.

From the official Spark documentation:

“coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions.”

However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.

Hence, df.coalesce(20) will still return a DataFrame with 10 partitions.

Adaptive Query Execution (AQE) in Spark 3.x automatically optimizes query plans at runtime based on the actual data characteristics observed during job execution.

Key features of AQE include:

Dynamic switching of join strategies: Changes between sort-merge join and broadcast join based on actual shuffle sizes.

Coalescing shuffle partitions: Reduces small tasks and improves parallelism efficiency.

Handling skew joins: Dynamically splits large partitions to avoid data skew.

Thus, the most accurate answer describing AQE’s function is “dynamically switching join strategies.”

Why the other options are incorrect:

B: Table statistics are collected manually or by the metastore, not by AQE.

C: AQE benefits multi-stage jobs involving shuffles, not single-stage jobs.

D: Delta file optimization is handled by Databricks utilities, not AQE.

The split() function in PySpark splits strings into an array based on a given delimiter.

Then, .getItem(index) extracts a specific element from the array.

Correct usage:

from pyspark.sql.functions import split, col

customerDF = customerDF \

withColumn("username", split(col("email"), "@").getItem(0)) \

withColumn("domain", split(col("email"), "@").getItem(1))

This creates two new columns derived from the email field:

"username" → text before @

"domain" → text after @

Why the other options are incorrect:

B: regexp_replace only replaces text; does not split into multiple columns.

C: .select() cannot alias multiple derived columns like this.

D: Column objects are not native Python strings; cannot use standard .split().

To compare a column value with a Python literal constant in a DataFrame expression, use F.lit() to convert it into a Spark literal.

Correct refactor:

from pyspark.sql import functions as F

min_price = 110.50

result_df = prices_df.filter(F.col("price") > F.lit(min_price)).agg(F.count("*"))

This avoids type mismatches and ensures Spark executes the filter expression on the cluster.

Why the other options are incorrect:

B: where() syntax is valid, but F.lit("price") is incorrect — wraps string literal, not a column.

C: withColumn adds a column, not needed for this aggregation.

D: Comparison logic reversed.