Associate-Developer-Apache-Spark - Latest Training Dumps

NO.64 The code block displayed below contains an error. The code block should return a new DataFrame that only contains rows from DataFrame transactionsDf in which the value in column predError is at least 5. Find the error.
Code block:
transactionsDf.where(“col(predError) >= 5”)

The argument to the where method should be “predError >= 5”.

Instead of where(), filter() should be used.

The expression returns the original DataFrame transactionsDf and not a new DataFrame. To avoid this, the code block should be transactionsDf.toNewDataFrame().where(“col(predError) >= 5”).

The argument to the where method cannot be a string.

Instead of >=, the SQL operator GEQ should be used.

NO.65 Which of the following code blocks produces the following output, given DataFrame transactionsDf?
Output:
1.root
2. |– transactionId: integer (nullable = true)
3. |– predError: integer (nullable = true)
4. |– value: integer (nullable = true)
5. |– storeId: integer (nullable = true)
6. |– productId: integer (nullable = true)
7. |– f: integer (nullable = true)
DataFrame transactionsDf:
1.+————-+———+—–+——-+———+—-+
2.|transactionId|predError|value|storeId|productId| f|
3.+————-+———+—–+——-+———+—-+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.+————-+———+—–+——-+———+—-+

transactionsDf.schema.print()

transactionsDf.rdd.printSchema()

transactionsDf.rdd.formatSchema()

transactionsDf.printSchema()

print(transactionsDf.schema)

NO.66 Which of the following code blocks returns a DataFrame where columns predError and productId are removed from DataFrame transactionsDf?
Sample of DataFrame transactionsDf:
1.+————-+———+—–+——-+———+—-+
2.|transactionId|predError|value|storeId|productId|f |
3.+————-+———+—–+——-+———+—-+
4.|1 |3 |4 |25 |1 |null|
5.|2 |6 |7 |2 |2 |null|
6.|3 |3 |null |25 |3 |null|
7.+————-+———+—–+——-+———+—-+

transactionsDf.withColumnRemoved(“predError”, “productId”)

transactionsDf.drop([“predError”, “productId”, “associateId”])

transactionsDf.drop(“predError”, “productId”, “associateId”)

transactionsDf.dropColumns(“predError”, “productId”, “associateId”)

transactionsDf.drop(col(“predError”, “productId”))

NO.67 Which of the following describes a shuffle?

A shuffle is a process that is executed during a broadcast hash join.

A shuffle is a process that compares data across executors.

A shuffle is a process that compares data across partitions.

A shuffle is a Spark operation that results from DataFrame.coalesce().

A shuffle is a process that allocates partitions to executors.

1. select
2. count
3. col(“itemNameElements”)
4. >3

1. filter
2. count
3. itemNameElements
4. >=3

1. select
2. count
3. “itemNameElements”
4. >3

1. filter
2. size
3. “itemNameElements”
4. >=3
(Correct)

1. select
2. size
3. “itemNameElements”
4. >3

NO.69 Which of the following describes characteristics of the Spark UI?

Via the Spark UI, workloads can be manually distributed across executors.

Via the Spark UI, stage execution speed can be modified.

The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.

There is a place in the Spark UI that shows the property spark.executor.memory.

Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.

NO.70 Which of the following describes Spark’s Adaptive Query Execution?

Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.

Adaptive Query Execution is enabled in Spark by default.

Adaptive Query Execution reoptimizes queries at execution points.

Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.

Adaptive Query Execution applies to all kinds of queries.

NO.71 Which of the following statements about garbage collection in Spark is incorrect?

Garbage collection information can be accessed in the Spark UI’s stage detail view.

Optimizing garbage collection performance in Spark may limit caching ability.

Manually persisting RDDs in Spark prevents them from being garbage collected.

In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.

Serialized caching is a strategy to increase the performance of garbage collection.

Explanation
Manually persisting RDDs in Spark prevents them from being garbage collected.
This statement is incorrect, and thus the correct answer to the question. Spark’s garbage collector will remove even persisted objects, albeit in an “LRU” fashion. LRU stands for least recently used.
So, during a garbage collection run, the objects that were used the longest time ago will be garbage collected first.
See the linked StackOverflow post below for more information.
Serialized caching is a strategy to increase the performance of garbage collection.
This statement is correct. The more Java objects Spark needs to collect during garbage collection, the longer it takes. Storing a collection of many Java objects, such as a DataFrame with a complex schema, through serialization as a single byte array thus increases performance. This means that garbage collection takes less time on a serialized DataFrame than an unserialized DataFrame.
Optimizing garbage collection performance in Spark may limit caching ability.
This statement is correct. A full garbage collection run slows down a Spark application. When taking about
“tuning” garbage collection, we mean reducing the amount or duration of these slowdowns.
A full garbage collection run is triggered when the Old generation of the Java heap space is almost full. (If you are unfamiliar with this concept, check out the link to the Garbage Collection Tuning docs below.) Thus, one measure to avoid triggering a garbage collection run is to prevent the Old generation share of the heap space to be almost full.
To achieve this, one may decrease its size. Objects with sizes greater than the Old generation space will then be discarded instead of cached (stored) in the space and helping it to be “almost full”.
This will decrease the number of full garbage collection runs, increasing overall performance.
Inevitably, however, objects will need to be recomputed when they are needed. So, this mechanism only works when a Spark application needs to reuse cached data as little as possible.
Garbage collection information can be accessed in the Spark UI’s stage detail view.
This statement is correct. The task table in the Spark UI’s stage detail view has a “GC Time” column, indicating the garbage collection time needed per task.
In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.
This statement is correct. The G1 garbage collector, also known as garbage first garbage collector, is an alternative to the default Parallel garbage collector.
While the default Parallel garbage collector divides the heap into a few static regions, the G1 garbage collector divides the heap into many small regions that are created dynamically. The G1 garbage collector has certain advantages over the Parallel garbage collector which improve performance particularly for Spark workloads that require high throughput and low latency.
The G1 garbage collector is not enabled by default, and you need to explicitly pass an argument to Spark to enable it. For more information about the two garbage collectors, check out the Databricks article linked below.

NO.72 Which of the following code blocks uses a schema fileSchema to read a parquet file at location filePath into a DataFrame?

spark.read.schema(fileSchema).format(“parquet”).load(filePath)

spark.read.schema(“fileSchema”).format(“parquet”).load(filePath)

spark.read().schema(fileSchema).parquet(filePath)

spark.read().schema(fileSchema).format(parquet).load(filePath)

spark.read.schema(fileSchema).open(filePath)

NO.73 The code block displayed below contains an error. The code block should save DataFrame transactionsDf at path path as a parquet file, appending to any existing parquet file. Find the error.
Code block:

transactionsDf.format(“parquet”).option(“mode”, “append”).save(path)

The code block is missing a reference to the DataFrameWriter.

save() is evaluated lazily and needs to be followed by an action.

The mode option should be omitted so that the command uses the default mode.

The code block is missing a bucketBy command that takes care of partitions.

Given that the DataFrame should be saved as parquet file, path is being passed to the wrong method.

NO.74 Which of the following describes a difference between Spark’s cluster and client execution modes?

In cluster mode, the cluster manager resides on a worker node, while it resides on an edge node in client mode.

In cluster mode, executor processes run on worker nodes, while they run on gateway nodes in client mode.

In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.

In cluster mode, a gateway machine hosts the driver, while it is co-located with the executor in client mode.

In cluster mode, the Spark driver is not co-located with the cluster manager, while it is co-located in client mode.

NO.75 Which of the following code blocks displays various aggregated statistics of all columns in DataFrame transactionsDf, including the standard deviation and minimum of values in each column?

transactionsDf.summary()

transactionsDf.agg(“count”, “mean”, “stddev”, “25%”, “50%”, “75%”, “min”)

transactionsDf.summary(“count”, “mean”, “stddev”, “25%”, “50%”, “75%”, “max”).show()

transactionsDf.agg(“count”, “mean”, “stddev”, “25%”, “50%”, “75%”, “min”).show()

transactionsDf.summary().show()

NO.76 The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, “itemId”, how=”broadcast”)

The syntax is wrong, how= should be removed from the code block.

The join method should be replaced by the broadcast method.

Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.

The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.

broadcast is not a valid join type.

NO.77 Which of the following code blocks stores a part of the data in DataFrame itemsDf on executors?

itemsDf.cache().count()

itemsDf.cache(eager=True)

cache(itemsDf)

itemsDf.cache().filter()

itemsDf.rdd.storeCopy()

NO.78 Which of the following code blocks shows the structure of a DataFrame in a tree-like way, containing both column names and types?

1.print(itemsDf.columns)
2.print(itemsDf.types)

itemsDf.printSchema()

spark.schema(itemsDf)

itemsDf.rdd.printSchema()

itemsDf.print.schema()

NO.79 The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__)

1. filter
2. “transactionId”, “predError”, “value”, “f”

1. select
2. “transactionId, predError, value, f”

1. select
2. [“transactionId”, “predError”, “value”, “f”]

1. where
2. col(“transactionId”), col(“predError”), col(“value”), col(“f”)

1. select
2. col([“transactionId”, “predError”, “value”, “f”])

NO.80 The code block shown below should return a copy of DataFrame transactionsDf with an added column cos.
This column should have the values in column value converted to degrees and having the cosine of those converted values taken, rounded to two decimals. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(__2__, round(__3__(__4__(__5__)),2))

1. withColumn
2. col(“cos”)
3. cos
4. degrees
5. transactionsDf.value

1. withColumnRenamed
2. “cos”
3. cos
4. degrees
5. “transactionsDf.value”

1. withColumn
2. “cos”
3. cos
4. degrees
5. transactionsDf.value

1. withColumn
2. col(“cos”)
3. cos
4. degrees
5. col(“value”)
E
. 1. withColumn
2. “cos”
3. degrees
4. cos
5. col(“value”)

itemsDf.select(explode(“attributes”).alias(“attributes_exploded”)).filter(attributes_exploded.contains(“i”))

itemsDf.explode(attributes).alias(“attributes_exploded”).filter(col(“attributes_exploded”).contains(“i”))

itemsDf.select(explode(“attributes”)).filter(“attributes_exploded”.contains(“i”))

itemsDf.select(explode(“attributes”).alias(“attributes_exploded”)).filter(col(“attributes_exploded”).contain

itemsDf.select(col(“attributes”).explode().alias(“attributes_exploded”)).filter(col(“attributes_exploded”).co

NO.82 Which of the following code blocks returns a new DataFrame in which column attributes of DataFrame itemsDf is renamed to feature0 and column supplier to feature1?

itemsDf.withColumnRenamed(attributes, feature0).withColumnRenamed(supplier, feature1)

1.itemsDf.withColumnRenamed(“attributes”, “feature0”)
2.itemsDf.withColumnRenamed(“supplier”, “feature1”)

itemsDf.withColumnRenamed(col(“attributes”), col(“feature0”), col(“supplier”), col(“feature1”))

itemsDf.withColumnRenamed(“attributes”, “feature0”).withColumnRenamed(“supplier”, “feature1”)

itemsDf.withColumn(“attributes”, “feature0”).withColumn(“supplier”, “feature1”)

Category Associate-Developer-Apache-Spark

Reliable Databricks Certification Associate-Developer-Apache-Spark Dumps PDF Apr 28, 2022 Recently Updated Questions [Q64-Q82]