Ten questions at a time, drawn from 130. Every answer is explained. Nothing is saved and no account is needed.
When a PySpark application runs, what is the primary responsibility of the Driver process?
Practice quiz for PySpark. Scores are not saved.
When a PySpark application runs, what is the primary responsibility of the Driver process?
Answer: Converting the logical plan into a physical execution plan and scheduling tasks. The Driver converts the high-level code into a DAG, optimizes it, and schedules tasks. It does not process actual data (Option 0), manage disk storage (Option 2), or handle direct DB reads (Option 3); those are the tasks of executors and the underlying I/O layers.
From lesson: Apache Spark Architecture — Driver, Executor, Cluster
What happens if you use a broadcast variable correctly in PySpark?
Answer: It caches the data in memory on every executor, reducing the need for shuffle operations. Broadcast variables allow executors to access a read-only variable locally without shuffling it across the network. Option 0 is wrong because the Driver does not 'copy to partitions', Option 2 is incorrect, and Option 3 describes a mechanism that does not exist.
From lesson: Apache Spark Architecture — Driver, Executor, Cluster
Why might a PySpark job fail with an OutOfMemoryError on the Driver even when executors have plenty of memory?
Answer: Because the user called collect() on a massive dataset that cannot fit into the Driver's memory. collect() gathers results from all executors into the Driver's memory. If the data is larger than the Driver's heap, it crashes. The other options describe behaviors that do not cause memory pressure on the Driver.
From lesson: Apache Spark Architecture — Driver, Executor, Cluster
In the context of the Spark execution model, what is a 'Stage'?
Answer: A sequence of tasks that can be performed without requiring a data shuffle. Stages are defined by shuffle boundaries. A stage is a set of tasks that can run in parallel without a shuffle. Hardware grouping (1), variables (2), and heartbeats (3) are unrelated to the definition of a Spark stage.
From lesson: Apache Spark Architecture — Driver, Executor, Cluster
How does the Cluster Manager interact with the Driver and Executors?
Answer: It allocates resources for the Driver and starts the Executors on worker nodes. The Cluster Manager (like YARN or K8s) handles resource requests. It starts the executors and provides the environment for the Driver. It does not run logic (0), replace the Driver (2), or handle serialization to SQL (3).
From lesson: Apache Spark Architecture — Driver, Executor, Cluster
Why is the 'Resilient' property in RDDs critical for PySpark performance?
Answer: It allows Spark to recompute lost partitions using lineage if a node fails.. Lineage tracks the operations that created the RDD, allowing recovery of lost data without replication. Option 0 is wrong because RDDs default to memory; Option 2 is wrong because lineage is about fault tolerance, not syntax; Option 3 is wrong because keeping everything in RAM is impossible for big data.
From lesson: RDDs — Resilient Distributed Datasets
What is the primary difference between transformations and actions in RDDs?
Answer: Transformations return a new RDD, while actions return a result to the driver or write to storage.. Transformations are lazy and define a DAG of operations, whereas actions trigger the execution. Option 1 is reversed. Option 2 is wrong because both run on executors. Option 3 is conceptually incorrect regarding RDD lifecycles.
From lesson: RDDs — Resilient Distributed Datasets
When should you use the .persist() method on an RDD?
Answer: When an RDD is reused multiple times in an iterative algorithm to avoid recomputing the lineage.. Persistence caches data in memory/disk to speed up re-access. Option 0 describes save operations, not persist. Option 1 describes an action. Option 3 describes clearing memory, which is the opposite of caching.
From lesson: RDDs — Resilient Distributed Datasets
What happens if you use a wide transformation like .reduceByKey() without enough partitions?
Answer: The operation will be forced to run on only one core, creating a bottleneck.. Wide transformations involve shuffling. Too few partitions cause high memory pressure on those specific nodes. Option 0 is false; serialization happens regardless of partition count. Option 1 is wrong because shuffle overhead increases. Option 3 is physically impossible.
From lesson: RDDs — Resilient Distributed Datasets
In a transformation like rdd.map(lambda x: x * 2), where does the actual multiplication occur?
Answer: On the worker nodes where the data partitions reside.. PySpark tasks are serialized and sent to executors (worker nodes) to process data locally. The driver only schedules the tasks, not the computation. Options 0, 2, and 3 misidentify the role of the driver and context.
From lesson: RDDs — Resilient Distributed Datasets
Which of the following scenarios best describes the primary purpose of lazy evaluation in PySpark?
Answer: To build an optimized execution plan based on the entire set of operations before executing. Lazy evaluation allows the catalyst optimizer to analyze the full DAG to improve efficiency. Option 0 is wrong because parallelism is handled by the scheduler, not laziness. Option 2 is wrong because caching is a separate step. Option 3 is wrong because laziness is independent of the library support.
From lesson: Transformations vs Actions
If you have a DataFrame and perform a .filter() followed by a .select(), when does PySpark start the actual computation?
Answer: Only when an action such as .collect() or .write() is invoked. PySpark only executes the query when an action is called. Options 0, 1, and 3 are incorrect because filter, select, and view registration are all transformations, which do not trigger execution.
From lesson: Transformations vs Actions
Why is it generally discouraged to call .count() inside a loop that processes millions of rows?
Answer: Because .count() triggers a full job execution, causing excessive overhead and latency. Every call to .count() is an action that triggers a Spark job. Doing this in a loop causes repeated, unnecessary scheduling overhead. Option 0 is wrong because it is an action, not a transformation. Option 2 is wrong because it doesn't loop the plan. Option 3 is wrong because it doesn't write to disk.
From lesson: Transformations vs Actions
Which set of operations contains ONLY transformations?
Answer: distinct, drop, join. Distinct, drop, and join are all transformations. Option 0 is wrong because reduce is an action. Option 1 is wrong because collect is an action. Option 3 is wrong because show, count, and saveAsTable are all actions.
From lesson: Transformations vs Actions
What is the result of applying a transformation to an existing DataFrame in PySpark?
Answer: It returns a new DataFrame containing the logical plan for the transformation. Spark DataFrames are immutable; transformations always produce a new DataFrame representing the updated plan. Option 0 is wrong because data is immutable. Option 2 is wrong because execution is lazy. Option 3 is wrong because transformations do not automatically convert types.
From lesson: Transformations vs Actions
If you have a chain of 100 transformations followed by a single count() action, how many times will the data be scanned?
Answer: Once, as the DAG is optimized into a single physical execution plan. Spark optimizes the entire DAG into a single physical plan before execution; therefore, the data is scanned only once. Option 0 is wrong because transformations are pipelined. Option 2 is incorrect because the scan happens regardless of partitioning. Option 3 is wrong because count() is an action, forcing execution.
From lesson: Lazy Evaluation and DAG
What is the primary benefit of the Spark Catalyst Optimizer in the context of DAGs?
Answer: It prunes unnecessary columns and pushes down filters before execution. Catalyst optimizes the logical plan by pushing down filters and selecting only required columns. Option 0 is wrong because caching is manual. Option 1 is wrong because Spark uses Tungsten, not Python-to-machine code translation. Option 3 is wrong because Spark is lazy.
From lesson: Lazy Evaluation and DAG
Which scenario best illustrates when to use the .persist() method?
Answer: When a specific DataFrame is used as a common source for multiple downstream actions. Persistence is used when a DataFrame is reused, avoiding re-computation of the DAG. Option 0 describes an action. Option 2 is incorrect because persistence happens in memory or disk, not external DBs. Option 3 is wrong as it doesn't help with memory usage during collection.
From lesson: Lazy Evaluation and DAG
Why does calling .explain() on a DataFrame not result in the processing of data?
Answer: Because .explain() only triggers the analysis of the logical plan. .explain() displays the logical and physical plan generated by the optimizer without running the actual tasks. Option 1 is wrong because it works on any plan. Option 2 is wrong because execution is distributed. Option 3 is wrong because it does not trigger a dry run.
From lesson: Lazy Evaluation and DAG
In a DAG, what is the significance of a 'Wide Dependency'?
Answer: It represents a transformation that triggers a shuffle of data across nodes. Wide dependencies (like groupBy or join) require a shuffle, which marks a boundary for stages in the DAG. Option 0 describes narrow dependencies. Option 2 is false as wide dependencies require communication. Option 3 is false as it is a architectural feature, not a failure warning.
From lesson: Lazy Evaluation and DAG
What is the primary role of the Spark Driver when running on a cluster?
Answer: To convert the logical plan into tasks and schedule them across executors. The driver acts as the brain, scheduling tasks and managing execution. Option 0 is wrong because executors perform transformations. Option 2 is wrong because the driver does not manage distributed disk storage. Option 3 is incorrect as the driver doesn't handle connection pooling for databases.
From lesson: Setting Up PySpark — Local and Cluster
Why is it necessary to configure PYSPARK_PYTHON in a cluster environment?
Answer: To ensure all worker nodes use the identical Python environment for task execution. Spark serializes Python objects; if versions differ, deserialization fails. Option 0 is wrong because Python doesn't compile Java. Option 1 describes SPARK_HOME. Option 3 is wrong because network speed is independent of the Python binary path.
From lesson: Setting Up PySpark — Local and Cluster
When deploying in 'local' mode versus 'cluster' mode, what is the main functional difference?
Answer: Local mode runs everything in one JVM, while cluster mode distributes processes across physical/virtual machines. Local mode is a simplified single-process setup, whereas cluster mode offloads tasks to distributed resources. Option 0 is wrong as serialization is consistent. Option 2 is false as both support all APIs. Option 3 is incorrect as both modes handle all PySpark APIs.
From lesson: Setting Up PySpark — Local and Cluster
What happens if the 'spark.executor.memoryOverhead' is not properly configured in a cluster?
Answer: The containers might be killed by the cluster manager due to out-of-memory errors. Memory overhead covers off-heap data. Without it, the cluster manager kills the process for exceeding limits. Option 0 is wrong because the driver handles scheduling, not worker memory. Option 1 is wrong because OOM errors crash, not speed up tasks. Option 3 is wrong because managers typically enforce, not extend, limits.
From lesson: Setting Up PySpark — Local and Cluster
Which file system path is most appropriate for a file intended to be read by all nodes in a cluster?
Answer: s3a://bucket-name/data/file.csv. Distributed file systems are accessible by all executors. Options 0, 1, and 3 refer to local file paths that would only exist on the driver node, causing errors when executors try to read them.
From lesson: Setting Up PySpark — Local and Cluster
What is the primary architectural difference in how PySpark and local data manipulation tools handle data?
Answer: PySpark operations are evaluated lazily, while local tools are evaluated eagerly. PySpark uses a query plan that is executed only when an action is triggered (lazy), whereas local tools execute code as soon as a command is called. Option 0 is wrong because PySpark is inherently distributed. Option 2 is wrong because PySpark spills to disk, and option 3 is wrong because PySpark DataFrames are immutable.
From lesson: PySpark DataFrames vs Pandas
Which of the following describes why performing a custom Python function using UDFs is generally slower than using PySpark built-in functions?
Answer: UDFs force the JVM to pass data back and forth to the Python process, breaking optimization. UDFs necessitate expensive serialization/deserialization between the JVM and Python. Built-in functions run directly within the JVM, allowing the Catalyst optimizer to see and improve the logic. Other options are incorrect descriptions of the underlying architecture.
From lesson: PySpark DataFrames vs Pandas
Why is it often discouraged to use the .collect() method in a production environment?
Answer: It forces all data to be aggregated into the memory of the single driver node. Collecting brings all distributed records to the driver. If the dataset size exceeds driver memory, it causes OOM errors. It does not cache data in the cluster, nor does it restart the cluster or ignore partitioning inherently.
From lesson: PySpark DataFrames vs Pandas
When joining a very small DataFrame with a very large DataFrame, what is the most performance-efficient approach in PySpark?
Answer: Using a broadcast join to send the small table to every executor. Broadcasting the small table avoids the 'shuffle' of the large table across the network. A shuffle join is slow for large datasets. Collecting to the driver will crash it, and increasing partitions unnecessarily adds overhead.
From lesson: PySpark DataFrames vs Pandas
What happens when you execute a filter transformation followed by a select transformation on a PySpark DataFrame?
Answer: A logical plan is built, which the optimizer then simplifies before execution. PySpark transformations build a lineage and a logical plan. The Catalyst optimizer merges these into an efficient physical plan before execution. Option 0 is wrong due to laziness. Option 1 is wrong because movement only happens on shuffles. Option 3 is wrong because the actual execution processes the data.
From lesson: PySpark DataFrames vs Pandas
When creating a DataFrame from a local Python list of rows, what is the primary architectural limitation?
Answer: The data is confined to the driver's memory. Option 1 is correct because local lists are processed by the driver and must fit in memory. Sorting (0) is not required for creation. RDDs (2) are not mandatory for DataFrame creation via createDataFrame. Types (3) are flexible.
From lesson: Creating DataFrames
Why is it recommended to define a schema programmatically using StructType instead of letting Spark infer it?
Answer: It avoids an unnecessary full-scan job. Option 2 is correct because schema inference requires Spark to scan the entire dataset first. SQL functions (0) work with inferred schemas too. Disk storage (1) is not related to schema definition. Renaming (3) is a separate operation.
From lesson: Creating DataFrames
Which of the following is the most efficient way to create a DataFrame from a large CSV file distributed across a cluster?
Answer: spark.read.csv(path, schema=my_schema). Option 0 leverages Spark's native readers to distribute the file reading process. (1) reads only locally on the driver. (2) and (3) are inefficient manual approaches that bypass optimized file formats.
From lesson: Creating DataFrames
You have an RDD of tuples and call .toDF(). What determines the column names if none are provided?
Answer: Automatic default names like _1, _2, etc.. Option 2 is correct: PySpark assigns default indexed names. Variable names (0), metadata (1), and data types (3) are not used to generate column names automatically in this process.
From lesson: Creating DataFrames
When creating a DataFrame via the 'fromPandas' method, what must be considered regarding memory usage?
Answer: Data is copied from the Python process to the JVM memory. Option 1 is correct: Moving data from Pandas (Python) to PySpark (JVM) involves a serialization and copy step. (0) is false, (2) is incorrect as it handles larger data if configured, and (3) is false as it utilizes executors.
From lesson: Creating DataFrames
When reading a Parquet dataset, why is it generally preferred over CSV for large-scale analytical workloads in PySpark?
Answer: Parquet is a columnar format that enables predicate pushdown and column pruning.. Parquet is columnar, which allows Spark to skip unnecessary columns and rows (pushdown/pruning), significantly reducing I/O. Options 1 and 2 are incorrect because Parquet is binary and columnar. Option 4 is false as Parquet enforces strict schema consistency.
From lesson: Reading Data Sources — Parquet, CSV, JSON, Delta
A team has a Delta table and needs to perform a time-travel query. What must be true for this to work?
Answer: The table must be accessed via the 'delta' format reader.. Time travel relies on the transaction log unique to the Delta format. Option 1 is wrong as Delta is a separate format. Option 3 is wrong because Delta manages its own files. Option 4 is incorrect because Delta is designed specifically for object storage/data lakes.
From lesson: Reading Data Sources — Parquet, CSV, JSON, Delta
When loading a multi-line JSON file into a DataFrame, which setting is essential to prevent parsing errors?
Answer: set multiline to true. PySpark's JSON reader defaults to one object per line. If the file contains objects spread across multiple lines, 'multiline' must be true. Option 2 is for CSV. Option 3 is irrelevant, and Option 4 would disable schema inference without providing a structure.
From lesson: Reading Data Sources — Parquet, CSV, JSON, Delta
Why does PySpark perform better when reading a single large file versus thousands of tiny files?
Answer: It reduces the overhead of listing files and metadata initialization.. Listing and tracking thousands of small files creates significant metadata overhead in the driver. Option 2 is false. Option 3 is false, and Option 4 is incorrect as Spark scales to millions of files.
From lesson: Reading Data Sources — Parquet, CSV, JSON, Delta
What is the primary benefit of explicitly providing a schema when reading CSV files in PySpark?
Answer: It avoids the costly 'schema inference' job that requires an extra pass over the data.. Schema inference requires reading the data twice (once to guess types, once to load). Providing the schema saves time. Option 1 is false. Option 3 is wrong because schemas don't 'fix' data. Option 4 is unrelated to providing a schema.
From lesson: Reading Data Sources — Parquet, CSV, JSON, Delta
When you want to retain only specific columns from a large DataFrame to optimize performance, which method is most appropriate?
Answer: select(). select() creates a new DataFrame with only the specified columns, which allows Spark to prune unnecessary data from the source. drop() is for removing columns, filter() selects rows, and withColumn() adds or replaces columns.
From lesson: Selecting, Filtering, and Aggregating
What is the primary difference between filter() and where() in PySpark?
Answer: There is no difference; they are aliases for each other.. In PySpark, where() and filter() are identical; where() was introduced to make the API feel familiar to SQL users. Both accept the same arguments and perform the same logic.
From lesson: Selecting, Filtering, and Aggregating
You need to count the number of unique items per category. Which sequence of operations is correct?
Answer: df.groupBy('category').agg(countDistinct('item')). groupBy() followed by agg(countDistinct(...)) correctly scopes the unique count to each group. Other options either count the wrong things or fail to associate the count with the category.
From lesson: Selecting, Filtering, and Aggregating
Which of the following describes how Spark handles a filter expression on a large dataset?
Answer: It pushes the predicate down to the data source to minimize data reading.. Spark's Catalyst Optimizer uses predicate pushdown to filter data as close to the source as possible, reducing I/O. The other options are inefficient or describe non-Spark processes.
From lesson: Selecting, Filtering, and Aggregating
If you perform a select() statement followed by a groupBy(), what happens to the columns not included in the groupBy() or the aggregate function?
Answer: The operation will throw an AnalysisException.. In relational algebra, non-aggregated columns must be part of the grouping key. PySpark enforces this to ensure the result is deterministic. The other options describe behaviors that would lead to ambiguous output.
From lesson: Selecting, Filtering, and Aggregating
What is the primary requirement to execute a SQL query on a PySpark DataFrame?
Answer: The DataFrame must be registered as a temporary view.. To run SQL, Spark needs to know the name of the table in the catalog. Option 2 is correct because .createOrReplaceTempView() adds the DataFrame to the catalog. Options 1, 3, and 4 are not required for basic SQL execution.
From lesson: SparkSQL — Running SQL on DataFrames
If you register a DataFrame as a temporary view named 'users', how does that view scope behave?
Answer: It is tied to the specific SparkSession used to create it.. Temporary views are session-scoped; they exist only for the duration of the current SparkSession. Option 1 is wrong because global views require 'GlobalTempView', and options 2 and 4 are incorrect behaviors for temp views.
From lesson: SparkSQL — Running SQL on DataFrames
What is the result of running 'spark.sql()' on a DataFrame that has NOT been registered as a view?
Answer: It throws an AnalysisException because the table is not found.. SparkSQL looks up table names in the internal catalog; if a name hasn't been registered, the catalyst analyzer fails, resulting in an AnalysisException. The other options describe incorrect or non-existent behaviors.
From lesson: SparkSQL — Running SQL on DataFrames
Which of the following describes the performance relationship between DataFrame API calls and SQL queries?
Answer: Both are equally performant because they are compiled into the same logical/physical plan.. Both approaches use the Catalyst Optimizer and Tungsten engine, meaning they are ultimately translated into the same underlying optimized plan. Options 1, 3, and 4 incorrectly imply performance differences or missing optimizations.
From lesson: SparkSQL — Running SQL on DataFrames
Why would a developer choose to use SparkSQL instead of the pure DataFrame API for a complex transformation?
Answer: To easily write complex window functions or nested aggregations using familiar SQL syntax.. SQL is often more readable for complex relational operations like multi-level joins or window functions. Option 2 is correct. Options 3 and 4 are false, and option 1 is incorrect as SparkSQL still enforces type safety.
From lesson: SparkSQL — Running SQL on DataFrames
When writing a DataFrame to a Parquet file, why is 'overwrite' mode generally safer for production pipelines than simply deleting the output directory?
Answer: It prevents partial data reads if the job fails mid-write.. Option 1 is correct because Spark writes to a temporary directory first and performs an atomic rename/commit, ensuring partial data is not visible. Options 2, 3, and 4 are incorrect because they relate to memory management, compression algorithms, and job lifecycle, none of which are the primary benefit of the atomic overwrite process.
From lesson: Writing Output — Parquet, Delta, JDBC
What is the primary architectural difference when writing to Delta Lake compared to standard Parquet files in PySpark?
Answer: Delta Lake maintains a transaction log that tracks file modifications.. Option 2 is correct because the transaction log provides ACID guarantees. Option 1 is false because Delta remains columnar; Option 3 is false as it is file-based; Option 4 is false as Delta preserves schema and data types.
From lesson: Writing Output — Parquet, Delta, JDBC
You are writing to a JDBC database using Spark. If the job fails, why might you find 'orphaned' rows in the destination table?
Answer: Because JDBC writes do not support transactional rollbacks by default.. Option 0 is correct because JDBC writes typically commit per partition or per batch, and if the job crashes, those committed batches are not rolled back automatically. Options 1, 2, and 3 describe incorrect behavior or unrelated configuration issues.
From lesson: Writing Output — Parquet, Delta, JDBC
Why is it recommended to perform a 'repartition' operation before saving a very large DataFrame to multiple Parquet files?
Answer: To manage the number of output files and control file size for better query performance.. Option 2 is correct because controlling the number of partitions directly dictates the number of files written, preventing the 'small file problem.' Option 0 is inefficient, Option 1 causes bottlenecks, and Option 3 defeats the purpose of distributed processing.
From lesson: Writing Output — Parquet, Delta, JDBC
When performing an 'upsert' (merge) operation in Delta Lake, what must exist to ensure the operation can successfully identify matches?
Answer: A common key or set of keys between the source and target DataFrames.. Option 1 is correct because the merge command requires a join condition (the 'on' clause) to map rows. Options 0, 2, and 3 are irrelevant to the logic of comparing source and target data sets for updates or inserts.
From lesson: Writing Output — Parquet, Delta, JDBC
When Spark performs a Broadcast Hash Join, where does the 'small' table reside during the operation?
Answer: In the memory of each executor task. The small table is broadcasted to every executor's memory so tasks can perform lookups locally. Option 0 is wrong because disk access would negate performance gains; option 2 is wrong because the Driver acts only as the coordinator; option 3 is wrong as HDFS is for persistence, not join runtime.
From lesson: Joins in Spark — Broadcast, Shuffle
What is the primary consequence of a Sort-Merge Join when the join keys are not perfectly aligned across partitions?
Answer: A full shuffle of both datasets is triggered. Sort-Merge joins require records with the same keys to be on the same partition, necessitating a full data shuffle. Option 0 is impossible; option 2 is incorrect because joins are mandatory; option 3 is wrong as the Driver does not manage executor memory via configuration changes during execution.
From lesson: Joins in Spark — Broadcast, Shuffle
How does salting help optimize a join operation in PySpark?
Answer: It breaks up hotspots caused by skewed join keys. Salting distributes data with frequent keys across multiple partitions to prevent one task from doing all the work. Option 0 refers to deduplication; option 1 is achieved via hints, not salting; option 3 is false as salting actually increases data size slightly.
From lesson: Joins in Spark — Broadcast, Shuffle
If you perform a join and notice one task takes significantly longer than others, what is the most likely cause?
Answer: Data skew in the join key. A 'straggler' task is almost always caused by data skew where one partition is much larger than others. Option 1 leads to OOM, not long tasks; option 2 would make all tasks fast; option 3 affects complexity but not individual task duration relative to others.
From lesson: Joins in Spark — Broadcast, Shuffle
Why does Spark prefer a Broadcast join over a Sort-Merge join for small-to-large table joins?
Answer: It avoids expensive network shuffles. Broadcast joins move the small table once, avoiding the heavy network I/O of shuffling large datasets. Option 0 is wrong because broadcasting consumes more heap memory; option 2 is incorrect as joins require equal keys; option 3 describes a different operation entirely.
From lesson: Joins in Spark — Broadcast, Shuffle
When using the rank() function, how does it handle ties in the data compared to row_number()?
Answer: rank() assigns the same value to ties and skips subsequent values, while row_number() assigns unique sequential values.. rank() creates gaps in numbering after a tie (e.g., 1, 2, 2, 4), whereas row_number() forces a unique sequential integer (1, 2, 3, 4). The other options incorrectly swap these behaviors or imply invalid non-deterministic behavior.
From lesson: Window Functions in PySpark
What happens if you apply a window function using window.partitionBy('colA') without an .orderBy('colB') clause?
Answer: The function will treat the window as having no specific order, which is valid for aggregations like sum() but invalid for ranking functions.. For aggregation functions like sum() or avg(), an order is not strictly necessary as the window is the entire partition. However, ranking functions (like rank or lead) require an order to be meaningful. The code won't crash on aggregations, making the third option the correct technical nuance.
From lesson: Window Functions in PySpark
In a rolling average calculation, what is the effect of using 'ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING'?
Answer: It computes the average of the previous row, the current row, and the next row.. The frame definition explicitly includes one row before, the current row, and one row after. Option 1 describes UNBOUNDED PRECEDING, option 3 ignores the specified frame, and option 4 describes an unbounded window.
From lesson: Window Functions in PySpark
Why is the performance of window functions often a bottleneck in PySpark?
Answer: They require a global sort or a massive shuffle to bring all rows of a partition onto the same worker.. Window functions require all data for a specific partition to reside on the same executor, necessitating a network-heavy shuffle. The other options are misconceptions: PySpark handles window logic in the JVM, and optimization is still possible.
From lesson: Window Functions in PySpark
Which of the following is true regarding the lead() and lag() functions in PySpark?
Answer: They allow you to access data from other rows based on a physical offset without self-joining the dataframe.. lead() and lag() are specifically designed to look forward or backward within a window, avoiding expensive self-joins. Option 1 is correct; others describe false constraints or incorrect behavior.
From lesson: Window Functions in PySpark
When comparing a PySpark UDF to a native PySpark function (e.g., when.otherwise()), why is the native function preferred?
Answer: Native functions allow the Catalyst Optimizer to optimize the execution plan more effectively.. Native functions are optimized by the Catalyst engine (predicate pushdown, etc.). UDFs are 'black boxes' that force data serialization between JVM and Python. The other options are incorrect because native functions are highly performant and the rest are factually wrong.
From lesson: UDFs — User Defined Functions
What happens if you omit the return type parameter when creating a UDF in PySpark?
Answer: The UDF defaults to StringType, potentially causing cast errors.. PySpark defaults to StringType if not specified. This causes errors if the code returns an Integer or Boolean. Inference is not supported for UDFs. Performance is not affected by schema declaration, only correctness.
From lesson: UDFs — User Defined Functions
Which of the following scenarios is the most appropriate use case for a PySpark UDF?
Answer: Applying a complex proprietary algorithm that involves external libraries not supported by PySpark SQL.. UDFs are necessary for complex logic impossible in standard SQL/functions. The other options (upper, timestamp diff, filtering) are trivial and performant using built-in PySpark functions.
From lesson: UDFs — User Defined Functions
Why does using a UDF often result in high serialization overhead?
Answer: Because PySpark must convert data from the JVM heap to Python memory objects and back.. The communication between the JVM (where data resides) and the Python process is the bottleneck. The other options describe false architectural bottlenecks (UDFs run on executors, not the driver).
From lesson: UDFs — User Defined Functions
If your UDF logic involves external libraries, where must those libraries be available for the UDF to execute correctly?
Answer: On all worker nodes in the cluster.. Since UDFs run on worker nodes where the data is partitioned, the code and dependencies must exist there. Driver-side or local notebook availability is insufficient.
From lesson: UDFs — User Defined Functions
If you perform a left join between two DataFrames on a join key that contains nulls on the left side, what happens?
Answer: The rows with null keys are matched to nulls on the right. In PySpark, nulls in join keys do not match other nulls, and they are preserved in a left join. Option 0 is wrong because inner joins drop null keys, not left joins. Option 2 is incorrect as this is standard behavior. Option 3 is false as no automatic casting occurs.
From lesson: Handling Null Values
Which approach is most efficient to replace null values in a specific numeric column with the column mean?
Answer: Collect the mean to the driver and use .fillna(). Collecting the mean to the driver is the standard, performant pattern. UDFs (Option 0) are slower due to serialization. Joining (Option 2) is overkill. Option 3 is syntactically incorrect for filling nulls.
From lesson: Handling Null Values
How does count(col('x')) behave when column 'x' contains null values?
Answer: It excludes null values from the count. The count(col) function specifically counts non-null values. Option 0 and 3 are wrong because nulls are ignored. Option 1 is incorrect as Spark handles nulls without crashing.
From lesson: Handling Null Values
When using .when(col('x').isNull(), 0).otherwise(col('x')), what is the primary impact on the resulting DataFrame schema?
Answer: The column remains nullable regardless of the replacement. PySpark schema inference typically keeps columns as nullable unless explicitly cast otherwise; replacing values does not change the schema metadata. Options 0, 2, and 3 misinterpret Spark's rigid schema handling.
From lesson: Handling Null Values
Which method should you use to remove rows where 'col_a' is null OR 'col_b' is null?
Answer: .dropna(subset=['col_a', 'col_b'], how='any'). Using 'any' in dropna removes the row if any of the specified columns are null. Option 1 is wrong because 'all' only removes rows if both are null. Option 2 is invalid syntax. Option 3 removes the columns entirely, not the rows.
From lesson: Handling Null Values
Which approach is most efficient for combining the columns 'first_name' and 'last_name' with a space in between?
Answer: concat_ws(' ', first_name, last_name). concat_ws is designed specifically for joining multiple columns with a separator and handles null values gracefully, unlike concat which returns null if any input is null. The '+' operator performs addition, and append does not exist as a column method.
From lesson: String and Date Functions
You have a string column 'date_str' formatted as 'dd/MM/yyyy'. How do you correctly convert this to a Spark DateType?
Answer: to_date(date_str, 'dd/MM/yyyy'). to_date expects a format string when the input does not match 'yyyy-MM-dd'. Option 1 and 2 fail because they assume the default format. Option 3 creates a timestamp, not a date.
From lesson: String and Date Functions
What is the result of applying substring('PySpark', 0, 2) in PySpark?
Answer: Empty string. PySpark string functions are 1-indexed. Starting at 0 in a 1-indexed system is treated as an invalid or empty range, resulting in an empty string rather than 'Py', which would be indexed as starting at 1.
From lesson: String and Date Functions
When working with dates, why would you prefer date_format() over using to_date()?
Answer: date_format() converts the date to a string based on a specific pattern. date_format() is used to transform a date/timestamp column into a string representation for display or reporting, whereas to_date() is used for parsing strings into date objects. The others are incorrect functions of the tools.
From lesson: String and Date Functions
How do you extract the year from a column named 'transaction_date' of type DateType?
Answer: year(transaction_date). year() is the dedicated PySpark SQL function to extract the year integer from a date or timestamp. Extract/date_part are syntax variations in other SQL dialects not used in PySpark's DataFrame API, and the method call syntax in option 3 is not supported for Column objects.
From lesson: String and Date Functions
When should you prefer coalesce() over repartition()?
Answer: When you are reducing the partition count and want to avoid a full data shuffle. Coalesce is designed to reduce partitions by merging existing ones, which avoids a full shuffle. Option 1 is incorrect because coalesce cannot increase partitions. Option 2 is incorrect because coalesce performs a minimal shuffle. Option 4 is incorrect because repartition is better suited for balancing skewed data.
From lesson: Partitioning and Shuffling
What is the primary performance drawback of repartitioning a DataFrame based on a column with high skew?
Answer: One task will take significantly longer to finish, bottlenecking the entire job. Skewed keys lead to uneven distribution where one partition receives significantly more records, creating a 'straggler' task. Option 1 refers to metadata overhead, which is secondary. Option 3 is irrelevant to shuffle mechanics. Option 4 is false; hash partitioning does not inherently lose 'data order' in a way that creates performance bottlenecks.
From lesson: Partitioning and Shuffling
How does the default partitioning in PySpark behave when reading from a file system like S3 or HDFS?
Answer: It attempts to map the number of partitions to the number of input splits determined by the file format and size. Spark's initial parallelism is driven by the file split information. Option 1 is incorrect because while 200 is a default for shuffles, initial input often respects splits. Option 3 is incorrect as executors, not the master, process data. Option 4 would defeat the purpose of distributed processing.
From lesson: Partitioning and Shuffling
If you perform a join on two DataFrames with different partition counts, what happens internally in PySpark?
Answer: The smaller DataFrame is automatically partitioned to match the larger one via a shuffle. To perform an equi-join, Spark must ensure records with matching keys reside on the same partition, which triggers a shuffle to align them. Option 1 is correct. Option 2 is incorrect; Spark handles this automatically. Option 3 and 4 are incorrect as they ignore the requirements of join alignment.
From lesson: Partitioning and Shuffling
Why is it often recommended to persist a DataFrame before performing multiple operations that involve heavy shuffling?
Answer: It avoids re-computing the entire lineage and re-shuffling the data for every subsequent action. Caching or persisting keeps the result of the shuffle in memory/disk, preventing the need to re-run the shuffle if the data is reused. Option 1 is incorrect as persistence doesn't change partitioning. Option 2 is incorrect as the driver always manages executors. Option 4 is a benefit of storage formats, not persistence logic.
From lesson: Partitioning and Shuffling
Which scenario provides the most significant performance benefit when using .cache()?
Answer: An iterative algorithm like PageRank that repeatedly accesses the same DataFrame.. Caching is beneficial for iterative algorithms where the same data is scanned multiple times. Option 1 is wrong because writing once doesn't require caching. Option 2 is wrong because the driver memory isn't the primary concern for distributed tasks. Option 4 is wrong because the overhead of caching outweighs the cost of reading a small file.
From lesson: Caching and Persistence
When you call .persist(StorageLevel.DISK_ONLY), what happens to the execution flow?
Answer: The data is marked for persistence, and will be written to disk only when an action is performed.. Spark operations are lazy; persistence only takes effect upon the next action. Option 1 is wrong because it isn't immediate. Option 3 is wrong because DISK_ONLY refers to local executor disks. Option 4 is wrong because Spark does not halt for persistence.
From lesson: Caching and Persistence
What is the primary consequence of failing to call .unpersist() on a very large, cached DataFrame?
Answer: The application may experience OutOfMemory errors in subsequent stages due to reduced available heap space.. Cached data consumes memory. If not cleared, it crowds out space needed for new tasks. Option 1 is incorrect as memory issues manifest as OOMs, not stack errors. Option 2 is a partial truth regarding eviction but doesn't explain the performance impact. Option 4 is wrong because cached data stays on executors.
From lesson: Caching and Persistence
If a cached RDD or DataFrame partition is lost, how does Spark handle it?
Answer: Spark uses the lineage information to recompute only the lost partition.. Spark's fault tolerance is built on recomputing only the lost parts of the graph. Option 1 is wrong because Spark is designed for resilience. Option 3 is wrong because user intervention is not required. Option 4 is wrong because data integrity must be maintained.
From lesson: Caching and Persistence
Why is it generally discouraged to cache the result of a very simple, fast transformation?
Answer: The serialization and deserialization overhead can be more expensive than the original computation.. Caching involves overhead (serialization, object management). For fast operations, it is cheaper to recompute. Option 1 is the correct rationale. Option 2 is false. Option 3 is false. Option 4 is false because caching happens on executors.
From lesson: Caching and Persistence
When should you use a Broadcast Variable instead of simply referencing a large variable inside a UDF or map function?
Answer: When the variable is large and read-only, to prevent re-serializing it for every task. Broadcasting prevents the overhead of sending the data with every task. Option 0 describes accumulators, option 1 is unnecessary for small data, and option 3 is not the primary purpose of broadcasting.
From lesson: Broadcast Variables and Accumulators
What is the primary behavior of an Accumulator in a PySpark job?
Answer: It is a write-only variable from the worker's perspective that aggregates results back to the driver. Accumulators are write-only for tasks and read-only for the driver. Option 0 describes Broadcasts, option 2 is incorrect because reading an accumulator in a task is unreliable, and option 3 describes broadcast-like or shuffle behavior.
From lesson: Broadcast Variables and Accumulators
Why is it dangerous to read the value of an Accumulator within a Spark transformation?
Answer: Because Spark's lazy evaluation and re-execution of tasks can lead to inaccurate counts. If a task fails, Spark may re-run it; if you read an accumulator inside a transformation, you may double-count values. Option 0 is false, option 2 is not a synchronization issue, and option 3 is false.
From lesson: Broadcast Variables and Accumulators
Which of the following scenarios is ideal for a Broadcast Variable?
Answer: Sending a 500MB lookup table to all executors to perform a map-side join. Broadcast variables excel at distributing large read-only lookup data. Option 0 is for accumulators, option 2 is better handled by log4j, and option 3 is invalid as broadcasts are immutable.
From lesson: Broadcast Variables and Accumulators
If you update an Accumulator inside a map function, but then execute a transformation instead of an action, what happens to the accumulator?
Answer: The accumulator will be updated only when an action triggers the job execution. Spark transformations are lazy; the code inside the map function only runs when an action is called. Option 0 is wrong due to laziness, option 2 is incorrect because it can be used in transformations, and option 3 is factually wrong.
From lesson: Broadcast Variables and Accumulators
When analyzing the physical plan output from explain(), what does an 'Exchange' node typically indicate?
Answer: A shuffle operation is occurring to redistribute data across partitions. An 'Exchange' indicates a shuffle, which happens during wide transformations like joins or repartitioning. Option 0 is wrong because that's usually a Scan, 2 is wrong because the plan is just a recipe, and 3 is wrong because caching is typically represented by 'InMemoryTableScan'.
From lesson: Query Execution Plan — explain()
Which of the following describes the correct way to read a PySpark execution plan?
Answer: From the bottom upwards, starting from the data sources. Execution plans represent a tree where data originates at the leaves (bottom) and is transformed as it moves up to the root (top). Options 0 and 1 are incorrect as they ignore the structure of the DAG. Option 3 is incorrect as other operators like Join and Filter are crucial.
From lesson: Query Execution Plan — explain()
Why might you prefer explain('cost') over a standard explain() call?
Answer: To see the logical and physical plans along with the optimizer's statistics. The 'cost' mode provides insights into the optimizer's decision-making process by showing table and column statistics. Option 0 is wrong because cost refers to query optimization math, not billing. Option 2 is irrelevant to query plans, and 3 is wrong because explain() never executes the query.
From lesson: Query Execution Plan — explain()
If you perform a join and notice a 'BroadcastHashJoin' in the explain() output, what is Spark likely doing?
Answer: It is sending the smaller table to all executors to avoid a massive shuffle. BroadcastHashJoin is an optimization where the smaller table is broadcast to all nodes. Option 0 describes a ShuffleHashJoin, 2 is false as the optimizer chooses this only if it fits, and 3 describes a SortMergeJoin.
From lesson: Query Execution Plan — explain()
What is the primary difference between 'Logical Plan' and 'Physical Plan' in the output of explain(True)?
Answer: The logical plan explains what data to get, while the physical plan explains how to execute it on the cluster. The logical plan outlines the intent of the operations, and the physical plan shows the specific strategies (like broadcast vs shuffle) to perform them. Option 1 is wrong because neither plan 'runs' the code. Option 2 is wrong as both apply to DataFrames, and 3 is wrong because both are automatically optimized.
From lesson: Query Execution Plan — explain()
What is the primary consequence of setting 'spark.sql.shuffle.partitions' to a value significantly higher than the actual data volume?
Answer: It leads to too many small tasks, increasing scheduling overhead and task startup time.. Increasing partitions too much creates 'small task' overhead, where the cost of launching and managing thousands of tiny tasks outweighs the execution time. Option 0 is wrong because overhead slows it down; 2 is wrong because memory is static; 3 is wrong because partitioning doesn't fix skew.
From lesson: Tuning — executor memory, cores, shuffle partitions
When configuring an executor with 5 cores, why is it recommended to cap it at this specific number?
Answer: It optimizes HDFS throughput and keeps garbage collection manageable within the JVM.. Five cores strike a balance between high parallelism and the HDFS client's inability to handle many concurrent writes effectively, while preventing excessive GC pauses. Others are incorrect because Spark can handle more cores, but with negative performance side effects.
From lesson: Tuning — executor memory, cores, shuffle partitions
If your application experiences 'ExecutorLostFailure' due to memory issues, what is the best first step to debug the configuration?
Answer: Check the executor logs to see if it is a container memory limit breach caused by overhead.. ExecutorLostFailure often points to the cluster manager killing the process due to exceeding memory limits (often related to off-heap/overhead). Checking logs confirms the cause. Other options don't directly address the OOM container termination.
From lesson: Tuning — executor memory, cores, shuffle partitions
How does Adaptive Query Execution (AQE) interact with shuffle partitions?
Answer: AQE dynamically coalesces small shuffle partitions into larger ones after the shuffle map stage.. AQE optimizes shuffle by combining small partitions into fewer, larger ones to reduce overhead. It doesn't force a set number (0), disable them (2), or force a single partition (3).
From lesson: Tuning — executor memory, cores, shuffle partitions
Why might increasing executor memory without changing 'spark.memory.fraction' fail to improve performance for a join-heavy job?
Answer: Because the default memory fraction might still limit the execution memory available for joins, causing spilling.. If 'spark.memory.fraction' restricts the execution memory portion, increasing total heap doesn't expand the workspace for joins enough to stop disk spilling. Other options are conceptually incorrect regarding how Spark manages memory.
From lesson: Tuning — executor memory, cores, shuffle partitions
Which component is primarily responsible for ensuring end-to-end fault tolerance in Structured Streaming?
Answer: Checkpointing and Write-Ahead Logs. Checkpointing and WALs track offset progress, allowing the system to restart from exactly where it left off. Optimizer is for query speed, HDFS is for data storage, and GC is for memory management.
From lesson: Spark Structured Streaming Overview
What happens if a streaming query is restarted without a checkpoint location?
Answer: It starts processing the stream from the beginning of the source. Without a checkpoint, the engine has no persistent state of processed offsets, so it defaults to the beginning of the stream. It does not resume (requires checkpoint), it does not error, and it does not hang.
From lesson: Spark Structured Streaming Overview
In a streaming join between two dataframes, why is the watermark essential?
Answer: To limit the amount of state stored for late-arriving data. Watermarking allows the engine to drop old state that is no longer needed for joining future incoming records, preventing memory overflow. It does not affect network speed, schema, or partitioning directly.
From lesson: Spark Structured Streaming Overview
What is the key difference between 'Complete' and 'Update' output modes?
Answer: Complete mode outputs the entire state table; Update mode outputs only changed rows. Complete mode rewrites the entire aggregate result set per trigger; Update mode only outputs rows that changed since the last trigger. The others are incorrect definitions regarding storage or syntax.
From lesson: Spark Structured Streaming Overview
Why is 'Trigger.Once' typically used in production streaming environments?
Answer: To execute a single batch and then shut down for cost-efficiency. Trigger.Once executes all pending data in one batch then terminates, saving compute costs. It does not run every millisecond, it does not restrict threading, and it does not remove the need for watermarking.
From lesson: Spark Structured Streaming Overview
Which configuration option ensures that a PySpark streaming job processes all data currently in the Kafka topic and then terminates?
Answer: trigger(availableNow=True). The 'availableNow=True' trigger is the recommended way to process all existing data and shut down. 'once=True' is deprecated. Ending offsets and max offsets control data volume per batch, not the job lifecycle.
From lesson: Reading from Kafka
Why is it important to use 'checkpointLocation' when reading from Kafka in a production streaming job?
Answer: To store offsets and metadata, allowing the stream to resume exactly where it left off after a failure. Checkpointing is essential for fault tolerance; it records the progress of the stream. It does not speed up reads, compress data, or shift storage duties to Kafka itself.
From lesson: Reading from Kafka
If you are reading from Kafka and the data is stored in JSON format within the value column, what is the best practice for schema enforcement?
Answer: Use 'from_json' with a predefined StructType schema. Defining a schema with 'from_json' provides better performance and type safety than inferring it every time. Manual parsing is error-prone, and 'StringType' prevents further analytical operations on the data.
From lesson: Reading from Kafka
When configuring a Kafka read stream, what does the 'minPartitions' option control?
Answer: The minimum number of Spark tasks created for the read operation. In PySpark, 'minPartitions' influences the number of Spark partitions created, thereby affecting the number of parallel tasks. It has no impact on Kafka broker settings, replication, or memory caching.
From lesson: Reading from Kafka
What happens if you increase 'maxOffsetsPerTrigger' in your Kafka read stream?
Answer: You increase the potential throughput per micro-batch but may increase latency. Setting 'maxOffsetsPerTrigger' allows you to process more data in each batch, which can improve throughput, but the batch takes longer to process, increasing latency. It does not change Kafka partitions or filter data by time.
From lesson: Reading from Kafka
If your watermark is set to 10 minutes and you receive an event with a timestamp of 10:00, what happens to an event with a timestamp of 09:45 that arrives immediately after?
Answer: It is dropped because it is older than the 10-minute threshold. The correct answer is that the event is dropped. Since the watermark is calculated as max(event_time) - 10 minutes, once 10:00 is seen, the watermark is 09:50. Events older than 09:50 are dropped. The other options are incorrect because the watermark acts as a hard cut-off for state cleanup, it does not automatically extend, nor does it retain late data indefinitely.
From lesson: Watermarking and Late Data
Which of the following is a direct consequence of specifying a watermark in a windowed aggregation?
Answer: The state store is purged of old window keys that are no longer needed. Specifying a watermark allows the engine to know when it can safely remove intermediate state for window keys that fall behind the watermark. Option 1 is false because it relates to processing, not state. Option 3 is false as it remains a streaming process. Option 4 is false as deduplication requires an explicit `dropDuplicates` call.
From lesson: Watermarking and Late Data
How does PySpark determine the watermark value when consuming from multiple partitions in a Kafka topic?
Answer: It uses the maximum event time across all partitions to calculate the watermark. PySpark tracks the maximum event time observed globally across the input data. Option 1 is wrong because watermarks are global for the query. Option 2 would result in a watermark that never advances if one partition is slow. Option 4 is incorrect because watermarks are applied within every micro-batch.
From lesson: Watermarking and Late Data
You want to deduplicate events based on a unique ID within a 1-hour window. Which approach is most efficient?
Answer: Use `withWatermark` followed by `dropDuplicates`. Using `withWatermark` followed by `dropDuplicates` ensures the state store doesn't grow infinitely by removing IDs older than the watermark. Option 1 leads to unbounded state memory usage. Option 2 is manually complex and inefficient. Option 4 counts items but does not provide a deduplicated stream of unique events.
From lesson: Watermarking and Late Data
What is the primary purpose of setting the watermark threshold?
Answer: To balance state memory usage against data completeness for late events. Watermarks create a tradeoff between memory consumption (keeping state) and data completeness (how late events are allowed to be). Option 1 is wrong as it is about data age, not processing time. Option 3 is wrong as it doesn't affect source throughput. Option 4 is wrong as watermarks are unrelated to task retries.
From lesson: Watermarking and Late Data
What is the primary architectural reason why PySpark RDDs are considered 'lazy'?
Answer: They build a Directed Acyclic Graph (DAG) and only execute tasks when an action is called. Lazy evaluation allows the driver to optimize the entire execution plan before running it. Option 0 is false as storage is handled by cache policies; option 2 is false as Spark automates memory management; option 3 is false as RDDs prioritize memory.
From lesson: PySpark Interview Questions
If you perform a 'map' operation followed by a 'filter' operation on a DataFrame, how does the Catalyst optimizer typically handle this?
Answer: It pushes the 'filter' down to the source level to minimize data processed. Catalyst performs predicate pushdown to reduce data volume as early as possible. Option 0 and 1 represent unoptimized execution, and option 3 defeats the purpose of the high-level API.
From lesson: PySpark Interview Questions
When is it most appropriate to use a coalesce() operation instead of repartition()?
Answer: When you are reducing the number of partitions and want to avoid a full shuffle. coalesce() avoids a full shuffle by merging existing partitions, making it efficient for downsizing. repartition() triggers a full shuffle, which is necessary for increasing partitions or data balancing, but unnecessarily expensive for shrinking.
From lesson: PySpark Interview Questions
Which of the following scenarios best justifies the use of broadcast joins?
Answer: Joining a very large fact table with a small reference table that fits in memory. Broadcast joins send the small table to every executor, avoiding a shuffle of the large table. The other options involve scenarios where broadcast would fail (OOM) or be less efficient than a sort-merge join.
From lesson: PySpark Interview Questions
What happens to the data inside a partition when a 'transformation' is applied?
Answer: A new RDD/DataFrame is created, reflecting the transformation as part of the lineage. Transformations are immutable and build the lineage graph for fault tolerance. Option 0 is impossible due to immutability; option 2 is inefficient for large scale; option 3 is incorrect as executors perform the work, not the driver.
From lesson: PySpark Interview Questions
Which of the following best describes the fundamental difference in execution between PySpark DataFrames and local data structures?
Answer: PySpark operations are lazily evaluated and optimized by the Catalyst optimizer.. PySpark builds a logical and physical plan to optimize execution before running any tasks. Option 0 is false because PySpark is not line-by-line. Option 2 is false because PySpark manages distributed cluster memory, not just local disk. Option 3 is false as PySpark manages memory automatically.
From lesson: PySpark vs Pandas — Key Differences
You have a large DataFrame and need to verify if your transformation logic works as expected. Which approach is most efficient?
Answer: Use .take(n) or .show(n) to view a limited sample.. Using .take() or .show() limits the data transferred to the driver. .collect() and .toPandas() are dangerous on large data as they force data into the driver's memory. .printSchema() only shows types, not the transformation logic results.
From lesson: PySpark vs Pandas — Key Differences
Why is it generally discouraged to use Python UDFs (User Defined Functions) when built-in functions are available?
Answer: Built-in functions allow the optimizer to push down predicates and avoid data serialization between the JVM and Python process.. Built-in functions allow Catalyst to optimize queries at the JVM level. Python UDFs require data to be serialized and sent to a Python worker process, causing high overhead. Options 0, 1, and 3 are either false or less critical than the serialization bottleneck.
From lesson: PySpark vs Pandas — Key Differences
When working with skewed data, what is the most effective strategy to ensure even distribution across partitions?
Answer: Using a salt column to break apart keys that cause heavy partitions.. Salting adds a random prefix to keys, forcing even distribution across tasks. Increasing workers doesn't help if one worker has 90% of the data. Reducing partitions makes the problem worse. Caching stores data but doesn't change the underlying partition distribution.
From lesson: PySpark vs Pandas — Key Differences
What is the primary role of the 'Driver' node in a PySpark application?
Answer: It coordinates the execution by converting code into tasks and distributing them to executors.. The Driver manages the SparkContext and orchestrates the distributed computation. The other options are incorrect because calculation happens on executors, the driver doesn't store the bulk data, and there is no arbitrary 2 GB limit.
From lesson: PySpark vs Pandas — Key Differences
