Databricks ML Data Scientist Databricks-Machine-Learning-Associate Question # 16 Topic 2 Discussion

Among the options listed, calculating the true median for imputing missing feature values is the least efficient to distribute. This is because the true median requires knowledge of the entire data distribution, which can be computationally expensive in a distributed environment. Unlike mean or mode, finding the median requires sorting the data or maintaining a full distribution, which is more intensive and often requires shuffling the data across partitions.

References

Challenges in parallel processing and distributed computing for data aggregation like median calculation:https://www.apache.org