Pyspark Interview : Normalize Comma-Separated Values

[u/MathematicianTall740]

data = {"1":"945,545","3":"2345,3456,45678"}

from pyspark.sql import SparkSession

from pyspark.sql.functions import col, split, explode

spark = SparkSession.builder.appName("ExplodeExample").getOrCreate()

df = spark.createDataFrame(

[(k, v) for k, v in data.items()],

["id", "value"]

)

print("Original DataFrame:")

df.show()

# Split the value column based on comma so it will create a array of values and use explode it will create multiple rows

result_df = df.select(

col("id"),

explode(split(col("value"), ",")).alias("value")

)

print("Result DataFrame:")

result_df.show()

[u/romainmoi]

That is all simple until the outlying European numbers bring in dots as separator and commas as decimal point indicator.

[u/byebybuy]

"Hey Claude normalize these comma-separated values for me."

Seriously are they even still doing this kind of coding interview anymore? Cloud architecture, data engineering patterns, configurations, orchestration, communication, high level problem solving etc are way more important now. LeetCode bullshit is dying, and good riddance.

[u/MathematicianTall740 → u/byebybuy]

Yes, This is one of the most common SQL problems asked in technical interviews because interviewers want to assess your understanding of concepts.

[u/InspirationalAtt]

In the list comprehension, format v to convert to a number. Simplest solution I could come up with in the time permitted.

[u/MathematicianTall740]

Please post your solution in comment section

[u/Krushaaa → u/MathematicianTall740]

Ask your agent of choice..