| Stage Id ▾ | Pool Name | Description | Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write | Failure Reason |
|---|---|---|---|---|---|---|---|---|---|---|
| 4 | default | PersistFrame parquet at PersistJob.java:35 org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:847) com.adb.hdh.spark.job.impl.PersistJob.exec(PersistJob.java:35) com.adb.hdh.spark.jobs.SparkJobRunnerService.runJob(SparkJobRunnerService.java:77) com.adb.hdh.spark.jobs.AbstractJobThread.run(AbstractJobThread.java:47) java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) java.base/java.util.concurrent.FutureTask.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) | 2025/11/03 07:16:18 | 0.6 s |
0/7
(6 failed)
(1 killed: Stage cancelled)
| 3.5 MiB | Job aborted due to stage failure: Task 4 in stage 4.0 failed 1 times, most recent failure: Lost task 4.0 in stage 4.0 (TID 8, 926ab5ff208a, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(UDFRegistration$$Lambda$4511/0x00007fb93bd71040: (string, struct<COL_78E288B2_7753_442C_B16A_E52F4C57C5CF:string,COL_4584417F_CE48_42EF_8E60_88E860F0CAB7:string,COL_BD1FBF2C_C36D_491D_AFB1_B342D040D62C:string,COL_9195E6E1_98EE_4D9B_B713_80A990F20E9F:string,COL_26ABE631_4CA5_4C98_9C8A_D974C21257D9:string,COL_AFB4434B_0574_4367_A593_DC9A5CD29825:string,COL_3EFD2E5B_CBA3_4894_AC4A_D1DE405D734D:string>) => double)Job aborted due to stage failure: Task 4 in stage 4.0 failed 1 times, most recent failure: Lost task 4.0 in stage 4.0 (TID 8, 926ab5ff208a, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(UDFRegistration$$Lambda$4511/0x00007fb93bd71040: (string, struct<COL_78E288B2_7753_442C_B16A_E52F4C57C5CF:string,COL_4584417F_CE48_42EF_8E60_88E860F0CAB7:string,COL_BD1FBF2C_C36D_491D_AFB1_B342D040D62C:string,COL_9195E6E1_98EE_4D9B_B713_80A990F20E9F:string,COL_26ABE631_4CA5_4C98_9C8A_D974C21257D9:string,COL_AFB4434B_0574_4367_A593_DC9A5CD29825:string,COL_3EFD2E5B_CBA3_4894_AC4A_D1DE405D734D:string>) => double) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.GreaterThanOrEqual_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown Source) at org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoinExec.$anonfun$boundCondition$1(BroadcastNestedLoopJoinExec.scala:85) at org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoinExec.$anonfun$boundCondition$1$adapted(BroadcastNestedLoopJoinExec.scala:85) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:513) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:488) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:265) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:210) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:463) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:466) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap') at com.adb.hdh.spark.job.udf.ScalaToJavaConverter.getNumberList(ScalaToJavaConverter.java:68) at com.adb.hdh.spark.job.udf.DecUdf.lambda$create$f33ef0bb$1(DecUdf.java:24) at org.apache.spark.sql.UDFRegistration.$anonfun$register$285(UDFRegistration.scala:762) ... 20 more Driver stacktrace: |