| Stage Id ▾ | Pool Name | Description | Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write | Failure Reason |
|---|---|---|---|---|---|---|---|---|---|---|
| 8 | default | DataFrameCommonJob_dataExecSummary count at ColumnStatJob.java:47 org.apache.spark.sql.Dataset.count(Dataset.scala:2980) com.adb.hdh.spark.job.impl.ColumnStatJob.stats(ColumnStatJob.java:47) com.adb.hdh.spark.job.impl.ExecDataJob.exec(ExecDataJob.java:35) com.adb.hdh.spark.jobs.SparkJobRunnerService.runJob(SparkJobRunnerService.java:142) com.adb.hdh.spark.jobs.AbstractJobThread.run(AbstractJobThread.java:47) java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) java.base/java.util.concurrent.FutureTask.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) | 2025/11/03 07:16:29 | 0.2 s |
0/7
(7 failed)
| 2.7 MiB | Job aborted due to stage failure: Task 6 in stage 8.0 failed 1 times, most recent failure: Lost task 6.0 in stage 8.0 (TID 20, 926ab5ff208a, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(UDFRegistration$$Lambda$4511/0x00007fb93bd71040: (string, struct<COL_78E288B2_7753_442C_B16A_E52F4C57C5CF:string,COL_4584417F_CE48_42EF_8E60_88E860F0CAB7:string,COL_BD1FBF2C_C36D_491D_AFB1_B342D040D62C:string,COL_9195E6E1_98EE_4D9B_B713_80A990F20E9F:string,COL_26ABE631_4CA5_4C98_9C8A_D974C21257D9:string,COL_AFB4434B_0574_4367_A593_DC9A5CD29825:string,COL_3EFD2E5B_CBA3_4894_AC4A_D1DE405D734D:string>) => double)Job aborted due to stage failure: Task 6 in stage 8.0 failed 1 times, most recent failure: Lost task 6.0 in stage 8.0 (TID 20, 926ab5ff208a, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(UDFRegistration$$Lambda$4511/0x00007fb93bd71040: (string, struct<COL_78E288B2_7753_442C_B16A_E52F4C57C5CF:string,COL_4584417F_CE48_42EF_8E60_88E860F0CAB7:string,COL_BD1FBF2C_C36D_491D_AFB1_B342D040D62C:string,COL_9195E6E1_98EE_4D9B_B713_80A990F20E9F:string,COL_26ABE631_4CA5_4C98_9C8A_D974C21257D9:string,COL_AFB4434B_0574_4367_A593_DC9A5CD29825:string,COL_3EFD2E5B_CBA3_4894_AC4A_D1DE405D734D:string>) => double) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.GreaterThanOrEqual_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown Source) at org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoinExec.$anonfun$boundCondition$1(BroadcastNestedLoopJoinExec.scala:85) at org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoinExec.$anonfun$boundCondition$1$adapted(BroadcastNestedLoopJoinExec.scala:85) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:513) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:488) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithoutKey_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:132) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:463) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:466) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap') at com.adb.hdh.spark.job.udf.ScalaToJavaConverter.getNumberList(ScalaToJavaConverter.java:68) at com.adb.hdh.spark.job.udf.DecUdf.lambda$create$f33ef0bb$1(DecUdf.java:24) at org.apache.spark.sql.UDFRegistration.$anonfun$register$285(UDFRegistration.scala:762) ... 23 more Driver stacktrace: |
