nebula-exchange_spark_2.4本地编译报错:org/bouncycastle/bcprov-jdk15on和Unsupported class file major version 59

想在本地编译一个exchange-3.0.0的包,spark2.4,依赖环境都满足:jdk1.8,scala2.11,但是很遗憾报错尝试未解决。

image
—满足依赖的环境。


[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project nebula-exchange_spark_2.4: Error creating shaded jar: Problem shading JAR /Users/jiaenqiu/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.69/bcprov-jdk15on-1.69.jar entry META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$Ed25519.class: java.lang.IllegalArgumentException: Unsupported class file major version 59 → [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :nebula-exchange_spark_2.4

maven啥版本?

apache-maven-3.8.5,jdk1.8.0_321

问题描述补充:

提交到sparkOnYarn上后报错,异常信息如下

Job aborted due to stage failure: 
Task 0 in stage 3.0 failed 4 times, most recent failure: 
Lost task 0.3 in stage 3.0 (TID 16, 10.203.85.73, executor 6): 
java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()
Lcom/google/common/base/Stopwatch;

解决思路:

  • 1.NoSuchMethodError一般就是依赖jar冲突导致的,上述异常是guava依赖包
  • 2.查看guava源码,发现14.0.1版本中确实没有com.google.common.base.Stopwatch.createStarted(),而16.0.1中有,所以把所有14.x的版本都排除掉
  • 3.提交任务还是报错,那就是yarn依赖的guava依赖包版本比较低,jar冲突。由于我这边环境无法私自改动,所以只能修改本地guava包路径,利用maven-shade-plugin插件,这样编译时就可以加载自定义包名就可以忽略yarn上低版本的guava包了
  • 4.另外所有依赖中的org.bouncycastle.bcprov-jdk15on也得在xml中排除掉,否则会影响shade打包,引起如博客主文描述的异常

解决方案:

<exclusions>
                <exclusion>
                    <groupId>org.bouncycastle</groupId>
                    <artifactId>bcprov-jdk15on</artifactId>
                </exclusion>
            </exclusions>
<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <!-- 源包名 -->
                                    <pattern>com.google.common</pattern>
                                    <!-- 目的包名 -->
                                    <shadedPattern>my_guava.common</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>org.bouncycastle</pattern>
                                    <shadedPattern>bcprov-jdk15on</shadedPattern>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
  • 如有异议,请回复
1 个赞

看下JAVA_HOME,那个报错看上去还是和jdk有关啊。
mvn编译的命令是啥?要不按它的建议 -e 看下错误堆栈?
我是怎么编都不报错 :upside_down_face:

1 个赞

➜echo $JAVA_HOME
➜/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home
➜mvn clean package -Dmaven.test.skip=true -Dgpg.skip -Dmaven.javadoc.skip=true -pl nebula-exchange_spark_2.4 -am -Pscala-2.11 -Pspark-2.4

  • 不知道为啥这个版本编译就报错,可能和我使用shade插件的原因,必须排除bcprov-jdk15on才行

浙ICP备20010487号