maven quality tool plugins by DaanHoogland · Pull Request #3956 · apache/cloudstack

DaanHoogland · 2020-03-10T10:44:34Z

Description

owasp dependency checker and other tools during the site build phase to be able to add them to apache jenkins. inluding updates and cleanups.

Some version discrepencies arose so version upgrades are included and mvn dependency:tree | grep <this or that> | sort -u gave some that need handling:

com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
com.fasterxml.jackson.core:jackson-core:jar:2.9.10:compile

com.google.code.findbugs:jsr305:jar:1.3.9:compile
com.google.code.findbugs:jsr305:jar:3.0.2:compile

com.google.http-client:google-http-client:jar:1.17.0-rc:compile
com.google.http-client:google-http-client:jar:1.20.0:compile

com.unboundid:unboundid-ldapsdk:jar:3.2.0:compile
com.unboundid:unboundid-ldapsdk:jar:4.0.14:test

commons-collections:commons-collections:jar:3.2.1:compile
commons-collections:commons-collections:jar:3.2.2:compile

commons-logging:commons-logging:jar:1.1.1:compile
commons-logging:commons-logging:jar:1.2:compile

javax.annotation:javax.annotation-api:jar:1.3.2:compile
# javax.annotation:javax.annotation-api:jar:1.3:compile

joda-time:joda-time:jar:2.10.5:compile
# joda-time:joda-time:jar:2.2:compile
# joda-time:joda-time:jar:2.8.1:compile
# joda-time:joda-time:jar:2.9.3:compile

org.slf4j:slf4j-api:jar:1.7.29:compile
org.slf4j:slf4j-log4j12:jar:1.7.29:compile

xerces:xercesImpl:jar:2.12.0:test
# xerces:xercesImpl:jar:2.8.0:compile
xml-apis:xml-apis-ext:jar:1.3.04:compile

# xml-apis:xml-apis:jar:1.0.b2:compile,test
# xml-apis:xml-apis:jar:1.4.01:test
xml-apis:xml-apis:jar:2.0.2:test,compile ???

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

This reverts commit 2d63ed5. (PR #2446) fails when network causes CVE DB not being able to be downloaded

DaanHoogland · 2020-03-13T13:38:21Z

@rhtyd @weizhouapache @GabrielBrascher finally apache builds gave a positive again on this maven update. can you approve?
cc @svenvogel @PaulAngus @AnyBody

DaanHoogland · 2020-03-13T16:43:30Z

ping @weizhouapache @kiwiflyer @GabrielBrascher @rhtyd @nathanejohnson @andrijapanic-dont-use-this-one @svenvogel @wido this is ready for review.

nvazquez · 2020-03-13T18:55:48Z

@DaanHoogland I got this error when building (with update dependecies flag)

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:24 min
[INFO] Finished at: 2020-03-13T14:11:21-03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project cloud-plugin-network-f5: Could not resolve dependencies for project org.apache.cloudstack:cloud-plugin-network-f5:jar:4.14.0.0-SNAPSHOT: Could not find artifact commons-discovery:commons-discovery:jar:1.0-dev in central (https://repo.maven.apache.org/maven2) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :cloud-plugin-network-f5

DaanHoogland · 2020-03-14T17:54:00Z

@nvazquez I had been using -Dnonoss which has not been in use for ever and a day. You are right the dependency on f5 doesn't work.
testing further with mvn clean install -Dnoredist -P developer,systemvm, but it needs to work with mvn site -Dnoredist as well, will push if i get both to work.

nvazquez · 2020-03-15T00:16:21Z

@DaanHoogland I checked that the version 1.0-dev for the dependency commons-discovery is not present in the artifactory, so I changed the version to 0.5 (he last available in the repo: https://mvnrepository.com/artifact/commons-discovery/commons-discovery/0.5).

After changing that, I'm hitting a different error:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16:42 min
[INFO] Finished at: 2020-03-14T20:48:46-03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project cloud-plugin-outofbandmanagement-driver-nested-cloudstack: Could not resolve dependencies for project org.apache.cloudstack:cloud-plugin-outofbandmanagement-driver-nested-cloudstack:jar:4.14.0.0-SNAPSHOT: Failed to collect dependencies at org.apache.cloudstack:cloud-server:jar:clover:4.14.0.0-SNAPSHOT -> org.apache.cloudstack:cloud-framework-jobs:jar:4.14.0.0-SNAPSHOT -> org.apache.cloudstack:cloud-framework-events:jar:4.14.0.0-SNAPSHOT -> com.google.guava:guava:jar:28.2-jre -> org.checkerframework:checker-qual:jar:2.10.0: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap') -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :cloud-plugin-outofbandmanagement-driver-nested-cloudstack

BTW: I'm compiling with:

mvn clean install -Dnoredist -P developer,systemvm -DskipTests -U

DaanHoogland · 2020-03-16T07:49:37Z

very frustrating, I'm hitting a different one:

[ERROR] Failed to execute goal on project cloud-vmware-base: Could not resolve dependencies for project org.apache.cloudstack:cloud-vmware-base:jar:4.14.0.0-SNAPSHOT: Failure to find com.sun.xml.ws:jaxws-ri:pom:2.6 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

nvazquez · 2020-03-16T11:38:16Z

@DaanHoogland can you try adding -U to force updating the dependencies?

DaanHoogland · 2020-03-16T12:32:02Z

mvn clean install -Dnoredist -P developer,systemvm -U tried, it doesn't complete with above error. Unfortunately having a compiled system is more important than having good analysis output so no go for this, for now.

DaanHoogland · 2020-03-23T09:44:09Z

@svenvogel why would you put a label security on this PR?

DaanHoogland · 2020-03-24T10:15:07Z

ok, builds now on

fedora 31 and
@nvazquez' ubuntu (what's the version) and
macos 10.15.3 (19D76) provided the sed hack is added (see 'sed -i' need postfix on mac #3312 )
jenkins job

@rhtyd @weizhouapache @GabrielBrascher can you approve?

yadvr · 2020-03-24T14:23:44Z

@blueorangutan package

blueorangutan · 2020-03-24T14:24:12Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

GabrielBrascher

LGTM based on code review and build success.

Environment:
Apache Maven 3.3.9
Java version: 11.0.5 (java-11-openjdk-amd64)
OS: Ubuntu 16.04, version: "4.4.0-173-generic"

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 32:40 min
[INFO] Finished at: 2020-03-24T11:20:58-03:00
[INFO] Final Memory: 202M/694M
[INFO] ------------------------------------------------------------------------

blueorangutan · 2020-03-24T14:50:02Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1082

blueorangutan · 2020-03-25T19:49:46Z

Trillian test result (tid-1302)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28513 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3956-t1302-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	0.34	test_kubernetes_supported_versions.py

no default build phase for dep-check

da66e17

DaanHoogland requested review from GabrielBrascher, weizhouapache and yadvr March 10, 2020 10:44

DaanHoogland added 3 commits March 10, 2020 14:04

replace findbugs with spotbugs

37654d0

for target site checkstyle needs explicit version

cdb7555

cleanup

60b3dba

DaanHoogland changed the title ~~no default build phase for dep-check~~ maven quality tool plugins Mar 10, 2020

DaanHoogland referenced this pull request Mar 10, 2020

Revert "CLOUDSTACK-10271 maven plugin for owasp dependency chec… (#3953)

016644d

This reverts commit 2d63ed5. (PR #2446) fails when network causes CVE DB not being able to be downloaded

DaanHoogland added 2 commits March 10, 2020 15:16

new clover

2c8143f

remove reports in error

7a8b79b

DaanHoogland closed this Mar 11, 2020

DaanHoogland reopened this Mar 11, 2020

DaanHoogland closed this Mar 11, 2020

DaanHoogland reopened this Mar 11, 2020

DaanHoogland added 4 commits March 11, 2020 12:57

jacoco and clover

cea7829

but it worked on my laptop??

94da290

version upgrades

68d6f7d

more dependencies managed

880beec

DaanHoogland closed this Mar 13, 2020

DaanHoogland reopened this Mar 13, 2020

google http client and jackson

2ce46c7

DaanHoogland closed this Mar 13, 2020

DaanHoogland reopened this Mar 13, 2020

DaanHoogland closed this Mar 13, 2020

DaanHoogland reopened this Mar 13, 2020

DaanHoogland marked this pull request as ready for review March 13, 2020 16:28

svenvogel added the type:security label Mar 16, 2020

downgrade discovery back to the official release

91144c0

jaxws version

f857477

DaanHoogland added component:build and removed type:security labels Mar 24, 2020

GabrielBrascher approved these changes Mar 24, 2020

View reviewed changes

DaanHoogland closed this Mar 24, 2020

DaanHoogland reopened this Mar 24, 2020

DaanHoogland merged commit 9b74a3a into apache:master Mar 25, 2020

DaanHoogland deleted the owasp-dep-tjek branch March 25, 2020 10:56

nvazquez mentioned this pull request Apr 5, 2020

Improvement on build time and new quality profile #4014

Merged

5 tasks

Conversation

DaanHoogland commented Mar 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

DaanHoogland commented Mar 13, 2020

Uh oh!

DaanHoogland commented Mar 13, 2020

Uh oh!

nvazquez commented Mar 13, 2020

Uh oh!

DaanHoogland commented Mar 14, 2020

Uh oh!

nvazquez commented Mar 15, 2020

Uh oh!

DaanHoogland commented Mar 16, 2020

Uh oh!

nvazquez commented Mar 16, 2020

Uh oh!

DaanHoogland commented Mar 16, 2020

Uh oh!

DaanHoogland commented Mar 23, 2020

Uh oh!

DaanHoogland commented Mar 24, 2020

Uh oh!

yadvr commented Mar 24, 2020

Uh oh!

blueorangutan commented Mar 24, 2020

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 24, 2020

Uh oh!

blueorangutan commented Mar 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DaanHoogland commented Mar 10, 2020 •

edited

Loading