some tutorials on Jenkins Pipelines

https://www.youtube.com/watch?v=-GsvomI4CCQ very pragmatic Simplilearn tutorial,

you can practice on https://www.katacoda.com/courses/jenkins/build-docker-images which gives you a dockerized Jenkins with console



Awesome power-jenkins tip-pack https://www.youtube.com/watch?v=6BIry0cepz4 :

- run Jenkins in Docker (jenkins/jenkins)

- select plugins you want to use (use plugins.txt to predefine a list of plugins)

- use agents, with swarm plugin to register, automate the agent provisioning and make them ephemeral

- don't use Maven jobs, because it's not reproduceable

- use pipelines, with Jenkinsfile (pipeline/stages/stage/steps)

- in pipelines, do all work on agents ("agent any")

- user input stage should be run on master, to avoid blocking executors ("agent none")

- limit number of stages

- don't change $env variable, use withEnv(["hello=world"]) instead

- parameters (?)

- use parallelism, for end to end tests , and performance tests and in separate nodes

- "scripted" is groovish for power user, declarative is ok for regular user

- pipelines should be small, they are orchestration tools... do the heavy stuff fin shell scripts which are easier to test

- in a pipeline everything is serializable so it can be resumed on failure (continuation-passing style)... but some classes are not serializable like groovy.text.StreamingTemplateEngine, then you have to wrap it

- BlueOcean plugin, with Editor for declarative pipelines

- use shared libraries in pipelines to reuse code, also reuse files

- use views to show only the jobs you are interested in

- BuildMonitor plugin to view jobs

- API in JSON or XML

- to-have plugins: BuildMonitor, Job Config History (to version freestyle jobs), Job DSL, Throttle Concurrent Builds, Timestamper, Version Number plugin & Build-name-setter

OpenShift CI/CD

https://www.youtube.com/watch?v=65BnTLcDAJI good video on CI/CD, part 1


https://www.youtube.com/watch?v=wSFyg6Etwx8 part 2



https://www.youtube.com/watch?v=kbbK0VEy2qM OpenShift 4 CI/CD

essential is to have installed in Jenkins the "OpenShift Jenkins Pipeline (DSL) Plugin" https://github.com/openshift/jenkins-client-plugin



https://www.youtube.com/watch?v=pMDiiW1UqLo Openshift Pipelines with Tekton https://cloud.google.com/tekton/ and here is the code https://github.com/openshift/pipelines-tutorial

Jenkins enable project based security

Sometimes you share a Jenkins instance amongst several projects (IMHO this is bad practice, each project should have its own Jenkins to minimize interference)

This is how to do it (copied from https://stackoverflow.com/questions/32111825/jenkins-how-to-set-authorization-on-project-basis )


a) make sure Matrix Authorization Strategy Plugin is installed (Manage Jenkins/Manage Plugins/Installed Plugins)

b) "Manage Jenkins", "Configure Global Security", add the target user to the "Project-based Matrix Authorization Strategy ",
add the target user with permissions "Overall/Read and Job/Read"

c) on the main page, select the target project, "Enable project-based security", add the target user and on the right, click on the "grant all permissions" button

at this point the user has login and can administer the target project, but only view other projects.

Jenkins would be a much better tool if all these configuration operations could be easily scriptable. Nowadays it's just a huge clickodrome and very awkward to manage, you have to wade through zillion of configuration pages and unless you are really experienced it's sometimes frustrating ... you don't even have a "search" functionality for configuration options, you have to remember all locations by heart...

Ruminations about Jenkins

The jobs are defined in here:

$JENKINS_HOME/jobs

For every project (item), there is a folder.

In every folder there is a config.xml file which hopefully should contain the entire project definition.

My assumption is that one should simply save the config.xml, and this is enough to recreate the project elsewhere.

I am worried because the Project definitions in Jenkins are not saved in bitbucket,
and we don’t have an automated way to export them from PROD and import them to a UAT instance.

One COULD tar the $JENKINS_HOME/jobs folder, but it’s very bulky.


[jobs]$ cd myproject/

[myproject]$ ls -ltra

drwxr-xr-x. 3 pippo pippogroup 4096 Jul 18 2016 modules
-rw-r--r--. 1 pippo pippogroup 6 Apr 25 2018 nextBuildNumber
lrwxrwxrwx. 1 pippo pippogroup 26 Apr 25 2018 lastSuccessful -> builds/lastSuccessfulBuild
lrwxrwxrwx. 1 pippo pippogroup 22 Apr 25 2018 lastStable -> builds/lastStableBuild
drwxr-xr-x. 5 pippo pippogroup 4096 Apr 25 2018 workspace
drwxr-xr-x. 24 pippo pippogroup 4096 Apr 25 2018 builds
-rw-r--r--. 1 pippo pippogroup 7257 Apr 25 2018 config.xml
drwxr-xr-x. 5 pippo pippogroup 4096 Apr 25 2018 .
drwxr-xr-x. 360 pippo pippogroup 20480 Mar 18 15:06 ..


especially the workspace is a PIG

So for the time being I will simply tar up all the config.xml and untar them in UAT:


cd $JENKINS_HOME/jobs

find . –maxdepth 2 –name config.xml | tar cvf /var/tmp/alljenkinsconfig.tar –T –

(maxdepth is important to avoid picking up files coming from the workspaces)

However we should really really IMHO push all those config.xml to bitbucket, regularly – ideally automatically whenever someone changes a config.xml:

git init
#set origin and remote
find . –maxdepth 2 –name config.xml –exec git add {} \;
git commit -m "blablabla"
git push


Incidentally, many project folders contain spaces, which makes it much trickier to write scripts to manipulate them.

I am not in favor of Capital Punishment, apart from cases when people create folders containing spaces.


Jenkins sucks anyway. Design is from Napoleonian Era, UI made by a freak, configuration freakishly XML based without fluent Administration Groovy API,
actually in the old times they were much better at designing stuff, think of the Tour Eiffel or the Pyramids and the Coliseum.
Nowadays any idiotic monkey can code freakish products like Maven or Jenkins and become a celebrity.

Jenkins Pipeline and OWASP dependency check

a) in Jenkins, configure a M3 instance in your "Global Tools" configuration section

b) make sure you have installed the OWASP dependency check plugin

c) create a Pipeline Jenkins project, name "owasptest", and simply paste this pipeline (it's a copy from the sample built-in "github and maven" pipeline):

node {
   def mvnHome
   stage('Preparation') { // for display purposes
      // Get some code from a GitHub repository
      git 'https://github.com/jglick/simple-maven-project-with-tests.git'
      // Get the Maven tool.
      // ** NOTE: This 'M3' Maven tool must be configured
      // **       in the global configuration.           
      mvnHome = tool 'M3'
   }
   stage('Build') {
      // Run the maven build
      if (isUnix()) {
         sh "'${mvnHome}/bin/mvn' -Dmaven.test.failure.ignore clean package"
      } else {
         bat(/"${mvnHome}\bin\mvn" -Dmaven.test.failure.ignore clean package/)
      }
   }
   stage('Results') {
      junit '**/target/surefire-reports/TEST-*.xml'
      archive 'target/*.jar'
   }
     stage("Dependency Check") {
    dependencyCheckAnalyzer datadir: 'dependency-check-data', isFailOnErrorDisabled: true, hintsFile: '', includeCsvReports: false, includeHtmlReports: false, includeJsonReports: false, isAutoupdateDisabled: false, outdir: '', scanpath: '', skipOnScmChange: false, skipOnUpstreamChange: false, suppressionFile: '', zipExtensions: ''

    dependencyCheckPublisher canComputeNew: false, defaultEncoding: '', healthy: '', pattern: '', unHealthy: ''

    archiveArtifacts allowEmptyArchive: true, artifacts: '**/dependency-check-report.xml', onlyIfSuccessful: true
  }
  
}

and leave "Use Groovy Sandbox" checked




Building jar: /home/centos/.jenkins/workspace/owasptest/target/simple-maven-project-with-tests-1.0-SNAPSHOT.jar

[DependencyCheck] Executing Dependency-Check with the following options:
[DependencyCheck] -name = owasptest
[DependencyCheck] -scanPath = /home/centos/.jenkins/workspace/owasptest
[DependencyCheck] -outputDirectory = /home/centos/.jenkins/workspace/owasptest
[DependencyCheck] -dataDirectory = /home/centos/.jenkins/workspace/owasptest/dependency-check-data
[DependencyCheck] -dataMirroringType = none
[DependencyCheck] -isQuickQueryTimestampEnabled = true
[DependencyCheck] -jarAnalyzerEnabled = true
[DependencyCheck] -nodePackageAnalyzerEnabled = true
[DependencyCheck] -nspAnalyzerEnabled = true
[DependencyCheck] -composerLockAnalyzerEnabled = true
[DependencyCheck] -pythonDistributionAnalyzerEnabled = true
[DependencyCheck] -pythonPackageAnalyzerEnabled = true
[DependencyCheck] -rubyBundlerAuditAnalyzerEnabled = false
[DependencyCheck] -rubyGemAnalyzerEnabled = true
[DependencyCheck] -cocoaPodsAnalyzerEnabled = true
[DependencyCheck] -swiftPackageManagerAnalyzerEnabled = true
[DependencyCheck] -archiveAnalyzerEnabled = true
[DependencyCheck] -assemblyAnalyzerEnabled = true
[DependencyCheck] -centralAnalyzerEnabled = true
[DependencyCheck] -nuspecAnalyzerEnabled = true
[DependencyCheck] -nexusAnalyzerEnabled = false
[DependencyCheck] -autoconfAnalyzerEnabled = true
[DependencyCheck] -cmakeAnalyzerEnabled = true
[DependencyCheck] -opensslAnalyzerEnabled = true
[DependencyCheck] -showEvidence = true
[DependencyCheck] -formats = XML
[DependencyCheck] -autoUpdate = true
[DependencyCheck] -updateOnly = false
[DependencyCheck] Data directory created
[DependencyCheck] Scanning: /home/centos/.jenkins/workspace/owasptest
[DependencyCheck] Analyzing Dependencies

[Pipeline] dependencyCheckPublisher

[DependencyCheck] Collecting Dependency-Check analysis files...
[DependencyCheck] Searching for all files in /home/centos/.jenkins/workspace/owasptest that match the pattern **/dependency-check-report.xml
[DependencyCheck] Parsing 1 file in /home/centos/.jenkins/workspace/owasptest
[DependencyCheck] Successfully parsed file /home/centos/.jenkins/workspace/owasptest/dependency-check-report.xml with 0 unique warnings and 0 duplicates.


At this point you can view the report in http://localhost:9090/job/owasptest/lastSuccessfulBuild/artifact/dependency-check-report.xml



To get interesting result you should use https://github.com/WebGoat/WebGoat.git

At the end, a monster DB (330 MB) is built at /home/centos/.jenkins/workspace/owasptest/dependency-check-data/dc.h2.db
To learn mode about this DB, read here https://github.com/jeremylong/DependencyCheck/tree/master/core/src/main/resources/data
and here https://github.com/jeremylong/DependencyCheck/blob/master/core/src/main/resources/dependencycheck.properties

Ref: https://issues.jenkins-ci.org/browse/JENKINS-37437

Jenkins sucks

Jenkins sucks. Ok it could suck more, but let me tell you what I find really frustrating from a begginner's perspective

a) documentation is pathetic. The Jenkins User Handbook until a few months ago was full of empty sections with only "work in progress".... now it's a bit better, but still....

b) plugins is a cosmic chaos... any basic installation would include at least some 100 common plugins, so why not include them in the Jenkins baseline and avoid all this chaos.... also, plugin documentation is most of the time really basic

c) no development IDE, no way to remote debug. Should I develop code in BlueOcean plugin in a browser??? are you serious??? When one is used to IntelliJ, having to code Jenkins Pipelines in Notepad++ is like going back to Stone Age. The Eclipse plugin is laughable, you can just remotely invoke the linter for validation.... no autocomplete, no inline javadoc, NOTHING

d) Groovy is good, but management of libraries is a bit shaky.... however this is not the weakest point, at least they don't make me code in XML like Maven or Ant do, or in a ridiculous DSL like Puppet does....

e) the UI was coded by children... you have to click for hours to find something... not even a search tool to point you quickly where a certain parameter is defined.... it's the typical click-o-drome of tools that are designed for non-programmers and end up being a nightmare of clicks and hidden corners.

Jenkins pipelines

https://www.youtube.com/watch?v=TsWkZLLU-s4&t=1188s


https://github.com/abayer/dec-jam-declarative-demos

http://www.mdoninger.de/2011/11/07/write-groovy-scripts-for-jenkins-with-code-completion.html Jenkins code completion in Eclipse (doh! Who would have thought of using a IDE to write code! The new frontier is write pipeline code in a browser on a github tab.... next they will ask you to write pcode in hex format... then we will eat bananas on trees again)

picture 1: Jenkins developers discussing the use of Notepad to improve coding experience



picture 2: Jenkins developers celebrate their first successful Scripted pipeline

picture 3: Jenkins developers discover IDE

Jenkins console

interesting presentation (skip first 9 minutes)



Scripts:

println "I hacked you"
new File('/etc/passwd').text


println "${Jenkins.instance.root}"

"ls -ltr /".execute().text

Jenkins.getInstance().metaClass.methods*.name.sort().unique()


on Jenkins CLI https://wiki.jenkins.io/display/JENKINS/Jenkins+CLI

the scripts by Sam https://github.com/samrocketman/jenkins-script-console-scripts

https://github.com/jenkinsci/jenkins-scripts


http://groovy-lang.org/learn.html

https://wiki.jenkins.io/display/JENKINS/Jenkins+Script+Console

Jenkins blueocean plugin with Docker

https://jenkins.io/doc/book/blueocean/getting-started/ and here the Docker installation instructions https://jenkins.io/doc/book/installing/#docker

Here all the Docker images https://hub.docker.com/r/jenkinsci/blueocean/tags/


Then I run this:

( http://devdocs.io/docker~1.12/engine/reference/commandline/run/index I am using Docker 1.12 )

docker run -u root -d -p 8080:8080 -p 50000:50000 -v jenkins-data:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock jenkinsci/blueocean

port 50000 is needed only if you plan to use slave build agents

(I had to remove --rm otherwise I get Conflicting options: --rm and -d )

docker ps gives you the containerid -> docker exec -ti 03e820715e74 bash

cat /var/jenkins_home/secrets/initialAdminPassword
copy and paste in localhost:8080 to unlock Jenkins




https://jenkins.io/doc/book/installing/#docker

Configure Jenkins to run the OWASP Security Check plugin

First make sure that your Jenkins installation is configured with Maven 3.5

You should also install the "OWASP Dependency-Check Plugin" plugin - for this, go to the "manage Jenkins", "manage plugins" and you should find it in the "available plugins" (if not, probably you have to download the hpi files, and copy them in the "plugins" directory under the Jenkins home folder... see my previous post on which plugin files are needed ).

create a Maven project:

webgoat_maven

Source Code Manamegent: Git

Repository URL = https://github.com/WebGoat/WebGoat.git

Branch specifier = */develop

Remove all "build triggers"

Pre-Steps : leave empty

Build/Root POM = pom.xml

Goals and Options = package -DskipTests=true

Post Steps (run regardless...) = Invoke OWASP dependency check analysis
click on "advanced", enable "Generate optional HTML report" and "Generate optional vulnerability report (HTML)"

Post-build Actions: add "Publish OWASP dependency check results"
click on "advanced", set 5 in the "failed" (red circle) "Priority high" column.... so the build will fail if there are more than 5 highly vulnerable components.



In the console output of the build, you should see something like this:

[DependencyCheck] OWASP Dependency-Check Plugin v3.0.0
[DependencyCheck] Executing Dependency-Check with the following options:
[DependencyCheck]  -name = Pierre
[DependencyCheck]  -scanPath = /path/to/workspace/Pierre
[DependencyCheck]  -outputDirectory = /path/to/workspace/Pierre
[DependencyCheck]  -dataDirectory = /path/to/workspace/Pierre/dependency-check-data
[DependencyCheck]  -dataMirroringType = none
[DependencyCheck]  -isQuickQueryTimestampEnabled = true
[DependencyCheck]  -jarAnalyzerEnabled = true
[DependencyCheck]  -nspAnalyzerEnabled = true
[DependencyCheck]  -composerLockAnalyzerEnabled = true
[DependencyCheck]  -pythonDistributionAnalyzerEnabled = true
[DependencyCheck]  -pythonPackageAnalyzerEnabled = true
[DependencyCheck]  -rubyBundlerAuditAnalyzerEnabled = true
[DependencyCheck]  -rubyGemAnalyzerEnabled = true
[DependencyCheck]  -cocoaPodsAnalyzerEnabled = true
[DependencyCheck]  -swiftPackageManagerAnalyzerEnabled = true
[DependencyCheck]  -archiveAnalyzerEnabled = true
[DependencyCheck]  -assemblyAnalyzerEnabled = true
[DependencyCheck]  -centralAnalyzerEnabled = true
[DependencyCheck]  -nuspecAnalyzerEnabled = true
[DependencyCheck]  -nexusAnalyzerEnabled = false
[DependencyCheck]  -autoconfAnalyzerEnabled = true
[DependencyCheck]  -cmakeAnalyzerEnabled = true
[DependencyCheck]  -opensslAnalyzerEnabled = true
[DependencyCheck]  -showEvidence = true
[DependencyCheck]  -formats = XML HTML VULN 
[DependencyCheck]  -autoUpdate = true
[DependencyCheck]  -updateOnly = false

If the "jarAnalyzerEnabled" is not true, then something is wrong.


If you see "org.owasp.dependencycheck.data.update.exception.UpdateException: Unable to download the NVD CVE data..... Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Unable to resolve domain 'nvd.nist.gov' " , most likely you are behind a proxy. You can still build the h2 DB - containing all the vulnerabilities feed - and provide it offline to the Jenkins job. But to build this DB you must run the job on a computer connected to internet, then look in the workspace for a db.h2 file.
This configuration is to be done in "Invoke OWASP dependency check analysis", then "advanced" and set "Data directory" to the folder where you have copied the h2 db file. Also, check the "Disable NVD auto-update" flag.

If this still fails with this error, then I really don't know where the issue is, probably AGAIN a proxy problem as also explained here https://github.com/jeremylong/DependencyCheck/issues/932:

[DependencyCheck] Message: Could not connect to Central search. Analysis failed.
[DependencyCheck] org.owasp.dependencycheck.analyzer.exception.AnalysisException: Could not connect to Central search. Analysis failed.
[DependencyCheck]  at org.owasp.dependencycheck.analyzer.CentralAnalyzer.analyzeDependency(CentralAnalyzer.java:244)
[DependencyCheck]  at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:137)
[DependencyCheck]  at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
[DependencyCheck]  at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
[DependencyCheck]  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[DependencyCheck]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[DependencyCheck]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[DependencyCheck]  at java.lang.Thread.run(Thread.java:745)
[DependencyCheck] Caused by: java.io.IOException: Finally failed connecting to Central search. Giving up after 5 tries.
[DependencyCheck]  at org.owasp.dependencycheck.analyzer.CentralAnalyzer.fetchMavenArtifacts(CentralAnalyzer.java:288)
[DependencyCheck]  at org.owasp.dependencycheck.analyzer.CentralAnalyzer.analyzeDependency(CentralAnalyzer.java:198)
[DependencyCheck]  ... 7 more


[DependencyCheck] Caused by: java.net.UnknownHostException: search.maven.org
[DependencyCheck]  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
[DependencyCheck]  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[DependencyCheck]  at java.net.Socket.connect(Socket.java:589)
[DependencyCheck]  at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
[DependencyCheck]  at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[DependencyCheck]  at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[DependencyCheck]  at sun.net.www.http.HttpClient.(HttpClient.java:211)
[DependencyCheck]  at sun.net.www.http.HttpClient.New(HttpClient.java:308)
[DependencyCheck]  at sun.net.www.http.HttpClient.New(HttpClient.java:326)
[DependencyCheck]  at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
[DependencyCheck]  at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
[DependencyCheck]  at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
[DependencyCheck]  at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
[DependencyCheck]  at org.owasp.dependencycheck.data.central.CentralSearch.searchSha1(CentralSearch.java:127)
[DependencyCheck]  at org.owasp.dependencycheck.analyzer.CentralAnalyzer.fetchMavenArtifacts(CentralAnalyzer.java:266)

one can try to set -Danalyzer.central.enabled=false
(see https://github.com/jeremylong/DependencyCheck/blob/master/dependency-check-core/src/main/resources/dependencycheck.properties ) and/or enable proxy for https://search.maven.org/solrsearch/select


It's nice to read https://jeremylong.github.io/DependencyCheck/general/internals.html on how the analyzer works.


The NVD (National Vulnerability Database) CVE (Common Vulnerabilities and Exposures) feeds are here https://nvd.nist.gov/vuln/data-feeds. They contain the Common Platform Enumeration CPE catalog of all known vulnerabilities and Common Weakness Enumeration (CWE) .


How to trigger a daily job to get NVD updates: https://medium.com/@PrakhashS/checking-vulnerabilities-in-3rd-party-dependencies-using-owasp-dependency-check-plugin-in-jenkins-bedfe8de6ba8