Things you shouldn’t do #2: Hack around maven-shade-plugin limitations
The maven-shade-plugin is a handy plugin which allows you to collect all of your project dependencies – incl. transitive dependencies – and put everything together into a single, shaded jar file. I use this plugin to deliver a Java application as a single jar file containing all needed resources to simplify the application launch descriptor. But sometimes you have more than one Maven project that you want to deliver as a single jar file. And sometimes you even have a parent Maven project that contains a list of dependencies and for each of them you want to create a single jar file. This kind of project setup might lead you to the idea to call the maven-shade-plugin (multiple times) directly from command line to create your jar files which will end up with the follow error message: “You have invoked the goal directly from the command line. This is not supported. Please add the goal to the default lifecycle via an element in your POM and use “mvn package” to have it run.”. If you still want to use the plugin and if you’re not afraid of bloody hacks than continue reading – otherwise you have to accept this limitation since it’s there for a good reason 
Assuming the following pom.xml file as a starting point which contains a list of dependencies (line 37-56). For each dependency we want to create a single jar file containing the transitive dependencies of each dependency. To do so we’ll call a Python script (line 24-25) via the maven-antrun-plugin:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | <?xml version="1.0"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd" > <modelVersion>4.0.0</modelVersion> <groupId>parent</groupId> <artifactId>parent</artifactId> <version>1.0-SNAPSHOT</version> <name>parent</name> <packaging>pom</packaging> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-antrun-plugin</artifactId> <version>1.6</version> <executions> <execution> <phase>install</phase> <configuration> <target> <exec executable="python" failonerror="true"> <arg value="src/main/python/createJars.py" /> </exec> </target> </configuration> <goals> <goal>run</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>dependency</groupId> <artifactId>A</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>dependency</groupId> <artifactId>B</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>dependency</groupId> <artifactId>C</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> </dependencies> </project> |
This Python script is the starting point of convincing the maven-shade-plugin to create an independent jar per listed dependency of the main pom.xml file. The basic idea is that we’ll create independent pom.xml files – one per dependency – where each will contain only one dependency of the original dependency list. Hacky, isn’t it? But it does solve several problems:
- An independent pom.xml containing only one of the dependencies isolates the transitive closure from the remaining dependencies of the main pom.xml file.
- You’re able to build each of those created pom.xml directly with the usual Maven commands.
- You can write some quick Python wiring-code to automate the whole process.
- You don’t have to modify the maven-shade-plugin or any other Maven component that would use all dependencies at the same time to determine the transitive closure.
To do so line 12 and 27 define a pom.xml header and footer that we’ll use to template our individual pom.xml files with. Therefore we have to inject the version and all the dependency details into the to be templated pom.xml files. Line 60 will resolve the version of the original pom.xml file and line 73 of the Python script will resolve all dependencies which is all done via simple XML operations.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 | #!/usr/bin/env python # -*- coding: utf-8 -*- import os import sys import shutil import subprocess import datetime import xml.dom.minidom from xml.dom import Node POM_XML_HEADER = """ <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd" > <modelVersion>4.0.0</modelVersion> <groupId>test</groupId> <artifactId>%(artifactId)s</artifactId> <version>%(version)s</version> <name>%(artifactId)s</name> <packaging>jar</packaging> <description>%(artifactId)s</description> <dependencies> """ POM_XML_FOOTER = """ </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>1.4</version> <executions> <execution> <id>shade</id> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <outputFile>target/%(artifactId)s-%(version)s.jar</outputFile> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> """ # check that the pom.xml file does exist if not os.path.exists("pom.xml"): print "ERROR: the pom.xml file doesn't exist!" sys.exit(-1) # resolve the version of the pom.xml file version = None try: document = xml.dom.minidom.parse("pom.xml") for childNode in document.childNodes[0].childNodes: nodeName = childNode.localName if nodeName != None and nodeName.find("version") != -1: version = childNode.childNodes[0].nodeValue break except Exception, (exception): print "ERROR: unable to resolve version from 'pom.xml' file due to thrown exception: " + str(exception) sys.exit(-2) # resolve the <dependencies> tag from the pom.xml file dependencies = None try: document = xml.dom.minidom.parse("pom.xml") for childNode in document.childNodes[0].childNodes: nodeName = childNode.nodeName if nodeName != None and nodeName.find("dependencies") != -1: dependencies = childNode break except Exception, (exception): print "ERROR: unable to resolve <dependencies> tag from 'pom.xml' file!" print exception sys.exit(-3) # make sure the directory which will stores the pom.xml files does exists mavenDir = "target/maven" if os.path.exists(mavenDir): print "WARN: the '" + mavenDir + "' directory already exists! deleting directory.." shutil.rmtree(mavenDir) os.makedirs(mavenDir) # iterate all direct dependencies try: for childNode in dependencies.childNodes: if childNode.nodeType != Node.ELEMENT_NODE: continue # resolve the artifactId of the given dependency artifactId = None for dependencyChildNode in childNode.childNodes: if dependencyChildNode.nodeType != Node.ELEMENT_NODE: continue if dependencyChildNode.localName.find("artifactId") != -1: artifactId = dependencyChildNode.childNodes[0].nodeValue break if artifactId == None: print "ERROR: unable to resolve artifactId!" sys.exit(-4) print "INFO: creating dedicated pom.xml file for dependency: " + artifactId dependencyMavenDir = os.path.join(mavenDir, artifactId) os.makedirs(dependencyMavenDir) pomFile = open(os.path.join(dependencyMavenDir, "pom.xml"), "w") templateDictionary = {'artifactId': artifactId, 'version': version} pomFile.write(POM_XML_HEADER % templateDictionary) pomFile.write("\t\t" + childNode.toxml()) pomFile.write(POM_XML_FOOTER % templateDictionary) pomFile.close() print "INFO: running Maven build for dependency: " + artifactId process = subprocess.Popen( "mvn clean install", cwd=dependencyMavenDir, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) for line in process.stdout: print line.strip() returncode = process.wait() if returncode != 0: print "ERROR: maven build for dependency: '" + artifactId + "' has failed!" print "ERROR: maven build has returned error code: " + str(returncode) print "ERROR: check the above log output for more information" sys.exit(-5) except Exception, (exception): print "ERROR: unable to parse the <dependencies> tag of the 'pom.xml' file!" print exception sys.exit(-6) |
Before we’re ready to iterate over the dependencies we have to make sure that the temporarily output directory in the target directory does exist. Than will use again some XML operations to get the details of each dependency (line 104) of the original pom.xml file which we’ll template into our pom.xml header and footer via Pythons build in string templating mechanism. The templated string is than written to an independent pom.xml file in the target directory (line 113).
After the independent pom.xml files have been templated and written to the temporarily target directory, line 121 of the script will use Pythons subprocess module to create a process to call mvn clean install for each pom.xml file. This Maven call will than use the maven-shade-plugin as defined in the pom.xml footer to create a single jar file in the local target directory. The Maven output for our main pom.xml file will look like this (note the [exec] prefixes for the launched Python script and Maven sub-builds via the maven-antrun-plugin):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | [INFO] Scanning for projects... [INFO] -------------------------------------------------------------------- [INFO] Building parent [INFO] task-segment: [clean, install] [INFO] -------------------------------------------------------------------- [INFO] [antrun:run {execution: default}] [INFO] Executing tasks main: [exec] INFO: creating dedicated pom.xml file for dependency: A [exec] INFO: running Maven build for dependency: A [exec] [INFO] Scanning for projects... [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] Building A [exec] [INFO] task-segment: [clean, install] [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] [shade:shade {execution: shade}] [exec] [INFO] Including dependency:A:jar:1.0-SNAPSHOT in the shaded jar. [exec] [INFO] [install:install {execution: default-install}] [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] BUILD SUCCESSFUL [exec] [INFO] ---------------------------------------------------------- [exec] INFO: creating dedicated pom.xml file for dependency: B [exec] INFO: running Maven build for dependency: B [exec] [INFO] Scanning for projects... [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] Building B [exec] [INFO] task-segment: [clean, install] [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] [shade:shade {execution: shade}] [exec] [INFO] Including dependency:B:jar:1.0-SNAPSHOT in the shaded jar. [exec] [INFO] [install:install {execution: default-install}] [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] BUILD SUCCESSFUL [exec] [INFO] ---------------------------------------------------------- [exec] INFO: creating dedicated pom.xml file for dependency: C [exec] INFO: running Maven build for dependency: C [exec] [INFO] Scanning for projects... [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] Building C [exec] [INFO] task-segment: [clean, install] [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] [shade:shade {execution: shade}] [exec] [INFO] Including dependency:C:jar:1.0-SNAPSHOT in the shaded jar. [exec] [INFO] [install:install {execution: default-install}] [exec] [INFO] ---------------------------------------------------------- [exec] [INFO] BUILD SUCCESSFUL [exec] [INFO] ---------------------------------------------------------- [INFO] Executed tasks [INFO] -------------------------------------------------------------------- [INFO] BUILD SUCCESSFUL [INFO] -------------------------------------------------------------------- |
After the Python script is done and every individual pom.xml file has been build by Maven you can easily pick up the created jar files from the target/maven/<artifactId>/ folders of the main Maven project. It’s also possible to add plugins like the maven-jarsigner-plugin to the pom.xml footer of the Python script if you e.g. need to sign your jar files afterwards.
I’m aware that this ‘solution’ looks quite nasty compared to the usual way of using Maven plugins to solve those kind of tasks but the overall-process is hidden behind a single Python script call in the main pom.xml file and adding more dependencies to the main pom.xml just works as usual. Also you don’t have to update the Python script every time you do add a new dependency to the pom.xml file.
Still it would be technically possible to add this to a self-written Maven plugin but I personally think that such a plugin looks even more wired and hacky since you would have to temporarily create Maven artefacts programmatically inside a Maven plugin including triggering serveral Maven sub-builds or to try to use the maven-shade-plugin code programmatically too. That’s why I prefer a Python script to do this job since it contains simple wiring-scripting code that just glues several Maven calls together where each can be used as just any other Maven project / build – this also includes the normal usage of the maven-shade-plugin.