Torsten Curdt’s weblog

Java classpath and directories

The java classpath and directories has always been quite a sad story. Of course you were always able to pass a directory of classes to the jvm like this:

java -classpath classes Main

But what you usually deal with these days are jars. Often quite a bunch of them. So fair enough – let’s add a directory with jars as well:

java -classpath classes:lib Main

Up until java 5 all you got was a ClassNotFoundException because java did not search for jars but classes in there. It just ignored the jars. So what pretty much everyone ended up doing is providing yet another shell script to build up the classpath and pass it to the jvm via command line. Of course for bigger projects this let to…

java -classpath lib/commons-logging-1.1.1.jar:lib/commons-jci-core-1.0.jar:commons-io-1.2.jar:lib/junit-3.8.2.jar:lib/maven-project-2.0.8.jar:lib/rhino-js-1.6.5.jar: ...

I guess you see where I am going …an unmanageable mess of a command line. But thankfully times have changed. Starting with java 6 (mustang) you can now use wildcards in the classpath:

java -classpath classes:lib/'*' Main

Naturally this means that the order of the jars is implicit. Which in turn means you need to be extra careful when there are classpath clashes. But in general I think this is a nice feature that should have been there from day one …and I just found out about it.

  • Thilina Mahesh Buddhika

    This is a saviour.


  • TheGuru
    Sounds like the whole classpath thing in java needs a rewrite/enhancement and transformed from being a simple search path directive into some sort of resource catalog e.g. classpath.xml
  • Steve
    I am wondering what will happen if I would like to use jar files and directories too in the classpath like this:

    java -classpath ../lib/'*';resources/xml;'.' com.myapp.MainClass

    And the MainClass is a separate class file which can accessible from the current directory ('.') and not part of the jar file.

    Is this will work?
    Thank you.
  • TheGuru
    More often than not one uses a script to construct the CLASSPATH, but what typically happens in the software development cycle is that slowly more tools, components, re-use, 3rd party products are introduced and the script automatically grows the CLASSPATH size to beyond the limit of either the LINE_SIZE or the memory size for shell variables depending on chosen implementation. Worse still, the architects who design the solution are often unaware of this silly (old) limitation. And worse again is that a lot of jars (e.g. commons) are often repeated between different 3rd party products and the script is often not smart enough to know that. And even worse again, when the application is delivered to the customer the whole application ends up being managed by some 3rd party JMX instrumenting management/framework tool that pre-pends its own CLASSPATH to the existing one.

    The disingenuous ways to crunch a CLASSPATH are.
    1) use symlinks e.g. ln -s /reallyannoyinglong3rdpartysoftwarepath2libdir /s1
    2) use dos SUBST command e.g. SUBST J: C:\reallyannoyinglong3rdpartysoftwarepath2libdir
    3) The following is dangerous, but can work when you know the products well.
    unjar all jars to a temp dir and jar them up as one jar file e.g. annoyinglybig3pp.jar
    4) Manually go through all jars of all 3rd party products and grow old figuring out what you don't use and somehow exclude it.
    5) Use a special shell or modify an existing one to cater for huge line or environment variable sizes.
    6) Incorporate some class loading facility within the running application itself.


    Thanks to this blog I now can use the escaped wildcard, but even better can use an application Manifest, but does anyone know if there is a limit to the size of the Class-Path: line in a Manifest file?

    The Manifest/Class-Path approach seems to imply that an app delivered with lots of lib/jars can be started by simply specifying the jar that has the full app Class-Path defined in its manifest. E.g. "java -cp /app1/lib/start_app1.jar app1.Main"

    I assume that if I'm using two 3rd party apps (app1 and app2) in my application (myapp) I should be able to create a trivial startup jar for each app using the above Manifest/Class-Path technique and run it as follows
    E.g. "java -cp /myapp/start_myapp.jar:/app1/lib/start_app1.jar:/app2/lib/start_app2.jar myapp.Main"

    If this works then this is great at solving the long CLASSPATH issue (and the spec says you can use the wildcard in the Manifest/Class-Path as well), but stay with me because I'm now going to present a java problem that has bothered me at times.

    When your application uses many external software components, then inside these external components may be common libraries that are at different revision levels. The first one that comes to mind is log4j. The developers of these external software components only think that you will be using the version that they supplied, however that particular version may interfere with other external software components when you put it at the start of the CLASSPATH. So you stick the newer version of log4j at the start of the CLASSPATH and you solve one problem but break something else. Unfortunately you don't own the external software and cannot apply a fix/update as it is part of a bigger software baseline. This is very unlike internal software component dependencies where you have better control and can use neat tools like IVY for managing them.

    However, if I were to use the Manifest/Class-Path approach and create trivial start_app jars for each app as before, is the class loader able to construct an internal search path that allows the correct use of the right version of log4j jar for the right app? That is, is there some internal search path group hierarchy that can be constructed using the above Manifest/Class-Path technique? My gut feeling is no.

    Would it be a good idea for java to facilitate some form of internal search path group hierarchy when it comes to searching for classes, other than just the URL/directory path, for the specific purpose of solving dependency issues when using common family components of varying revisions across many external apps?
    Using the Manifest/Class-Path technique to specify a group CLASSPATH to me seems like a good place to control this group search hierarchy, so when a class is requested by a running application the search is first performed in the local group hashtable for that class and if not found then the rest of the class hashtable is searched. I don’t think it would add too much of a runtime overhead as the whole class list is only searched once (in 2 steps that can even be run in parallel).

    Unfortunately I no longer have a need to test this because I no longer work in a place that uses a horrendously complex mix of software suites (i.e. telco), but would still appreciate the feedback.
  • M. A. Sridhar
    If you're running on linux, you can use the find and shell features to achieve this goal:

    java -classpath `find jarsDir -name '*.jar' | sed -e 's/ /:/g'` Main

    The find command produces all .jar files anywhere in the jarsDir directory, separated by spaces, and the sed command replaces spaces with colons, as needed by the classpath variable.
  • Ron
    On Linux, the script to add the jars in a certain directory is fairly simple...not quite so clean in Windows batch files. Could probably use Powershell, but I dug this up from the site listed below and adapted it for our use. It works. BTW, thanks for the Java6 tip!


    setLocal EnableDelayedExpansion
    set JarsDir=somedirectory\lib
    if defined classpath (set classpath=%classpath%;.) else (set classpath=.)
    for %%i in ("%JarsDir%\*.jar") do set classpath=!classpath!;%%i
    endlocal & set classpath=%classpath%
    set classpath=%CLASSPATH%
    echo %CLASSPATH%
  • Caligula
    @Tony: Sure, but sometimes that isn't what you need, like if you're running a JRuby REPL into some components of a web app--so you need all of Hibernate, your domain objects, JDBC driver, etc. Setting the classpath on the command line or a script is the only practical way to do that, other than creating a JAR that starts the REPL and has the (possibly changing) library requirements in the manifest.
  • Uuuh - nice! I didn't know you can do that! Thanks, Tony!
  • Tony
    Good info, but, as I code for the lowest release of Java 5, it wont help me. However just one thing.... Ewwww!!

    I never have a .bat / .sh file to launch my java code. Instead, I store the classpath in the main .jar's manifest, the .jar with the class that contains the 'public static void main(args[]s)' . Somewhat like this:

    Manifest-Version: 1.0
    Main-Class: com.test.Launcher

    Just ensure that each jar file has a space before and after on each line. Once done this way, The java application can be launched by using java -jar {mainjar}.jar , or even just double clicking from the desktop.

    Much easier, and more cross platform compatable (no .bat for windows, .sh for OS X, etc.)
  • TheGuru
    I actually hit the limits of the command interpreter's LINE_SIZE many times with needing to explicitly specify the path of all jars and had to do some disingenous work to overcome this.

    Generally, I think the java community should look at improving this even further considering the popularity of using such references (to a bucket load of external jars).
  • ant
    Thanks for posting this, I didn't know about the wildcard facility either.
blog comments powered by Disqus