About Dynamic Adding to Classpath in Clojure
Well, I guess people will just inevitably get into the problem of classpath, one way or another. The Classpath is a Lie described the problem very well: classpath is a lie. classpath, per se, is a simple list separated by colons, however, the real work is done by the Classloader.
Nevertheless, this isn't a post talking about classpath and ClassLoader. There are already a lot of great articles talking about it (links at the end of this post), and I can't claim I understand ClassLoaders to the extent that I can confidently teach others about it either.
This is a blog about what I have found during the process of trying to add new directories to classpath and require Clojure files in them at runtime. ClassLoader in Clojure is something very messy. The best strategy probably is to avoid the problem altogether. But still, if you really want to do it, I wish the following content can offer some help.
Use Builtin clojure.core/add-classpath
Clojure has a builtin add-classpath function. Although it has been deprecated, it works for simple use cases.
(defn check-dynamic-load []
(let [tmp-dir (.toFile (Files/createTempDirectory "classpath-demo" (into-array java.nio.file.attribute.FileAttribute [])))
tmp-clj (File/createTempFile "demo" ".clj" tmp-dir)
tmp-name (subs (.getName tmp-clj)
0
(.lastIndexOf (.getName tmp-clj)
"."))]
;; Add `tmp-dir` to `classpath` using builtin `add-classpath`.
(add-classpath (.toURL tmp-dir))
;; Put a Clojure file under the directory
(spit tmp-clj
(str "(ns " tmp-name ") (def a 1)"))
;
;; `require` the Clojure file, and resolve the variable
(assert (= 1 (var-get (requiring-resolve (symbol tmp-name
"a")))))
;; Update the Clojure file
(spit tmp-clj
(str "(ns " tmp-name ") (def a 2)"))
;; Reload the Clojure file
(require (symbol tmp-name)
:reload-all)
;; We can read the new value
(assert (= 2 (var-get (requiring-resolve (symbol tmp-name
"a")))))
(println "success")))
;; success
(check-dynamic-load)
If we evaluate the above code in a REPL or in cider, it works and prints "success". As we can see from the code, we can require a Clojure file whose path determined at runtime, and reload it to get the updated value.
However, there is a reason of it being deprecated. We can check its source code:
// The method used by clojure.core/add-classpath
static public void addURL(Object url) throws MalformedURLException{
URL u = (url instanceof String) ? toUrl((String) url) : (URL) url;
ClassLoader ccl = Thread.currentThread().getContextClassLoader();
if(ccl instanceof DynamicClassLoader)
((DynamicClassLoader)ccl).addURL(u);
else
throw new IllegalAccessError("Context classloader is not a DynamicClassLoader");
}
It checks if the current thread's ContextClassLoader is a DynamicClassLoader. If so, it will call the DynamicClassLoader's addURL method.
That means, this method will fail if there's some code set the current ContextClassLoader to something other than a DynamicClassLoader.
NoteWe expect
add-classpathcontinues to work in case because the convention of setting a newClassLoaderis to set the currentClassLoaderas the parent of the newly createdClassLoader. However,clojure.core/add-classpathonly checks the currentClassLoader, more on this in the section.
(let [future
(future
(let [cl (.getContextClassLoader (Thread/currentThread))]
(.setContextClassLoader (Thread/currentThread)
(java.net.URLClassLoader. (into-array java.net.URL [])
cl)))
(try (check-dynamic-load)
(assert "unreachable")
(catch Throwable t
(println "dynamic loading failed"))))]
;; "dynamic loading failed"
@future)
Use add-classpath from pomegranate
pomegranate provides a add-classpath that solves the problem described in the previous section.
The only thing we need to change is to replace clojure.core/add-classpath with cemerick.pomegranate/add-classpath.
(ns demo
(:require
[cemerick.pomegranate :as pomegranate]))
(defn check-dynamic-load-using-pomegranate []
(let [;; ...
]
;; same code as `check-dynamic-load`
(pomegranate/add-classpath (.toURL tmp-dir))
;; same code as `check-dynamic-load`
))
(let [future
(future
(let [cl (.getContextClassLoader (Thread/currentThread))]
(.setContextClassLoader (Thread/currentThread)
(java.net.URLClassLoader. (into-array java.net.URL [])
cl)))
(check-dynamic-load-using-pomegranate)
(print "success"))]
;; success
@future)
Unlike add-classpath from clojure.core, pomegranate's add-classpath try to find the ClassLoader closest to the Primordial ClassLoader that is compatible with add-classpath, and call the addURL method from it.
;; in `add-classpath` function in pomegranate.clj
(let [classloaders (classloader-hierarchy)]
(if-let [cl (last (filter modifiable-classloader? classloaders))]
(add-classpath jar-or-dir cl)
(throw (IllegalStateException. (str "Could not find a suitable classloader to modify from "
(mapv (fn [^ClassLoader c]
(-> c .getClass .getSimpleName))
classloaders))))))
Create a DynamicClassLoader When There isn't One
If you are running Clojure in REPL or with nrepl, the ContextClassLoader of the current thread will certainly be DynamicClassLoader, set by one of those tools. However, when you run clj command in a non-interactive manner, or use a AOT-compiled jar file, this wouldn't be the case.
This problem is quite easy to solve, we just need to set the ContextClassLoader to a DynamicClassLoader created by ourselves in the entrypoint of the program.
(defn -main [& args]
(let [cl (.getContextClassLoader (Thread/currentThread))]
(.setContextClassLoader (Thread/currentThread) (clojure.lang.DynamicClassLoader. cl)))
;; other code...
)
When Using kaocha
So far, so good. Except when you finish the code and try to test some code and run it under the kaocha test runner. kaocha also did its own thing with ClassLoader and provides a add-classpath method. It breaks the previous method.
By detecting kaocha's presence and calling its add-classpath, alongside with the pomegranate one solves the issue for me.
(when (find-ns 'kaocha.classpath)
((intern 'kaocha.classpath
'add-classpath)
new-path))
Pitfall When Using a Threadpool
When you create a thread, the new thread will inherit the ContextClassLoader of the thread created it. However when you explicitly or implicitly (like when using future) use a threadpool, the executor may choose an existing thread, which could have a ContextClassLoader different from the calling thread.
You may consider creating a thread directly instead of relying on future in this case.
(doto (Thread/new
(bound-fn []
;; code
))
(.start))
A Few More Words
This post is definitely not comprehensive, and there are still a lot things I currently do not understand. The method I have described works for me for now. If you want to understand more about this topic, I have listed a few links below.