Most of the tools I push to github are created out of something that I needed at the moment but could not find a good alternative for. cprop was one of such libraries. It sat there on github all alone for quite some time, and was used only by several people on my team, until it was integrated into Luminus.
Suddenly I started talking to many different people who found flaws in it, or just wanted to add features. I learned a couple of interesting usages from Heroku guys, as well as the importance of merging creds with Vault, coexisting with configs from other fault tolerant and external services such as Consul and more.
One of the useful cprop features is merging configs from various sources. Which is quite an open extension point: i.e. once cprop does its work and comes up with an app config, you can decide how and what will be merged with it before it really becomes a thing. It can be a local map, a .properties file, more ENV vars, more system properties, more configs from anywhere else, including remote/external resources, result from which can be converted to an EDN map.
To enable this merge extension point cprop has several tools that in practice could be really useful on its own: i.e. can be used outside of the (load-config) scope.
Loading from various sources
Could be used as OS, file system and edn oriented I/O tools. Also quite useful in the REPL.
Loading form a classpath
(require '[cprop.source :as cs]) (cs/from-resource "path/within/classpath/to-some.edn") |
Loads an EDN file anywhere from within a classpath into a Clojure map.
Loading from a file system
(require '[cprop.source :as cs]) (cs/from-file "/path/to/something.edn") |
Loads an EDN file from a file system with an absolute path into a Clojure map.
Loading from system properties
(require '[cprop.source :as cs]) (cs/from-system-props) |
Loads all the system properties into a Clojure map. i.e. all the properties that are set with
-Dkey=value, or programmatically set with System.setProperty(key, value), etc.
System properties are usually separated by . (periods). cprop will convert these periods to - (dashes) for key separators.
In order to create a structure in the resulting EDN map use _ (an underscore).
For example:
-Dhttp_pool_socket.timeout=4242 |
or
System.setProperty("http_pool_socket.timeout" "4242"); |
will be read into:
{:http {:pool {:socket-timeout 4242}}} |
notice how . was used as - key separator and _ was used to “get-in”: i.e. to create a hierarchy.
Loading from OS (ENV variables)
(require '[cprop.source :as cs]) (cs/from-env) |
Loads all the environment variables into a Clojure map.
ENV variables lack structure. The only way to mimic the structure is via use of an underscore character. The _ is converted to - by cprop, so instead, to identify nesting, two underscores can be used.
For example:
export HTTP__POOL__SOCKET_TIMEOUT=4242 |
would be read into:
{:http {:pool {:socket-timeout 4242}}} |
Notice how two underscores are used for “getting in” and a single underscore just gets converted to a dash as a key separator. More about it, including type inference, in the docs
Loading from .properties files
(require '[cprop.source :as cs]) (cs/from-props-file "/path/to/some.properties") |
Loads all the key value pairs from .properties file into a Clojure map.
The traditional syntax of a .properties file does not change. For example:
.means structure
four.two=42 would be translated to {:four {:two 42}}
_would be a key separator
fourty_two=42 would be translated to {:forty-two 42}
,in a value would be a seq separator
planet.uran.moons=titania,oberon would be translated to {:planet {:uran {:moons ["titania" "oberon"]}}}
For example let’s take a solar-system.properties file:
## solar system components components=sun,planets,dwarf planets,moons,comets,asteroids,meteoroids,dust,atomic particles,electromagnetic.radiation,magnetic field star=sun ## planets with Earth days to complete an orbit planet.mercury.orbit_days=87.969 planet.venus.orbit_days=224.7 planet.earth.orbit_days=365.2564 planet.mars.orbit_days=686.93 planet.jupiter.orbit_days=4332.59 planet.saturn.orbit_days=10755.7 planet.uran.orbit_days=30688.5 planet.neptune.orbit_days=60148.35 ## planets natural satellites planet.earth.moons=moon planet.jupiter.moons=io,europa,ganymede,callisto planet.saturn.moons=titan planet.uran.moons=titania,oberon planet.neptune.moons=triton # favorite dwarf planet's moons dwarf.pluto.moons=charon,styx,nix,kerberos,hydra |
(cs/from-props-file "solar-system.properties") |
will convert it to:
{:components ["sun" "planets" "dwarf planets" "moons" "comets" "asteroids" "meteoroids" "dust" "atomic particles" "electromagnetic.radiation" "magnetic field"], :star "sun", :planet {:uran {:moons ["titania" "oberon"], :orbit-days 30688.5}, :saturn {:orbit-days 10755.7, :moons "titan"}, :earth {:orbit-days 365.2564, :moons "moon"}, :neptune {:moons "triton", :orbit-days 60148.35}, :jupiter {:moons ["io" "europa" "ganymede" "callisto"], :orbit-days 4332.59}, :mercury {:orbit-days 87.969}, :mars {:orbit-days 686.93}, :venus {:orbit-days 224.7}}, :dwarf {:pluto {:moons ["charon" "styx" "nix" "kerberos" "hydra"]}}} |
Converting for other sources
Most Java apps store their configs in .properties files. Most docker deployments rely on ENV variables. cprop has some open tools it uses internally to work with these formats to bring EDN closer to non EDN apps and sources.
EDN to .properties
(require '[cprop.tools :as t]) (t/map->props-file config) |
Converts config map into a .properties file, saves the file under temp directory and returns a path to it.
For example, let’s say we have a map m:
{:datomic {:url "datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic"}, :source {:account {:rabbit {:host "127.0.0.1", :port 5672, :vhost "/z-broker", :username "guest", :password "guest"}}}, :answer 42} |
(t/map->props-file m) |
would convert it to a property file and would return an OS/env specific path to it, in this case:
"/tmp/cprops-1483938858641-2232644763732980231.tmp" |
$ cat /tmp/cprops-1483938858641-2232644763732980231.tmp answer=42 source.account.rabbit.host=127.0.0.1 source.account.rabbit.port=5672 source.account.rabbit.vhost=/z-broker source.account.rabbit.username=guest source.account.rabbit.password=guest datomic.url=datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic |
EDN to ENV
(require '[cprop.tools :as t]) (t/map->env-file config) |
Converts config map into a file with ENV variable exports, saves the file under temp directory and returns a path to it.
For example, let’s say we have a map m:
{:datomic {:url "datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic"}, :source {:account {:rabbit {:host "127.0.0.1", :port 5672, :vhost "/z-broker", :username "guest", :password "guest"}}}, :answer 42} |
(t/map->env-file m) |
would convert it to a property file and would return an OS/env specific path to it, in this case:
"/tmp/cprops-1483939362242-8501882574334641044.tmp" |
$ cat /tmp/cprops-1483939362242-8501882574334641044.tmp export ANSWER=42 export SOURCE__ACCOUNT__RABBIT__HOST=127.0.0.1 export SOURCE__ACCOUNT__RABBIT__PORT=5672 export SOURCE__ACCOUNT__RABBIT__VHOST=/z-broker export SOURCE__ACCOUNT__RABBIT__USERNAME=guest export SOURCE__ACCOUNT__RABBIT__PASSWORD=guest export DATOMIC__URL=datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic |
notice the double underscores to preserve the original map’s hierarchy.
.properties to one level EDN
(require '[cprop.source :as cs]) (cs/slurp-props-file "/path/to/some.properties") |
Besides the from-props-file function that converts .properties file to a map with hierarchy, there is also a slurp-props-file function that simply converts a property file to a map without parsing values or building a hierarchy.
For example this “solar-system.properties” file:
## solar system components components=sun,planets,dwarf planets,moons,comets,asteroids,meteoroids,dust,atomic particles,electromagnetic.radiation,magnetic field star=sun ## planets with Earth days to complete an orbit planet.mercury.orbit_days=87.969 planet.venus.orbit_days=224.7 planet.earth.orbit_days=365.2564 planet.mars.orbit_days=686.93 planet.jupiter.orbit_days=4332.59 planet.saturn.orbit_days=10755.7 planet.uran.orbit_days=30688.5 planet.neptune.orbit_days=60148.35 ## planets natural satellites planet.earth.moons=moon planet.jupiter.moons=io,europa,ganymede,callisto planet.saturn.moons=titan planet.uran.moons=titania,oberon planet.neptune.moons=triton # favorite dwarf planet's moons dwarf.pluto.moons=charon,styx,nix,kerberos,hydra |
by
(cs/slurp-props-file "solar-system.properties") |
would be converted to a “one level” EDN map:
{"star" "sun", "planet.jupiter.moons" "io,europa,ganymede,callisto", "planet.neptune.moons" "triton", "planet.jupiter.orbit_days" "4332.59", "planet.uran.orbit_days" "30688.5", "planet.venus.orbit_days" "224.7", "planet.earth.moons" "moon", "planet.saturn.orbit_days" "10755.7", "planet.mercury.orbit_days" "87.969", "planet.saturn.moons" "titan", "planet.earth.orbit_days" "365.2564", "planet.uran.moons" "titania,oberon", "planet.mars.orbit_days" "686.93", "planet.neptune.orbit_days" "60148.35" "dwarf.pluto.moons" "charon,styx,nix,kerberos,hydra", "components" "sun,planets,dwarf planets,moons,comets,asteroids,meteoroids,dust,atomic particles,electromagnetic.radiation,magnetic field"} |