cprop: internal tools worth opening

Most of the tools I push to github are created out of something that I needed at the moment but could not find a good alternative for. cprop was one of such libraries. It sat there on github all alone for quite some time, and was used only by several people on my team, until it was integrated into Luminus.

Suddenly I started talking to many different people who found flaws in it, or just wanted to add features. I learned a couple of interesting usages from Heroku guys, as well as the importance of merging creds with Vault, coexisting with configs from other fault tolerant and external services such as Consul and more.

One of the useful cprop features is merging configs from various sources. Which is quite an open extension point: i.e. once cprop does its work and comes up with an app config, you can decide how and what will be merged with it before it really becomes a thing. It can be a local map, a .properties file, more ENV vars, more system properties, more configs from anywhere else, including remote/external resources, result from which can be converted to an EDN map.

To enable this merge extension point cprop has several tools that in practice could be really useful on its own: i.e. can be used outside of the (load-config) scope.

Loading from various sources

Could be used as OS, file system and edn oriented I/O tools. Also quite useful in the REPL.

Loading form a classpath

(require '[cprop.source :as cs])
 
(cs/from-resource "path/within/classpath/to-some.edn")

Loads an EDN file anywhere from within a classpath into a Clojure map.

Loading from a file system

(require '[cprop.source :as cs])
 
(cs/from-file "/path/to/something.edn")

Loads an EDN file from a file system with an absolute path into a Clojure map.

Loading from system properties

(require '[cprop.source :as cs])
 
(cs/from-system-props)

Loads all the system properties into a Clojure map. i.e. all the properties that are set with
-Dkey=value, or programmatically set with System.setProperty(key, value), etc.

System properties are usually separated by . (periods). cprop will convert these periods to - (dashes) for key separators.

In order to create a structure in the resulting EDN map use _ (an underscore).

For example:

-Dhttp_pool_socket.timeout=4242

or

System.setProperty("http_pool_socket.timeout" "4242");

will be read into:

{:http
 {:pool
  {:socket-timeout 4242}}}

notice how . was used as - key separator and _ was used to “get-in”: i.e. to create a hierarchy.

Loading from OS (ENV variables)

(require '[cprop.source :as cs])
 
(cs/from-env)

Loads all the environment variables into a Clojure map.

ENV variables lack structure. The only way to mimic the structure is via use of an underscore character. The _ is converted to - by cprop, so instead, to identify nesting, two underscores can be used.

For example:

export HTTP__POOL__SOCKET_TIMEOUT=4242

would be read into:

{:http
 {:pool
  {:socket-timeout 4242}}}

Notice how two underscores are used for “getting in” and a single underscore just gets converted to a dash as a key separator. More about it, including type inference, in the docs

Loading from .properties files

(require '[cprop.source :as cs])
 
(cs/from-props-file "/path/to/some.properties")

Loads all the key value pairs from .properties file into a Clojure map.

The traditional syntax of a .properties file does not change. For example:

  • . means structure

four.two=42 would be translated to {:four {:two 42}}

  • _ would be a key separator

fourty_two=42 would be translated to {:forty-two 42}

  • , in a value would be a seq separator

planet.uran.moons=titania,oberon would be translated to {:planet {:uran {:moons ["titania" "oberon"]}}}

For example let’s take a solar-system.properties file:

## solar system components
components=sun,planets,dwarf planets,moons,comets,asteroids,meteoroids,dust,atomic particles,electromagnetic.radiation,magnetic field
 
star=sun
 
## planets with Earth days to complete an orbit
planet.mercury.orbit_days=87.969
planet.venus.orbit_days=224.7
planet.earth.orbit_days=365.2564
planet.mars.orbit_days=686.93
planet.jupiter.orbit_days=4332.59
planet.saturn.orbit_days=10755.7
planet.uran.orbit_days=30688.5
planet.neptune.orbit_days=60148.35
 
## planets natural satellites
planet.earth.moons=moon
planet.jupiter.moons=io,europa,ganymede,callisto
planet.saturn.moons=titan
planet.uran.moons=titania,oberon
planet.neptune.moons=triton
 
# favorite dwarf planet's moons
dwarf.pluto.moons=charon,styx,nix,kerberos,hydra
(cs/from-props-file "solar-system.properties")

will convert it to:

{:components ["sun" "planets" "dwarf planets" "moons" "comets"
              "asteroids" "meteoroids" "dust" "atomic particles"
              "electromagnetic.radiation" "magnetic field"],
 :star "sun",
 :planet
 {:uran {:moons ["titania" "oberon"],
         :orbit-days 30688.5},
  :saturn {:orbit-days 10755.7,
           :moons "titan"},
  :earth {:orbit-days 365.2564,
          :moons "moon"},
  :neptune {:moons "triton",
            :orbit-days 60148.35},
  :jupiter {:moons ["io" "europa" "ganymede" "callisto"],
            :orbit-days 4332.59},
  :mercury {:orbit-days 87.969},
  :mars {:orbit-days 686.93},
  :venus {:orbit-days 224.7}},
 :dwarf {:pluto {:moons ["charon" "styx" "nix" "kerberos" "hydra"]}}}

Converting for other sources

Most Java apps store their configs in .properties files. Most docker deployments rely on ENV variables. cprop has some open tools it uses internally to work with these formats to bring EDN closer to non EDN apps and sources.

EDN to .properties

(require '[cprop.tools :as t])
 
(t/map->props-file config)

Converts config map into a .properties file, saves the file under temp directory and returns a path to it.

For example, let’s say we have a map m:

{:datomic
 {:url
  "datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic"},
 :source
 {:account
  {:rabbit
   {:host "127.0.0.1",
    :port 5672,
    :vhost "/z-broker",
    :username "guest",
    :password "guest"}}},
 :answer 42}
(t/map->props-file m)

would convert it to a property file and would return an OS/env specific path to it, in this case:

"/tmp/cprops-1483938858641-2232644763732980231.tmp"
$ cat /tmp/cprops-1483938858641-2232644763732980231.tmp
answer=42
source.account.rabbit.host=127.0.0.1
source.account.rabbit.port=5672
source.account.rabbit.vhost=/z-broker
source.account.rabbit.username=guest
source.account.rabbit.password=guest
datomic.url=datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic

EDN to ENV

(require '[cprop.tools :as t])
 
(t/map->env-file config)

Converts config map into a file with ENV variable exports, saves the file under temp directory and returns a path to it.

For example, let’s say we have a map m:

{:datomic
 {:url
  "datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic"},
 :source
 {:account
  {:rabbit
   {:host "127.0.0.1",
    :port 5672,
    :vhost "/z-broker",
    :username "guest",
    :password "guest"}}},
 :answer 42}
(t/map->env-file m)

would convert it to a property file and would return an OS/env specific path to it, in this case:

"/tmp/cprops-1483939362242-8501882574334641044.tmp"
$ cat /tmp/cprops-1483939362242-8501882574334641044.tmp
export ANSWER=42
export SOURCE__ACCOUNT__RABBIT__HOST=127.0.0.1
export SOURCE__ACCOUNT__RABBIT__PORT=5672
export SOURCE__ACCOUNT__RABBIT__VHOST=/z-broker
export SOURCE__ACCOUNT__RABBIT__USERNAME=guest
export SOURCE__ACCOUNT__RABBIT__PASSWORD=guest
export DATOMIC__URL=datomic:sql://?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic

notice the double underscores to preserve the original map’s hierarchy.

.properties to one level EDN

(require '[cprop.source :as cs])
 
(cs/slurp-props-file "/path/to/some.properties")

Besides the from-props-file function that converts .properties file to a map with hierarchy, there is also a slurp-props-file function that simply converts a property file to a map without parsing values or building a hierarchy.

For example this “solar-system.properties” file:

## solar system components
components=sun,planets,dwarf planets,moons,comets,asteroids,meteoroids,dust,atomic particles,electromagnetic.radiation,magnetic field
 
star=sun
 
## planets with Earth days to complete an orbit
planet.mercury.orbit_days=87.969
planet.venus.orbit_days=224.7
planet.earth.orbit_days=365.2564
planet.mars.orbit_days=686.93
planet.jupiter.orbit_days=4332.59
planet.saturn.orbit_days=10755.7
planet.uran.orbit_days=30688.5
planet.neptune.orbit_days=60148.35
 
## planets natural satellites
planet.earth.moons=moon
planet.jupiter.moons=io,europa,ganymede,callisto
planet.saturn.moons=titan
planet.uran.moons=titania,oberon
planet.neptune.moons=triton
 
# favorite dwarf planet's moons
dwarf.pluto.moons=charon,styx,nix,kerberos,hydra

by

(cs/slurp-props-file "solar-system.properties")

would be converted to a “one level” EDN map:

{"star" "sun",
 
 "planet.jupiter.moons" "io,europa,ganymede,callisto",
 "planet.neptune.moons" "triton",
 "planet.jupiter.orbit_days" "4332.59",
 "planet.uran.orbit_days" "30688.5",
 "planet.venus.orbit_days" "224.7",
 "planet.earth.moons" "moon",
 "planet.saturn.orbit_days" "10755.7",
 "planet.mercury.orbit_days" "87.969",
 "planet.saturn.moons" "titan",
 "planet.earth.orbit_days" "365.2564",
 "planet.uran.moons" "titania,oberon",
 "planet.mars.orbit_days" "686.93",
 "planet.neptune.orbit_days" "60148.35"
 
 "dwarf.pluto.moons" "charon,styx,nix,kerberos,hydra",
 
 "components" "sun,planets,dwarf planets,moons,comets,asteroids,meteoroids,dust,atomic particles,electromagnetic.radiation,magnetic field"}