"; */ ?>
clojure — Comments Off on Disturbance in The Force
10
Jan 16

Disturbance in The Force

I’d like to lay out a couple of thoughts here to discuss further to get more insight from people with different mindsets / opinions / facts / views. Reasons are mostly selfish: I know what I don’t know and I’d like to understand it better.

Folds and Braids

As Clojure developers we are in this “Cult of Simple”. Simple “was defined to us” :) as one fold/braid/twist, which does not mean “one thing”, and is really about the interleaving, not the cardinality. We also know that simple is “objective”, we can look at it and see the “number of folds”.

I don’t think it’s such a bad cult to be in, but.. It seems that the above “simple definition of simple” is now taken to an extreme where, as definition, it is interleaved / complected with real problems and is losing its intended power.

The definition of simple is now used as a shield, rather than a tool.

It might have to do with the very subjective definition of “one fold”. It is easier to understand what “one fold” is when thinking about “primitives”: i.e. Rich’s example of “Sets vs. Lists”, where Sets are simpl(er) since Lists introduce order. But it is not as clear what “one fold” is in a more “complected problems” whether these are business problems or tool libraries.

Keep Your Functions Close, but “Just In Case” Closer

Clojure protocols are super powerful, and I would say “simple”.

I don’t think Clojure records are powerful, but good (in my experience) for type driven polymorphism, and they hold fields a bit more efficiently than maps. In reality “type driven polymorphism” would be the only reason I would use them for.

If “type driven polymorphism” is all that’s needed I would first reach out to deftype instead of records, since records complect data with types: two folds :)

However, since intuitively, building solutions with protocols feels a lot more extensible, robust and flexible, I think they get applied where a simple set of functions over multiple namespaces should have been used instead. This problem, I think, is caused by the invisible seduction of “easy”, which is defined to us as “lie near”:

“since my solution should be robust and extensible, I’ll use records and protocols, since I know they will make it so“.

In other words, for initial application design: “records and protocols lie near”.

That is not to say that protocols or types should be avoided, quite the contrary, dynamic type dispatch, libraries with (internal) abstractions, host interop: all great cases to use and love them.

But I don’t think it is wise, in the Clojure Universe, to make people create records / types and use protocols when they need to use your library to develop products. This is not the absolute truth, but for most cases, when developing a business application, I would rather use a Clojure function: one fold :)

Humor Driven Development

I spend a couple of years working in Scala. An implicit type in Scala was one of the most common causes of confusion. It is used everywhere internally in the language itself, as well as advocated to use in every day Scala programs. This and many other examples, teach us that “implicit” is “complex”. The flip side of that is where the problem lies. In Clojure “formal circles” the “inferential” belief is that “explicit” is “simple”.

I believe that neither implicit nor explicit can be applied to simple without a context. And no, a “well implied implicit” does not mean complex. And no, explicit does not mean simple.

For example, I need to create a local scope and bind some values: if I am given a choice to write an identity monad or to use a Clojure let binding, I would choose the let binding, because it is a great syntactic implicit. An identity monad would also work, but being explicit here does not buy me any simplicity.

An interesting quality of a good implicit, by the way, is “automatic” understanding of what’s implied: that’s how we laugh :)

Respect and Doubts

This brings me to the overuse of “explicit formalism” in Clojure. On one hand it cannot be subjectively complex, since we know that “simple” is objective. On the other hand it can, because a “single fold” can be defined very differently in the explicitly formal solution by me and by people who created it.

I say: let’s listen to each other, rather than teach and preach.

clojure — Comments Off on Clojure Mindsets
07
Jan 16

Clojure Mindsets

After Dan’s post, we had a great discussion on reddit that was not a religious discussion, but a “civil and thought-provoking” discussion. And that, in my mind, is the greatest power of Clojure.

And The Winner is…

Yes, we have not established which is better mount or component, but that was never the point. The great summary of the discussion was a realization that @yogthos and @weavejester made at the end:

“Overall, it sounds like both approaches are actually fairly similar: Mount uses namespaces and vars where Component uses records and protocols, but the purpose is broadly the same.”

This is interesting for two major reasons:

* It advocates the proper usage of Component with using protocols over records/components

* It draws a clear parallel between Mount and Component, so it is a lot easier to choose which one fits your coding style / (formal vs. simple) mindset.

Libraries vs. Frameworks

Since I am obviously biased towards Mount :)..

Component manages protocols and records, and in order to do that it requires a whole app buyin, which makes it a framework. Mount does not need to manage namespaces and vars, since it is very well managed by the Clojure Compiler. Which makes it a library, which I prefer. You may not, and I respect that.

Math vs. Music

Another thing I keep noticing is that “formally” inclined people prefer records and protocols for many solutions, and for them it is easier to reason about applications since it fits better into the formal mindset.

What I disagree with is equating “more formal” with simpler. It is simpler for a formal mindset, yes, but it is not simpler for me. Explicit formalism complects things for my mindset. I do not believe there is an objective truth here, since we are just different. But I believe Clojure has both: which allows for both mindsets to coexist.

What I like protocols for is an optimized dynamic dispatch, creating libraries (which use protocols internally), and a way to tame the expression problem if and when it arises. Protocols make it simple for me.

“Frogs All Look The Same”

And to make it more interesting, I am currently on the train going back to Philly from the great NYC Lisp meetup, where Gerald Sussman presented a way to write systems that evolve and expand to handle any future functionality that is thrown at them (“extensible generics”).

He drew an interesting parallel between software systems and biology saying that “many biological mutations are fatal, but as a result we end up with extremely robust systems“, doesn’t sound very formal to me ;)

clojure — Comments Off on Don’t Fear: The Quite Global Mutable State
07
Jan 16

Don’t Fear: The Quite Global Mutable State

Global Mutable State is bad.
Global Mutable State is bad.
Global Mutable State is bad.

But which state and in what context?

Your Oracle is Bad

Do you use a database? Because if you do, it is a Global Mutable State. Is it bad? I don’t think so, it’s very useful.

The mutability concept is “bent” by databases like Datomic since it introduces a time dimension, but even in Datomic, for most cases, you only need to know “the latest” value, and it is going to mutate. Yes, over time, but nevertheless, for an application that cares about a stock price: it mutates. This is just how we understand the world: it is our own Global State and it Mutates over time, and it’s great!

Your Oracle Connection is Bad

Do you ever connect to a database? Because if you do, it is a Global Mutable State. Oh, wait, it should not be global, it should be well encapsulated and only used in functions / components / classes / modules / local bindings that need it right? We should narrow it down as much as possible.

All correct. But… narrow it down all you want, if you rely on this database connection in 42 places in your codebase and this connection dies, it affects all 42 places, no matter how well “localized” they are. So why not embrace this fact and just admit that this is a Global Mutable State: e.g. you can call connection.disconnect() in any of these 42 places, and it is going to affect all the rest. If it is easier, we can call it a Quite Global Mutable State.

Your Configuration is Bad

Do you rely on any configuration files that you load from Zookeper / etcd / file system? Ah.. but they are not mutable. Ah.. but they are.

Say you support reloading these configs at runtime, a pretty common pattern, and quite useful for certain systems. Every time you mutate it and reload, it affects all components of an application that rely on it. “Yes, but don’t put it as a top level mutable reference, so a beginner programmer can mutate it by mistake, and then it is not easy to reason about it”… Ok, somewhat a fair point => you can still keep it as a global reference, if you need, but point it to an immutable map: done.

Your Salary is Bad

Some questions with the same answers as above:

* Is your network socket connection bad?
* Is your global threadpool bad?
* Is your queue bad?
* …

But this is not to say that any state is naturally a Global Mutable State, or that we, as programmers should not worry about it, quite the contrary, we should take a great care about the “business”/”problem” state.

Say you have an application that deducts taxes from peoples’ salaries, and does so concurrently. If you have your “next salary” as a Global Mutable State, you might end up borrowing for the mortgage this month.. computer error, oops.

The point is: resources, I/O, external storage, etc.. are all examples of the real Quite Global Mutable State. Yes, this state can be hidden from certain components, and only available in some, but its change(s) immediately affect all the components that rely on it, which is “quite global” as the state goes.

You are Bad

You change over time, well I do. We evolve and mutate. Is it bad? Should we be caged into the smallest possible cages so it is easier to reason about us? :) I don’t think so. Ok, so instead should we expose ourselves to anybody who can hurt us? Also, no. We choose how to present ourselves to the Universe, so why can’t we have the same choices in software? We are smart, we can do this!

boot / clojure / clojurescript / lein / mount — Comments Off on The Story of Booting Mount
22
Dec 15

The Story of Booting Mount

Feeling The Code

I don’t agree with the opinion that “cool kids now use boot“. People who say that are just missing out on the power of “feeling the code” rather than being abstracted from the code by a “better XML”. Same deal with people 10 years ago who said “cool kids are using functional languages”.

Don’t get me wrong I like lein a lot. It is simple to start with, it is well documented, it is very googlable, it is sharing platform (i.e. templates), mature, etc.. But boot is very different, it does not aim to do what lein does, it aims to do “what you want”. There is a difference.

Mounting a Bootable Partition

Since the late 90s when I got in to Linux, I found bootable partitions most exciting, they actually bootstrap everything, they were these wizards waving their magic wands and systems appeared. Granted the wave could take minutes, but we are humans, we always wait for the magic, even if it takes the whole life.

First thing that needs to be done for the magic to happen, this bootable partition needs to be mounted.

I wanted to do it for some time now, when, I could not figure out why ClojureScript brought in as a dependency with :classifier “aot” caused compilation problems with lein/cljsbuild, David Nolen suggested that this is rather due to the lein environment issues. So 2 and 2 together: it was the right time to “boot” myself up.

And since the partition was already mounted it was ready to boot.

Grokking the New Simple

Rather than tell you how great boot is, I’ll share non obvious (to me) things that I stumbled upon converting mount from lein to boot. Let’s rock & roll:

REPL is just REPL

Since I needed a support for both Clojure and ClojureScript, I looked at many examples and noticed a pattern: usually in a dev mode one task groups several, where most of the examples have a (watch) task in that group.

I just wanted to start out, so I decided that at a minimum I need a REPL and (I guess) this watcher to be able to mimic the lein repl behavior, so I did:

(deftask dev [] 
  (comp
    (watch)
    (repl)))

And it worked! I ran boot dev and I got a REPL which would see all the updates from vim (via the updated vim-fireplace).

But then I decided to stop the REPL, and it just froze.. I ran jstack on the PID and saw lots of watcher threads locking and derefing futures. Ok, so that’s not a good combination.

The answer is simpler than I expected: it’s just boot repl. Nothing else is needed to get to the lein repl functionality.

“Bring on Your Own Data Readers” Party

The Clojure mount example app uses in memory Datomic, so when I tried to start the app, boot told me:

no reader to handle the #db/id tag

This was easily googlable, and revealed that boot has a (load-data-readers!) function that “refreshes *data-readers* with readers from newly acquired dependencies”.

An interesting bit here is that (load-data-readers!) can’t be a part of a “top level” task that is executed with boot since:

java.lang.IllegalStateException: Can't set!: *data-readers* from non-binding thread

So calling boot dev, in case “load-data-readers!” is there, is not an option. But getting into a REPL “boot repl“, and then calling (dev) works beautifully.

REPL Logging

At this point I could get into the boot REPL and start the mount example app. A slight problem was that I did not see any logging from the app within the REPL.

That’s when I found boot-logservice that brought the logging back to the REPL:

(def log4b
  [:configuration
   [:appender {:name "STDOUT" :class "ch.qos.logback.core.ConsoleAppender"}
    [:encoder [:pattern "%-5level %logger{36} - %msg%n"]]]
   [:root {:level "TRACE"}
    [:appender-ref {:ref "STDOUT"}]]])
;; ...
 
(deftask dev []
 
  ;; ...
 
  (alter-var-root #'log/*logger-factory* 
                  (constantly (log-service/make-factory log4b)))
  ;; ... 
)
Shaking up tools.namespace

While it is not a requirement, and most of the time unnecessary, the example app uses tools.namespace to make it easier for people who rely on it heavily to get into mount.

By default “tools.namespace” won’t find anything to refresh, since boot uses its own “secret” temp directories for sources, and “tools.namespace” simply does not know about them.

This was an easy one, since it is well documented by boot. Hence having (apply set-refresh-dirs (get-env :directories)) in the “dev” task pointed “tools.namespace” to the right directories to refresh.

The Joy of Deploy: Build and Publish

At this point having the Clojure part figured out, before moving to the ClojureScript support, I decided to deploy mount to Clojars, to understand how it’s done with boot.

I found bootlaces, and just plugged it in, it was very straightforward:

(def +version+ "0.1.7-SNAPSHOT")
 
(bootlaces! +version+)
 
;; other things.. and
 
(task-options!
  pom {:project     'mount
       :version     +version+
       :description "managing Clojure and ClojureScript app state since (reset)"
       :url         "https://github.com/tolitius/mount"
       :scm         {:url "https://github.com/tolitius/mount"}
       :license     {"Eclipse Public License"
                     "http://www.eclipse.org/legal/epl-v10.html"}})

Then I did:

boot build-jar push-snapshot

and everything was going smoothly, it asked for my Clojars username, then password.. but then:

clojure.lang.ExceptionInfo: java.lang.AssertionError: 
Assert failed: current git branch is 0.1.7 but must be master
               (or (not ensure-branch) (= b ensure-branch))

Boot told me that it prefers publishing snapshots from the “master”. I don’t disagree, but for some projects I like snapshots from version branches. I don’t really like “git flow”, I like “git freedom”.

Looking at the bootlaces code it seems that “master” is hardcoded. By this time I already started to feel the concept of a “boot task” and noticed that it is hardcoded under the “push” internal task, which means that this task’s options can potentially be overridden:

;; ...
 
(task-options!
 
  push {:ensure-branch nil}       ;; <<<<<<<<<<
 
  pom {:project     'mount
       :version     +version+
       ;; ... 
       })

And what d’you know, it worked! This was most likely the first “aha moment” which wired some of my neurons in boot ways.

Shall Not Pass!

Mount’s “test” root has both cljc tests and clj/cljs test apps that these tests use. The structure looks similar to:

|~test/
| |~clj/...
| | `+tapp/
| |~cljs/...
| | `+tapp/
| |~mount/
| | |+test/...
| | `-test.cljc

In lein, I can give “test” + “test/clj” for Clojure tests, and “test” + “test/cljs” for ClojureScript tests as the sources paths.

In boot I can’t do that, boot says:

java.lang.AssertionError: Assert failed: 
The :source-paths, :resource-paths, and :asset-paths must not overlap.
    (empty? (set/intersection paths parents))

Since boot already read everything under “test”, it does not want to merge things from “test/clj”. Fair enough, so I had to change the structure a bit to make it work:

|~test/
| |~clj/
| | `+tapp/
| |~cljs/
| | `+tapp/
| |~core/
| | `~mount/
| |   |+test/
| |   `-test.cljc

Now I can give “test/core” + “test/clj” and “test/core” + “test/cljs” respectively.

ClojureScript is Clojure, but.. not Always

ClojureScript took some time to get right. Many examples helped a lot especially these three: boot-cljs-example, tenzing and boot-cljs-multiple-builds.

The concept of dividing “cljs” options between “xyz.cljs.edn” and “task options” did not sink in immediately, and required some code digging to figure out where to put what and how to make sure it is being used.

It ends up to be quite simple. Options that are provided via “xyz.cljs.edn” can be referenced from task options via ids option:

(cljs :optimizations :advanced :ids #{"mount"})

would mean that it would look for mount.cljs.edn file within the classpath. That file should point to the entry point of the ClojureScript app. In case of the mount example app it would just be:

{:require  [app.example]}

where init-fns and compiler-options can be also added.

Testing ClojureScript

“mount does doo” for ClojureScript testing, and boot-cljs-test does it as well.

I would expect it to pick up “xyz.cljs.edn” files in the same way as “boot-cljs”, but it does it a bit differently. It is not all that obvious at first, but looking at the code I saw that it has a different name for ids, it calls it out-id. It also does not just take an “id”, it takes an “id” + “.js”, as I saw from the code.

So to get it to work is quite simple:

(tcs/test-cljs :out-file "mount.js"))

which would look for the same mount.cljs.edn file within the classpath.

Power it Up

There were other discoveries, like

* tasks are functions, but not really, they take arguments in the particular format and they better return a fileset

* tasks: “comp us please”. They like to be (comp ..)ed. Otherwise no go.

* there were others, but I liked Pods the most.

At this point I got all up and pumping, deployed to CircleCi using boot to build and run tests, published to Clojars as snapshot and release, etc.

One of the greatest things that I loved while debugging dependencies is boot show -p, it’s amazing!

Get up! Boot yourself up! Enjoy the runtime!

clojure / scala — Comments Off on Functional Programming for Humans
21
Dec 15

Functional Programming for Humans

A couple of years ago, maybe even three, we had our Chariot Day. It’s a conference within our small company where we, developers, talk to us developers.

Not everyone at that point did functional programming, and it was fun to go over the ultimate FP power and some of the “why”s.

So Mr. Dan and I sat down and created a talk to lure “non functional” people in. We talked about assignment, concurrency, equality, thinking in sequences, took over the enterprise by writing Yelp and Trading Forecast in both: imperative and functional style, took on some design patterns, etc..

We had great discussions during the talk, but years went by and it remained to be internal. This is now fixed:

← Older Entries
Newer Entries →