HBase Cookies
“Big data” is this great old buzz that no longer surprises anyone. Even politicians used it way back in 2012 to win the elections.
But the field became so large and fragmented that simple CRUD operations, that we used to just take for granted before, now require the whole new approach depending on which data store we have to deal with.
This short tale is about a tiny little feature in the universe of HBase. Namely a “delete by” feature. Which is quite simple in Datomic or SQL databases, but it is not that trivial in HBase due to the way its cookie crumbles.
Delete it like you mean it!
There is often a case where rows need to be deleted by a filter, that is similar to the one used in scan (i.e. by row key prefix, time range, etc.) HBase does not really help there besides providing a BulkDeleteEndpoint coprocessor.
This is not ideal as it delegates work to HBase “stored procedures” (effectively this is what coprocessors are). It really pays off during massive data manipulation, since it does happen directly on the server, but in simpler cases, which are many, coprocessors are less than ideal.
cbass achieves “deleting by anything” by a trivial flow: “scan + multi delete” packed in a “delete-by” function which preserves the “scan“‘s syntax:
user=> (scan conn "galaxy:planet") {"earth" {:age "4.543 billion years", :inhabited? true, :population 7125000000}, "neptune" {:age "4.503 billion years", :inhabited? :unknown}, "pluto" {}, "saturday" {:age "24 hours", :inhabited? :sometimes}, "saturn" {:age "4.503 billion years", :inhabited? :unknown}} |
user=> (delete-by conn "galaxy:planet" :from "sat" :to "saz") ;; deleting [saturday saturn], since they both match the 'from/to' criteria |
look ma, no saturn, no saturday:
user=> (scan conn "galaxy:planet") {"earth" {:age "4.543 billion years", :inhabited? true, :population 7125000000}, "neptune" {:age "4.503 billion years", :inhabited? :unknown}, "pluto" {}} |
and of course any other criteria that is available in “scan” is available in “delete-by”.
Delete Row Key Function
Most of the time HBase keys are prefixed (salted with a prefix). This is done to avoid “RegionServer hotspotting“.
“delete-by” internally does a “scan” and returns keys that matched. Hence in order to delete these keys they have to be “re-salt-ed” according to the custom key design.
cbass addresses this by taking an optional delete-key-fn, which allows to “put some salt back” on those keys.
Here is a real world example:
;; HBase data user=> (scan conn "table:name") {"���|8276345793754387439|transfer" {...}, "���|8276345793754387439|match" {...}, "���|8276345793754387439|trade" {...}, "�d\k^|28768787578329|transfer" {...}, "�d\k^|28768787578329|match" {...}, "�d\k^|28768787578329|trade" {...}} |
a couple observations about the key:
- it is prefixed with salt
- it is piped delimited
In order to delete, say, all keys that start with 8276345793754387439, besides providing :from and :to, we would need to provide a :row-key-fn that would de salt and split, and then a :delete-key-fn that can reassemble it back:
(delete-by conn "table:name" :row-key-fn (comp split-key without-salt) :delete-key-fn (fn [[x p]] (with-salt x p)) :from (-> "8276345793754387439" salt-pipe) :to (-> "8276345793754387439" salt-pipe+)))) |
*salt, *split and *pipe functions are not from cbass, since they are application specific. They are here to illustrate the point of how “delete-by” can be used to take on the real world.
;; HBase data after the "delete-by" user=> (scan conn "table:name") {"�d\k^|28768787578329|transfer" {...}, "�d\k^|28768787578329|match" {...}, "�d\k^|28768787578329|trade" {...}} |