Handling retries and validation of HTTP requests with missionary

{:deps {org.clojure/clojure {:mvn/version "1.10.3"}
        missionary/missionary {:mvn/version "b.46"}
        hato/hato {:mvn/version "1.0.0"}
        org.clojure/tools.logging {:mvn/version "1.3.0"}
        ;; complient is used for autocompletion
        ;; add your libs here (and restart the runtime to pick up changes)
        compliment/compliment {:mvn/version "0.3.9"}}}
Extensible Data Notation

In the previous post, I mentioned that I would discuss how to use leonoel/missionary together with the decorator pattern to keep HTTP request handling both efficient and manageable, even in the presence of retries and validation errors.

Given how quickly things move in software development, that post feels even older than it actually is. However, I still find the handling of retries and response validation to be both simple and elegant. I focus on the structure rather than a step-by-step walkthrough, and leaves some details implicit.

So what did I mean in my last post by "high performance and reduced complexity" in the context of HTTP requests?

  1. High performance means issuing as many requests in parallel as possible while using a minimal amount of resources.

  2. Reduced complexity means keeping responsibilities separate unless there is a strong reason to combine them.

Missionary’s asynchronous processes support efficient parallelism, while the decorator pattern keeps the structure flexible by allowing configurable behavior without entangling unrelated concerns.

Implementation

The first step is defining the namespace and introducing a small helper macro for default values:

(ns me.lomin
  (:require
   [clojure.tools.logging :as log]
   [hato.client :as hc]
   [missionary.core :as m])
  (:import
   (java.util.concurrent Future)))
(def DEFAULT ::default)
(defmacro either [either-form or-form]
  `(let [result# ~either-form]
     (if (= result# DEFAULT)
       ~or-form
       result#)))
9.8s

Next, Hato is combined with missionary to create asynchronous, non-blocking requests:

(defprotocol Request
  (create [self opts]))
(defn request-async [client Request]
  (fn [success! failure!]
    (let [fut (hc/request (create Request {:http-client client
                                           :throw-exceptions? false
                                           :async?      true})
                          success!
                          failure!)]
      #(.cancel ^Future fut true))))
0.0s

Hato is configured not to throw exceptions on HTTP errors. This allows error handling to be managed explicitly and makes it possible to validate additional aspects of a response, such as schema correctness. Validation is implemented as a decorator over missionary tasks:

(defn with-validation
  "simple validation on status only" 
  [task]
  (m/sp
   (let [response (m/? task)
         status (:status response)]
     (cond (= status 200) [status (:body response)]
           :else (throw (ex-info "non-200-status"
                                 {:type :non-200-status
                                  :status status}))))))
0.3s

When an error occurs, retries are needed, but only up to a certain limit and with configurable delays. This behavior is added through another decorator:

(defprotocol Retry
  (delays [self])
  (error [self exception retry-left?])
  (exit [self]))
(defn with-retry [Retry task]
  (m/sp
   (loop [delay-seq (either (delays Retry) [1000])]
     (if-some [response (try (m/? task)
                             (catch Exception e
                               (either (error Retry e (seq delay-seq))
                                       (log/error (ex-data e)))))]
       response
       (if-some [[d & ds] delay-seq]
         (do (m/? (m/sleep d))
             (recur ds))
         (either (exit Retry) (log/info "Delays exhausted")))))))
0.6s

Different retry strategies can then be defined independently:

(defrecord ExponentialBackoff []
  Retry
  (delays [self] [1000 2000 4000 8000])
  (error [self exception retry-left?] DEFAULT)
  (exit [self] DEFAULT))
(defrecord RetryForever []
  Retry
  (delays [self] (repeat 1000))
  (error [self exception retry-left?] DEFAULT)
  (exit [self] DEFAULT))
0.6s

The following demonstrates how the components can be composed:

(def client (hc/build-http-client {}))
(defrecord RandomRequest [urls]
  Request
  (create [_self opts]
    (assoc opts :url (rand-nth urls))))
(defn request [req]
  (->>
    (request-async client (map->RandomRequest req))
    (with-validation)
    (with-retry (->RetryForever))))
(defn pull [par reqs]
  (m/reduce conj
    []
    (m/ap
      (let [req (m/?> par (m/seed reqs))]
               (m/? (request req))))))
(defn random-reqs [cnt]
  (repeat cnt {:urls ["https://httpbin.org/status/200"
                      "https://httpbin.org/status/500"]}))
(m/? (pull 4 (random-reqs 4)))
2.2s
Runtimes (1)