From Scratch - Clojure web app from the ground up

This is your first activity. Write code to import your database credentials from an EDN file, instead of hard-coding them. Then you can move your datasource and connection out of your rich comment. Don't forget to put your EDN file in .gitignore

When you're finished, here's how I did it:

✎ changes

Your second activity is to implement Integrant for our project. To do this you will need to consume the pertinent learning materials:

• the project README
• the video

Define methods for:

• reading our credentials file
• starting and stopping our database connection
• creating our handler
• starting and stopping our webserver

Define Integrant config, using references to indicate the dependencies between components.

(i) I know our handler doesn't actually use the database object yet, so your handler function strictly doesn't need to be created by Integrant. But of-course it will so just stub it out for now.

When you're finished, here's how I did it:

✎ changes

So that we can compile our app into something deployable, we need a -main function which will use integrant to start our application. Then we can use tools.build to compile it.

Update your code with these three functions to start your app, and handle the shutdown logic:

(defn system-config []
  (ig/read-string (slurp "resources/system.edn")))

(defn halt-system [system]
  (ig/halt! system))

(defn -main [& args]
  (let [system (ig/init (system-config))]
    (.addShutdownHook (Runtime/getRuntime)
                      (Thread. #(halt-system system)))))

In addition to having a -main function, one more thing we need is to add (:gen-class) to your namespace declaration. Like this:

(ns polls.core
  (:require ...)
  (:gen-class))

Now you need to add an alias to your deps.edn. Do that following the guide here.

Now finally create a build.clj in your top-level directory (along-side your deps.edn) and paste in the example from the guide uberjar example.

Fix that in the three places it refers to my/lib1 or my.lib.main to just polls and polls.core.

Now you may build your app with this command:

clj -T:build uber

Your jar file is now created under target/.

Shutdown your system in your REPL (to free up port 3000), then test you can run your standalone app:

java -jar target/polls-1.2.5-standalone.jar

✎ changes

To support interactive development, we need some convenient REPL commands to allow reloading our code.

Remember, the "user" namespace is the default namespace your REPL uses. But at the moment it is an empty namespace because we haven't defined it.

Make a top-level directory env/dev/ containing:

• a user.clj
• another directory called resources/

Make our env directory:

mkdir -p env/dev/resources/ env/prod/resources/

Now, following the guidelines in the integrant-repl README, we can create our user namespace at env/dev/user.clj:

(ns user
  (:require [integrant.core :as ig]
            [integrant.repl :refer [clear go halt prep init reset reset-all]]
            [polls.core :as pc]
            [clojure.repl :refer [doc]]))

(integrant.repl/set-prep! #(ig/prep (pc/system-config)))

As this namespace is only used for development, and has an extra dependency, so we need an alias in our deps.edn which loads it. Add this alias to your deps.edn inside :aliases:

:dev {:extra-paths ["env/dev" "env/dev/resources"]
      :extra-deps {integrant/repl {:mvn/version "0.3.3"}}}

Now you need to ensure that when you start your REPL it will use the dev profile. If you are using Emacs with CIDER you can do this by creating a .dir-locals.el:

((nil . ((cider-clojure-cli-aliases . "dev"))))

Once you have that working and you start up your REPL, you should have these functions available in your REPL, in the default user namespace:

(go)
(halt)
(reset)

Now whenever you change your code you can reload it with (reset).

Now we need to move some things around to setup our profiles properly.

Move the system.edn and credentials file you made earlier into "env/dev/resources". And write a function that finds system.edn in the classpath (instead of hardcoded path).

Then update build.clj, adding the the prod resources directory to the src-dirs.

✎ changes

We want to setup migratus so that we can run the migrations either from the user namespace, or from the command-line.

In user.clj, require [migratus.core :as migratus].

Now in core.clj you need integrant to create a migratus config:

;; migratus
(defmethod ig/init-key :migratus/config [_ {:keys [database]}]
  {:store :database
   :migration-dir "migrations/"
   :db {:connection database
        :managed-connection? true}})

ⓘ note the option :managed-connection? true. This is needed otherwise migratus will close our database connection after it runs.

And edit your system.edn to start that up:

:migratus/config (:database #ig/ref :db.sql/connection)

Now restart everything and you should be able to do this in your REPL, in user namespace:

(migratus/migrate (:migratus/config integrant.repl.state/system))

It won't do anything because we don't have any migrations so expected output is nil.

ⓘ don't forget to run (go) so your system is actually running.

🏃 activity: create 0-argument functions in your user namespace called migrate, rollback and create-migration which run those migratus commands.

✎ changes

Test your create-migration function to make our first migration file (create-migration "question-and-choice").

Now, fill in your migrations:

In up:

create table question (
  question_id serial not null primary key,
  question_text text,
  pub_date timestamp default current_timestamp
);

--;;

create table choice (
  choice_id serial not null primary key,
  question_ref integer references question (question_id) on delete cascade,
  choice_text text,
  votes integer default 0 not null
);

In down:

drop table choice;

--;;

drop table question;

Run the migration:

(migrate)

Response code of nil indicates no errors. So you now have the database tables to support your app.

To allow our project to actually be deployed you will need to be able to also run the migration functions once your app is compiled (and you no-longer have access to your REPL).

🏃 activity: create an extra Integrant system config, which will only run the bits we need for migrations. i.e. read in the database credentials, connect to the db, and initialise the migratus config.

And edit our main function, so that it reads it's arguments, and starts different systems accordingly:

• with no args, it prints usage
• with "run-app", it starts up the main system as before
• with "migrate" or "rollback" it will run the more minimal system (db only) run the pertinent command, then shut-down the system.

Now compile your app and test you can run your migration on the command-line, something like this:

java -jar target/polls-1.2.5-standalone.jar migrate

When you're finished, here's how I did it:

✎ changes

Create your handlers namespace with some stub pages rendered by hiccup:

(ns polls.handlers
  (:require [polls.queries :as q]
            [hiccup2.core :as h]))

(defn render-message
  "render a message on an html page"
  [msg]
  (str (h/html [:html [:head] [:body [:p msg]]])))

(defn home-handler
  "stub"
  [x]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (render-message "Question list here")})

(defn question-handler
  "stub"
  [x]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (render-message "View Question")})

(defn post-handler
  "stub"
  [x]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (render-message "Answer Question")})

(defn results-handler
  "stub"
  [x]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (render-message "See results")})

Now we make a "routes" namespace. Our handler will send requests here to be routed by Reitit.

We adapt the reitit ring example.

(ns polls.routes
  (:require [muuntaja.core :as m]
            [reitit.ring :as ring]
            [reitit.coercion.spec]
            [reitit.ring.coercion :as rrc]
            [reitit.ring.middleware.muuntaja :as muuntaja]
            [reitit.ring.middleware.parameters :as parameters]
            [polls.handlers :as hl]))

(def app
  (ring/ring-handler
    (ring/router
      [["/" {:handler hl/home-handler}]
       ["/question" {:get {:handler hl/question-handler}
                     :post {:handler hl/post-handler}}]
       ["/results" {:handler hl/results-handler}]]
      
      ;; router data affecting all routes
      {:data {:coercion   reitit.coercion.spec/coercion
              :muuntaja   m/instance
              :middleware [parameters/parameters-middleware
                           rrc/coerce-request-middleware
                           muuntaja/format-response-middleware
                           rrc/coerce-response-middleware]}})))

Now in your core namespace, update your :polls/handler to just return the app function from your new routes namespace.

✎ changes

Check that works and you can access those three pages.

Now we have reitit let's just quickly setup request logging with ring-logger.

Add it to your deps.edn.

In your routes.clj require [ring.logger :as logger].

Now add logger/wrap-with-logger to the end of your :middleware.

Finally edit your log4j.properties file and replace DEBUG with INFO. This is so the INFO messages ring.logger creates are not drowned out by all the DEBUG spam.

✎ changes

One thing that's annoying is the favicon tracebacks in the request log and generally the lack of error handling if hitting a URL that's not setup.

So we make a 404 handler. The docs for how we do this is here.

In handlers.clj, add:

(defn not-found-handler
  "display not found page"
  []
  {:status 404
   :headers {"Content-Type" "text/html"}
   :body (ph/render-message "Page Not Found")})

In routes.clj, add this as the second argument to ring/ring-handler:

(constantly (hl/not-found-handler))

✎ changes

Let's setup our index page so it fetches the list of questions from the database and displays them.

In-order to do that we need to allow our handlers to run queries, which means they need a function pre-loaded with the :polls/queryfn. So start by updating your system.edn to pass in your query builder function instead of database connection:

:polls/handler {:queryfn #ig/ref :polls/queryfn}

Now, so that all our handlers have access to this "queryfn", we configure reitit to inject it into the request object using a middleware.

Update your handler method in core.clj:

(defmethod ig/init-key :polls/handler [_ {:keys [queryfn]}]
  (routes/app queryfn))

Now in routes.clj, re-write your "app" function like this:

(defn app
  "reitit with format negotiation and input & output coercion"
  [queryfn]
  ;; we define a middleware that includes our query builder
  (let [wrap-query-builder (fn [handler]
                             (fn [request]
                               (handler (assoc request :queryfn queryfn))))]

    (ring/ring-handler
     (ring/router
      [["/" {:handler hl/home-handler}]
       ["/question" {:get {:handler hl/question-handler}
                     :post {:handler hl/post-handler}}]
       ["/results" {:handler hl/results-handler}]]
      
      ;; router data affecting all routes
      {:data {:coercion   reitit.coercion.spec/coercion
              :muuntaja   m/instance
              :middleware [parameters/parameters-middleware
                           rrc/coerce-request-middleware
                           muuntaja/format-response-middleware
                           rrc/coerce-response-middleware
                           wrap-query-builder]}}))))

See how we define our middleware in a let, and add it to the middleware at the bottom.

✎ changes

In handlers.clj, split your render-message so you have a re-usable base template and individual templates for different pages:

(defn render-base
  "the basic structure of an html document. takes a title and list of elements"
  [title contents]
  (-> [:html
       [:head [:title title]]
       [:body
        [:header [:h1 title]]
        (into [:main]
              contents)]]
      h/html
      str))

(defn render-message
  "render a message on an html page"
  [msg]
  (let [contents [:p msg]]
    (render-base "Message" [contents])))

(defn render-questions
  "render a message on an html page"
  [questions]
  (let [contents (into [:ul] (for [{:question/keys [question_text
                                                    question_id
                                                    pub_date]} questions]
                               [:li
                                (str pub_date ": ")
                                [:a {:href (str "/question/" question_id)}
                                 question_text]]))]
    
    (render-base "Questions List" [contents])))

Now you may setup your home-handler to get the query-builder function from the request object (where the middleware put it) and use it to run the queries:

(defn home-handler
  "display the list of questions"
  [{exec-query :queryfn}]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (render-questions (exec-query (q/get-questions)))})

Now you should be able to see your questions listed on the index page. I know the date needs formatting but you can fix that yourself if you wish.

✎ changes

Next is to setup the question page. As it relies on the URL path to know the question id so we need to configure reitit to validate and coerce.

Note that there are three coercion modules for reitit. We are using the spec one, with clojure.spec.

Add [clojure.spec.alpha :as s] to your routes.clj requires. Define a spec for validating question ids:

(s/def ::id int?)

Now you need to update your path to extract the question_id, and add a :parameters key to specify the validation. The parameters-middleware will read and action the validation.

["/question/:question_id" {:get {:handler hl/question-handler
                                 :parameters {:path {:question_id ::id}}}
                           :post {:handler hl/post-handler
                                  :parameters {:path {:question_id ::id}}}}]

Now in the handlers namespace, update your question-handler to print out the question specified by the id:

(defn question-handler
  "display a specific question"
  [{exec-query :queryfn
    {{:keys [question_id]} :path} :parameters}]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (ph/render-message (-> (exec-query (q/get-question question_id))
                                first
                                :question/question_text))})

Now create your query to get a specific question from an id:

(defn get-question
  "We output a query and post-processor"
  [question_id]
  (-> (select :question_text)
      (from :question)
      (where [:= :question_id question_id])))

Check that works.

✎ changes

Something that's a bit inelegant about what we just did is that we had to post-process our query to extract the part we want.

(-> (exec-query (q/get-question question_id))
    first
    :question/question_text)

So let's enhance our execute-query function to take two values, a query and a post-processor function. Also we add an optional debug parameter so we can see what Honey SQL has generated.

(defn execute-query
  "takes a connection, a hugsql query, and a post-processor function"
  [conn [query processor] & [debug]]
  (let [formatted-query (sql/format query)]
    (when debug (println (str "formatted-query: " formatted-query)))
    (-> (jdbc/execute! conn formatted-query)
        processor)))

Update all the query functions to return a vector of two values:

• the query
• the identity function as a placeholder for the post-processor

✎ changes

🏃 activity: implement a post-processor for get-question, so you can remove the threading from the question-handler.

When you're finished, here's how I did it:

✎ changes

It's highly likely you will want to serve static assets at some point (CSS, JavaScript, images). We can do that with ring.middleware.resource, ring.middleware.content-type and a reitit catch-all parameter.

Make a directory resources/public/style.

Put in there a sample CSS file:

body{
    margin:40px auto;
    max-width:650px;
    line-height:1.6;
    font-size:18px;
    color:#444;
    padding:0 10px
}
h1,h2,h3{
    line-height:1.2
}

Now in routes.clj, add to your requires:

• [ring.middleware.resource :refer [wrap-resource]]
• [ring.middleware.content-type :refer [wrap-content-type]]

Add a new route like this:

["/public/*path" {:get {:middleware [wrap-content-type
                                     [wrap-resource ""]]
                        :handler hl/not-found-handler}}]

Now you should be able to load http://localhost:3000/public/style/style.css

And check it has an appropriate Content-Type header.

Now just update your render-base template to use our new CSS.

✎ changes