Alex's Slip-box

These are my org-mode notes in sort of Zettelkasten style

publish.org

:ID: 33D6368F-C063-40E0-8369-9FA8954C8A46

# Overview

This is a self-documenting org-mode publishing script. It is run by executing all the source code blocks herein. The actual publish.el script just loads this file in a buffer and calls org-babel-execute-buffer followed by the function that starts the publishing process.

# Learning resources for org-mode publishing

# Usage

# Development

# Assets

Assets like CSS and JS files are stored in a git backed directory site_asstes. The files therein are symlinked in the public directory which is gitignored. ln -s site_assets/js public/js ln -s site_assets/css public/css ln -s site_assets/favicon public/favicon

# Serving locally

After building locally to the public directory, I use the emacs-web-server package to serve it locally. Use httpd-serve-directory and point it at the publishing directory. It will serve the site on http://localhost:8080/.

# Building

Run build_script.sh. Among other things, this calls out to functions defined in publish.el, which itself uses this file.

This build requires having Ruby and lunr installed. I use these to pre-build the search index.

Files will build into the public/ dir

In order to not have to mess with build dependencies, use docker:

# With docker

  • Build a tagged container: docker build -t build-site .
  • Run the build script against the container: docker run -it --rm -v "$PWD/public":/app/public build-site

# Deploying

# Dependencies

# Package repositories

  • Adds melpa and elpa archives.
  • Sets the package archives directory so that packages aren’t installed in ~/.emacs.d/elpa.
 1: (require 'package)
 2: 
 3: (setq package-user-dir (expand-file-name "./.packages"))
 4: 
 5: (setq package-archives '(("melpa" . "https://melpa.org/packages/")
 6:                         ("elpa" . "https://elpa.gnu.org/packages/")))
 7: 
 8: (package-initialize)
 9: (unless package-archive-contents
10:   (package-refresh-contents))

# use-package

See use-package repo for more on this. It’s a clean way to handle dependencies in emacs.

11: (unless (package-installed-p 'use-package)
12:   (package-install 'use-package))
13: (require 'use-package)

# esxml

esxml provides elisp functions to generate HTML markup. Basically this means less reliance on ugly concat and format function calls.

Breaking Change Using esxml below is currently broken due to a change in esxml that escapes HTML when inserting strings. See this issue. Using a function to render the header, renders a string that gets HTML escaped when inserted into the main template below. Hence, the HTML is broken in this case and others that follow this pattern.

To temporarily get around that, I’ve “locked” the esxml package by copying a working version of it to .packages as part of the build process. Then it gets manually loaded here if it exists. Kinda gross, but it works for now.

14: (let ((path (car (file-expand-wildcards "./.packages/esxml*/esxml.el"))))
15:   (if (file-exists-p path)
16:       (load-file path)))
17: (use-package esxml :ensure t)

# htmlize

  • I don’t really know much about emacs-htmlize and all of its capabilities, but in the context of this script, it provides CSS styling for code syntax highlighting.
  • I believe the default is to use inline CSS, but it can generate a style sheet based on your emacs theme by calling org-html-htmlize-generate-css. I did that then linked the stylesheet in the HTML document <head>.
  • Tell it to use a stylesheet over line styles by setting the org-html-htmlize-output-type variable. See below.
  • Check out Org css for more on this.
18: (use-package htmlize :ensure t)

# ts

ts.el for sanity when formatting and parsing dates.

19: (use-package ts :ensure t)

# s

s.el for sanity when working with strings.

20: (use-package s :ensure t)

# ox-publish

The publishing system for org-mode

21: (require 'ox-publish)

# Variables

# Site variables

These get referenced when generating the HTML.

22: (setq my/site-title   "Alex's Slip-box"
23:       my/site-tagline "These are my org-mode notes in sort of Zettelkasten style"
24:       my/sitemap-title "")

# Org publish and export variables

I’m not going to bother explaining all these since they’re thoroughly explained with describe-variable

25: (setq org-publish-use-timestamps-flag t
26:       org-publish-timestamp-directory "./.org-cache/"
27:       org-export-with-section-numbers nil
28:       org-export-use-babel nil
29:       org-export-with-smart-quotes t
30:       org-export-with-sub-superscripts nil
31:       org-export-with-tags 'not-in-toc
32:       org-export-date-timestamp-format "Y-%m-%d %H:%M %p"
33:       org-id-locations-file-relative t
34:       org-id-locations-file "./.org-id-locations"
35:       org-id-track-globally t)

# HTML exporter variables

  • Tell htmlize to use a CSS stylesheet rather than inline styles.
  • Use describe-variable to learn about the rest of them.
36: (setq org-html-metadata-timestamp-format "%Y-%m-%d"
37:       org-html-checkbox-type 'site-html
38:       org-html-html5-fancy nil
39:       org-html-htmlize-output-type 'css
40:       org-html-self-link-headlines t
41:       org-html-validation-link nil
42:       org-html-inline-images t
43:       org-html-doctype "html5")

# Other variables

This is backed by a git repository, so we don’t need backups

44: (setq make-backup-files nil)

# Export document

# Site header

  • This function is called when generating the HTML template below.
    • info arg is a plist from which we can get configuration details about the org document. I’m not using it here, but it comes in handy in other functions to get things like the document title, date, etc.
  • Here I am using esxml to declare the markup in elisp.
    • It’s quoted (with `) but we can use , to selectively evaluate expressions therein. Noice.
    • @ function is for declaring node attributes like class, id or whatever.
45: (defun my/site-header (info)
46:   (sxml-to-xml
47:    `(div (@ (class "header uk-section uk-section-primary"))
48:          (div (@ (class "heading uk-container"))
49:               (div (@ (class "site-title-container uk-flex uk-flex-middle"))
50:                    (h1 (@ (class "site-title uk-h1 uk-heading-medium")) ,my/site-title)
51:                    (form (@ (class "uk-search uk-search-default search-form")
52:                             (id "search-form"))
53:                          (span (@ (uk-search-icon "")) "")
54:                          (input (@ (class "uk-search-input")
55:                                    (type "search")
56:                                    (placeholder "Search")))))
57:               (div (@ (class "site-tagline uk-text-lead")) ,my/site-tagline))
58:          (div (@ (class "uk-container"))
59:               (nav (@ (class "uk-navbar-container uk-navbar-transparent")
60:                       (uk-navbar))
61:                    (div (@ (class "uk-navbar-left"))
62:                         (ul (@ (class "uk-navbar-nav"))
63:                             (li (a (@ (class "nav-link") (href "/")) "Notes"))
64:                             (li (a (@ (class "nav-link") (href "https://github.com/apmiller108")) "Github"))
65:                             (li (a (@ (class "nav-link") (href "https://alex-miller.co")) "alex-miller.co")))))))))

# Site footer

  • This function is called when generating the HTML template below.
  • creator is Emacs {{version}} (Org mode {{version}})
66: (defun my/site-footer (info)
67:   (sxml-to-xml
68:   `(footer (@ (class "footer uk-section uk-section-secondary"))
69:             (div (@ (class "uk-container footer-container"))
70:                  (div (@ (class "footer-links"))
71:                       (a (@ (href "https://notes.alex-miller.co/")
72:                             (class "footer-link")
73:                             (uk-icon "icon: album"))
74:                             "notes")
75:                       (a (@ (href "https://github.com/apmiller108")
76:                             (class "footer-link")
77:                             (uk-icon "icon: github-alt"))
78:                             "github")
79:                       (a (@ (href "https://twitter.com/apmiller108")
80:                             (class "footer-link")
81:                             (uk-icon "icon: twitter"))
82:                          "@apmiller108")
83:                       (a (@ (href "https://www.reddit.com/user/apmillz")
84:                             (class "footer-link")
85:                             (uk-icon "icon: reddit"))
86:                          "u/apmillz"))
87:                  (div (@ (class "made-with"))
88:                       (p "Made with " ,(plist-get info :creator)))))))

# The HTML Template

  • This is the whole page layout. It makes use of the header and footer functions above and injects the org-mode document exported HTML (the contents arg).
  • I think all of this is pretty self explanatory, but one thing I should call out is the use of :filetags to generate the tag links. I’m not entirely sure I had to do this, but I declared as a custom export option in the derived backend. See below.
  • Same with the :updated property.
 89: (defun my/org-html-template (contents info)
 90:   (concat
 91:   "<!DOCTYPE html>"
 92:   (sxml-to-xml
 93:     `(html (@ (lang "en"))
 94:           (head
 95:             (meta (@ (charset "utf-8")))
 96:             (meta (@ (author "Alex P. Miller")))
 97:             (meta (@ (name "viewport")
 98:                     (content "width=device-width, initial-scale=1, shrink-to-fit=no")))
 99:             (link (@ (rel "apple-touch-icon")
100:                     (sizes "180x180")
101:                     (href "/favicon/apple-touch-icon.png?v=1")))
102:             (link (@ (rel "icon")
103:                     (type "image/png")
104:                     (sizes "32x32")
105:                     (href "/favicon/favicon-32x32.png?v=1")))
106:             (link (@ (rel "icon")
107:                     (type "image/png")
108:                     (sizes "16x16")
109:                     (href "/favicon/favicon-16x16.png?v=1")))
110:             (link (@ (rel "manifest")
111:                     (href "/favicon/manifest.json?v=1")))
112:             (link (@ (rel "mask-icon")
113:                     (href "/favicon/safari-pinned-tab.svg?v=1")))
114:             (link (@ (rel "stylesheet")
115:                     (href "/css/uikit.min.css")))
116:             (link (@ (rel "stylesheet")
117:                     (href "/css/code.css")))
118:             (link (@ (rel "stylesheet")
119:                     (href "/css/site.css")))
120:             (script (@ (src "/js/uikit.min.js")) nil)
121:             (script (@ (src "/js/uikit-icons.min.js")) nil)
122:             (script (@ (src "/js/lunr.min.js")) nil)
123:             (script (@ (src "/js/site.js")) nil)
124:             (script (@ (src "https://www.googletagmanager.com/gtag/js?id=G-YM3EHHB2YQ")) nil)
125:             (script
126:             "window.dataLayer = window.dataLayer || [];
127:               function gtag(){dataLayer.push(arguments);}
128:               gtag('js', new Date());
129:               gtag('config', 'G-YM3EHHB2YQ');"
130:             )
131:             (title ,(concat (org-export-data (plist-get info :title) info) " - Alex's Notes")))
132:           (body
133:             ,(my/site-header info)
134:             (div (@ (class "main uk-section uk-section-muted"))
135:                   (div (@ (class "note uk-container"))
136:                       (div (@ (class "note-content"))
137:                             (h1 (@ (class "note-title uk-h1"))
138:                                 ,(org-export-data (plist-get info :title) info))
139:                             (div (@ (class "note-meta"))
140:                                 ,(when (plist-get info :date)
141:                                     `(p (@ (class "note-created uk-article-meta"))
142:                                         ,(format "Created on %s" (ts-format "%B %e, %Y" (ts-parse (org-export-data (plist-get info :date) info))))))
143:                                 ,(when (plist-get info :updated)
144:                                     `(p (@ (class "note-updated uk-article-meta"))
145:                                         ,(format "Updated on %s" (ts-format "%B %e, %Y" (ts-parse (plist-get info :updated)))))))
146:                             ,(let ((tags (org-export-data (plist-get info :filetags) info)))
147:                                (when (and tags (> (length tags) 0))
148:                                  `(p (@ (class "blog-post-tags"))
149:                                      "Tags: "
150:                                      ,(mapconcat (lambda (tag) (format "<a href=\"/?tag=%s\">%s</a>" tag tag))
151:                                                  (plist-get info :filetags)
152:                                                  ", "))))
153:                             ,contents)
154:                       ,(when (not (string-equal my/sitemap-title (org-export-data (plist-get info :title) info)))
155:                           '(script (@ (src "https://utteranc.es/client.js")
156:                                       (repo "apmiller108/slip-box")
157:                                       (issue-term "title")
158:                                       (label "comments")
159:                                       (theme "boxy-light")
160:                                       (crossorigin "anonymous")
161:                                       (async))
162:                                   nil))))
163:                   ,(my/site-footer info)
164:                   (div (@ (id "search-results")
165:                           (class "search-results")
166:                           (uk-modal ""))
167:                        (div (@ (class "uk-modal-dialog uk-modal-body"))
168:                             (h2 (@ (class "uk-modal-title")
169:                                    (id "search-results-title"))
170:                                 "Search Results")
171:                             (div (@ (id "search-results-body")
172:                                     (class "search-results-body"))
173:                                  "")
174:                             (a (@ (class "uk-modal-close-default")
175:                                        (uk-close ""))
176:                                     ""))))))))

# Element customization

# Links and Images

  • The link paths need to match the actual file paths of the exported files. So for file links, the exported link paths are downcased and without filename extensions. So, this function ensures the link paths match that format. So [[file:my_post.org][My Post]] becomes <a href="my_post">My Post</a> (no .html on the path).
  • Org-roam uses the ID property for linking notes (ie, no file path). To get around this I do the following:
    1. In my my publish.el file, I generate the .org-id-locations file. This file is committed since it is also used on CI where I couldn’t even generate this file as part of the build process.
    2. Again in publish.el , set the my/org-id-locations variable to a hashtable generated from the .org-id-locations file.
    3. For fuzzy type links, find the path from the hashtable. Oh, somehow the fuzzy type links are the ID links.
    4. Seriously, what a pain in the arse.
  • I have some inline images in my org files. These are file links without a label that point to files with image extensions. Mostly these are plantuml renderings. They get converted to HTML img tags.
  • For everything else, just render a good old fashion anchor tag.
177: (defun my/format-path-for-anchor-tag (path)
178:   (concat "/"
179:           (downcase
180:            (file-name-sans-extension
181:             path))))
182: (defun my/org-html-link (link contents info)
183:   "Removes file extension and changes the path into lowercase org file:// links.
184:   Handles creating inline images with `<img>' tags for png, jpg, and svg files
185:   when the link doesn't have a label, otherwise just creates a link."
186:   ;; TODO: refactor this mess
187:   (if (string= 'fuzzy (org-element-property :type link))
188:       (let ((path (gethash (s-replace "id:" "" (org-element-property :path link)) my/org-id-locations)))
189:         (if path
190:             (org-element-put-property link :path
191:                                       (my/format-path-for-anchor-tag
192:                                        (car (last (s-split "/" path))))))))
193:   (when (and (string= 'file (org-element-property :type link))
194:             (string= "org" (file-name-extension (org-element-property :path link))))
195:     (org-element-put-property link :path
196:                               (my/format-path-for-anchor-tag
197:                                         (org-element-property :path link))))
198: 
199:   (if (and (string= 'file (org-element-property :type link))
200:           (file-name-extension (org-element-property :path link))
201:           (string-match "png\\|jpg\\|svg\\|webp"
202:                         (file-name-extension
203:                           (org-element-property :path link)))
204:           (equal contents nil))
205:       (format "<img src=/%s >" (org-element-property :path link))
206:     (if (and (equal contents nil)
207:             (or (not (file-name-extension (org-element-property :path link)))
208:                 (and (file-name-extension (org-element-property :path link))
209:                       (not (string-match "png\\|jpg\\|svg\\|webp"
210:                                         (file-name-extension
211:                                           (org-element-property :path link)))))))
212:         (format "<a href=\"%s\">%s</a>"
213:                 (org-element-property :raw-link link)
214:                 (org-element-property :raw-link link))
215:       (format "<a href=\"%s\">%s</a>"
216:               (org-element-property :path link)
217:               contents))))

# Headings

This part is largely unchanged from David Wilson’s publish.el on which this is based.

  • Maybe something else already requires subx-r.el, but we make sure we can use thread-last.
  • This helper function is used when rendering headlines. It kebab cases the cases the headline text for use as the HTML element’s ID.
    • Sometimes heading words are fenced with ~, so the code tag is removed.
218: (require 'subr-x)
219: 
220: (defun my/make-heading-anchor-name (headline-text)
221:   (thread-last headline-text
222:     (downcase)
223:     (replace-regexp-in-string " " "-")
224:     (replace-regexp-in-string "</?code>" "")
225:     (replace-regexp-in-string "[^[:alnum:]_]" "")))
  • Basically, this translates the org-mode headlines to HTML h tags of the corresponding level with anchor tag handles, IDs that can be easily linked to, while respecting export options.
226: (defun my/org-html-headline (headline contents info)
227:   (let* ((text (org-export-data (org-element-property :title headline) info))
228:         (level (org-export-get-relative-level headline info))
229:         (level (min 7 (when level (1+ level))))
230:         (anchor-name (my/make-heading-anchor-name text))
231:         (attributes (org-element-property :ATTR_HTML headline))
232:         (container (org-element-property :HTML_CONTAINER headline))
233:         (container-class (and container (org-element-property :HTML_CONTAINER_CLASS headline))))
234:     (when attributes
235:       (setq attributes
236:             (format " %s" (org-html--make-attribute-string
237:                            (org-export-read-attribute 'attr_html
238:                                                       `(nil
239:                                                         (attr_html ,(split-string attributes))))))))
240:     (concat
241:      (when (and container (not (string= "" container)))
242:        (format "<%s%s>" container (if container-class (format " class=\"%s\"" container-class) "")))
243:      (if (not (org-export-low-level-p headline info))
244:          (format "<h%d%s><a id=\"%s\" class=\"anchor\" href=\"#%s\"><i># </i></a>%s</h%d>%s"
245:                 level
246:                 (or attributes "")
247:                 anchor-name
248:                 anchor-name
249:                 text
250:                 level
251:                 (or contents ""))
252:        (concat
253:         (when (org-export-first-sibling-p headline info) "<ul>")
254:         (format "<li>%s%s</li>" text (or contents ""))
255:         (when (org-export-last-sibling-p headline info) "</ul>")))
256:      (when (and container (not (string= "" container)))
257:        (format "</%s>" (cl-subseq container 0 (cl-search " " container)))))))

# The Sitemap (the home page)

# Sitemap Entry

Formats sitemap entry as {date} {title} ({filetags}). Returns a list containing the sitemap entry string and the filetags. A unique list of the filetags is created on the sitemap page from this list, that’s why they’re returned from this function.

258: (defun my/sitemap-format-entry (entry style project)
259:   (let* ((filetags (org-publish-find-property entry :filetags project 'site-html))
260:          (created-at (format-time-string "%Y-%m-%d"
261:                                          (date-to-time
262:                                           (format "%s" (car (org-publish-find-property entry :date project))))))
263:          (entry
264:           (sxml-to-xml
265:            `(li (@ (data-date ,created-at)
266:                    (class ,(mapconcat (lambda (tag) tag) filetags " ")))
267:                 (span (@ (class "sitemap-entry-date")) ,created-at)
268:                 (a (@ (href ,(file-name-sans-extension entry)))
269:                    ,(org-publish-find-title entry project))
270: 
271:                 ,(if filetags
272:                      `(span (@ (class "sitemap-entry-tags"))
273:                             ,(concat "("
274:                                      (mapconcat (lambda (tag) tag) filetags ", ")
275:                                      ")")))))))
276:         (list entry filetags)))

# Sitemap page

From the function above, the filetags are placed into a flattened list, duplicate values removed and sorted alphabetical ascending. These are turned into tags on the page used for filtering the entries by topic. All of the JS used for filtering is provided by the UIkit CSS framework.

277: (defun my/sitemap (title list)
278:   (let* ((unique-tags
279:           (sort
280:           (delete-dups
281:             (flatten-tree
282:               (mapcar (lambda (item) (cdr (car item)))
283:                       (cdr list))))
284:           (lambda (a b) (string< a b)))))
285:     (concat
286:     "#+TITLE: " title "\n\n"
287:     "#+BEGIN_EXPORT html\n\n"
288:     (sxml-to-xml
289:      `(div (@ (id "tag-filter-component")
290:               (uk-filter "target: .js-filter"))
291:            (div (@ (class "tags uk-subnav uk-subnav-pill"))
292:                 (span (@ (uk-filter-control "group: tag"))
293:                       (a (@ (href "#")) "ALL"))
294:                 ,(mapconcat (lambda (item)
295:                               (format "<span id=\"%s\" uk-filter-control=\"filter: .%s; group: tag\"><a href=\"#\">%s</a></span>"
296:                                       (concat "filter-" item)
297:                                       item
298:                                       item))
299:                             unique-tags
300:                             "\n"))
301:            (ul (@ (class "uk-subnav uk-subnav-pill"))
302:                (li (@ (uk-filter-control "sort: data-date; group: date"))
303:                    (a (@ (href "#")) "Ascending"))
304:                (li (@ (uk-filter-control "sort: data-date; order: desc; group: date")
305:                       (class "uk-active"))
306:                    (a (@ (href "#")) "Descending")))
307:            (ul (@ (class "sitemap-entries uk-list uk-list-emphasis js-filter"))
308:                ,(mapconcat (lambda (item) (car (car item)))
309:                           (cdr list)
310:                           "\n"))))
311:     "\n#+END_EXPORT\n")))

# Derived backend

You can derive a custom backend from an existing one and can override certain functions. In this example, my-site-html derives from html and overrides template, link, and headline functions.

312: (org-export-define-derived-backend
313:     'site-html
314:     'html
315:   :translate-alist
316:   '((template . my/org-html-template)
317:     (link . my/org-html-link)
318:     (headline . my/org-html-headline))
319:   :options-alist
320:   '((:page-type "PAGE-TYPE" nil nil t)
321:     (:html-use-infojs nil nil nil)
322:     (:updated "UPDATED" nil nil t)
323:     (:filetags "FILETAGS" nil nil split)))

# Publishing

# Output paths

This is a helper function that converts an org-mode file name to a directory of the same name, downcased and without the filename extension. So if the filename is my-post.org, a sub-directory would be created in the publishing directory called my-post/. The sitemap is indented to be at the root of the publishing directory (ie, the homepage). This function is called in the next code block.

324: (defun get-article-output-path (org-file pub-dir)
325:   (let ((article-dir (concat pub-dir
326:                             (downcase
327:                               (file-name-as-directory
328:                               (file-name-sans-extension
329:                                 (file-name-nondirectory org-file)))))))
330:     (if (string-match "\\/sitemap.org$" org-file)
331:         pub-dir
332:         (progn
333:           (unless (file-directory-p article-dir)
334:             (make-directory article-dir t))
335:           article-dir))
336:     ))

# The publishing function (and conditional TOCs)

This function does a few things:

  • It adds the export option to generate a table of contents only if there are more than 3 headlines. Otherwise, I don’t see a point to rendering a TOC.
  • Next it calls the helper function above to create the output directory and appends index.html to the result. This ends up being the article-path for a post. For example, if the filename is my-post.org, the article path would be /my-post/index.html.
  • Finally, it calls org-publish-org-to which publishes a file using the selected backend.
337: (defun my/org-html-publish-to-html (plist filename pub-dir)
338:   (with-current-buffer (find-file filename)
339:     (when (> (length (org-map-entries t)) 3)
340:       (insert "#+OPTIONS: toc:t\n")))
341:   (let ((article-path (get-article-output-path filename pub-dir)))
342:     (cl-letf (((symbol-function 'org-export-output-file-name)
343:               (lambda (extension &optional subtreep pub-dir)
344:                 (concat article-path "index" extension))))
345:       (org-publish-org-to 'site-html
346:                           filename
347:                           (concat "." (or (plist-get plist :html-extension) "html"))
348:                           plist
349:                           article-path))))
350: 

# The project alist

This is the configuration for the publishable projects. Each project can be published independently with org-publish and the project name (eg (org-publish "site")), or all of them with org-publish-all.

351: (setq org-publish-project-alist
352:       (list
353:        (list "notes.alex-miller.co"
354:              :base-extension "org"
355:              :base-directory "./"
356:              :publishing-function '(my/org-html-publish-to-html)
357:              :publishing-directory "./public"
358:              :auto-sitemap t
359:              :sitemap-function 'my/sitemap
360:              :sitemap-title my/sitemap-title
361:              :sitemap-format-entry 'my/sitemap-format-entry
362:              :sitemap-sort-files 'alphabetically
363:              :with-title nil
364:              :with-toc nil)
365:        (list "images"
366:              :base-extension "png\\|jpg\\|svg\\|webp"
367:              :base-directory "./images"
368:              :publishing-directory "./public/images"
369:              :publishing-function 'org-publish-attachment)
370:        (list "site" :components '("notes.alex-miller.co" "images"))))

# notes.alex-miller.co

This publishes the org-mode files. I keep them in the root directory. I have a few other folders for other note types that I don’t publish. The HTML output is placed in the ./public directory which is gitignored. The sitemap functions are documented above. TOCs are only generated for notes that have more than 3 headlines.

# images

I sometimes link and display images in my org-notes, like plantuml renderings. I put these in the ./images directory. This basically just copies them over to the /public/images directory of the site. This ensure that links and/or inline images work. (See this emacs.stackexchange answer for where I got the idea).

# site

It contains everything needed to build the site.

Search Results