Pepijn Looije

Real World Ledger part 1: Weighing Eggs in Baskets

Thu, 23 Aug 2018 00:00:00 -0300

Do you ever feel like you’re losing grip on your personal finances? You deposit into your savings account in one currency, buy stocks and bonds in another, and maybe even hodl on to some cryptocurrency. You keep a finger on the pulse and occasionally check your assets’ value, but volatile prices and exchange rates make it challenging. Ledger is a command-line accounting tool that addresses these issues. In this post I’ll introduce you to it.

Introduction

The world around you is changing: interest rates on savings accounts slide to 0%, stock markets bubble and crash, and global political debate intensifies. Above all, your bank or government expects you to pay back that student loan sometime in the future. It is safe to say that our economy is a rough sea.

This post is part one in a series where I show how to map and plan your financial positions in Ledger so you’re able to navigate those real-world issues with confidence. How to do that exactly isn’t trivial, though. With this blog series I hope to fill that skills gap—the same that I encountered when starting with Ledger. I found many online examples too abstract and missing human/societal context for someone unfamiliar with accounting.

That said, I won’t be explaining to you the theory behind techniques like double-entry accounting and investment portfolios in this post because I am neither an accountant nor am I an investment advisor. The Ledger documentation does a good job of explaining double-entry¹ and Investopedia explains portfolios well². I will limit myself to the narrative and practical examples—those are the things I’ve personally experienced.

Part 1: Weighing Eggs in Baskets

The example story through this blog series features the development of a basic financial situation into an investment portfolio with carefully weighted stocks, government bonds, cryptocurrency, and of course cash. All is fine and balanced until a cryptocurrency hype comes knocking at the door and exposes the portfolio manager, you, to a novel risk.

We get started in this post by diversifying our savings into stocks, bonds, and cryptocurrency. This way, we won’t have all our eggs in one basket. We also discuss how you convert different assets to one currency so we can reliably weigh them: apples to apples. The weighing is essential when creating your investment portfolio allocation—which we’ll discuss in the next post.

“Ledger is a powerful, double-entry accounting system that is accessed from the UNIX command-line.” — https://www.ledger-cli.org

To follow along with the story below, you will need a terminal with Ledger installed and a plain text file editor, such as Sublime Text. If you use macOS, installing Ledger is easy using Homebrew: brew install ledger. Feel free to make your workflow more pleasant by installing a Ledger mode in your text editor—this gives you syntax highlighting. Sublime and other well-known editors (like Emacs and vim) have Ledger modes readily available online.

The basics: a single currency and a single asset class

Our first posting: adding our savings account

Let’s start simple: we have a savings account at a bank called ASN bank³ in our home country where most of our money resides. This account already has money in it—obviously we don’t start owning assets the moment we begin using Ledger—so we have to initialize our balance by moving money from somewhere. Idiomatically that somewhere is an account called ‘opening balances’. When we express this in Ledger, this is what the file postings1.dat (.dat is commonly used with Ledger, but feel free to use something else like .txt) looks like:

2018-01-01 Opening Balances
    Assets:NL:ASN:Savings                 € 1,337.00
    Equity:Opening Balances              € -1,337.00

How do we interpret these three lines? Every posting has a date (2018-01-01) and a payee (Opening Balances). Then, what follows directly beneath it are the entries belonging to that posting. In this case, we move the € 1337 from ‘opening balances’ to the savings account. Most of the labels here are arbitrary and depend on your preference and taste. I like to structure actual bank accounts as follows: country, name of bank, type of account. That results in Assets:NL:ASN:Savings.

Now we run our first query using the Ledger command-line tool. We ask for the balance of accounts that match assets in the file postings.dat.

ledger --file postings1.dat balance assets

The result, as expected, the balance of one asset account:

€ 1,337.00  Assets:NL:ASN:Savings

Our first mutation: interest from savings, and deposits and withdrawals

Fast forward 6 months. We have received some interest from the bank and did a couple of deposits and withdrawals. We could add postings for all the deposits and withdrawals, but that’s a lot of premature work and definitely not the required to benefit from Ledger. That’s why we’re using an ‘adjustment’ account in the following addition to our postings1.dat file, calling it postings2.dat.

2018-06-01 ASN
    Assets:NL:ASN:Savings                 € 3,787.50
    Income:Interest                         € -42
    Equity:Adjustment

A net amount of € 3,787.50 was added to the savings account, of which € 42 was interest received on the principal. The rest was the result of deposits and withdrawals. We don’t really care about tracking all those transactions in detail right now, so we lazily use an adjustment account. Lastly, we’re able to omit the amount of Equity:Adjustment because there’s only one possibility: € -3,787.50 - € 42 = € -3,745.5.

The adjustment account resolves a common discouragement of adopting Ledger that I keep hearing—people think that Ledger requires them to arduously type in all transactions like a monkey. You don’t, and above all, you can always do that later or build scripts to do it for you should you so desire.

We now rerun the Ledger command-line tool. This time, we ask for the balance of all accounts, not just assets:

ledger --file postings2.dat balance

Please note that the total of all accounts always sums to zero—that condition is the main property of double-entry accounting:

          € 5,124.50  Assets:NL:ASN:Savings
         € -5,082.50  Equity
         € -3,745.50    Adjustment
         € -1,337.00    Opening Balances
            € -42.00  Income:Interest
--------------------
                   0

Going deeper: multiple currencies and asset classes

Diversifying into multiple assets

We decided to diversify, hoping to get a better return than the ~0% interest rate on your savings account⁴ in our ~2% inflation habitat⁵. But, at the same time, you don’t want to go all-in on stocks because it’s generally considered a bad idea to put all your eggs in one basket. That’s why we diversify and buy some government bonds and cryptocurrency too. ‘Interactive Brokers’ and ‘Binck Bank’ in the file below are examples of stock/bond brokers. postings3.dat:

2018-07-01 Interactive Brokers
    Assets:NL:ASN:Savings                   € -1,285
    Assets:US:Interactive Brokers:Cash       $ 1,500

2018-07-02 Binck Bank
    Assets:NL:ASN:Savings                   € -2,000
    Assets:NL:BinckBank:Cash

2018-07-03 Interactive Brokers
    Assets:US:Interactive Brokers:Stocks      6 AAPL @ $ 183.92
    Assets:US:Interactive Brokers:Cash

2018-07-04 Binck Bank
    Assets:NL:BinckBank:Bonds      1,100 "NL2014-47" @ € 1.39
    Assets:NL:BinckBank:Stocks                5 HEIA @ € 86.08
    Assets:NL:BinckBank:Cash

2018-07-05 Coinbase
    Assets:Cryptocurrency:BTC wallet         BTC 0.1
    Assets:NL:ASN:Savings                     € -561

In the example above we use different syntax to reach the same goal: buying one commodity by selling another commodity (such as stocks from US dollars and Bitcoin from euros). The Ledger docs explain the differences clearly⁶.

Let’s check the impact of our asset diversification buying spree on our balance:

ledger --file postings3.dat balance assets --no-total --flat

Please be advised that I passed two new arguments: --no-total and --flat. The total is superfluous because we’re only looking at assets. Conversely, the total is valuable when you’re looking at both assets and liabilities. Subtracting them yields net worth⁷. And --flat is purely aesthetic. It suppresses Ledger’s automatic hierarchy view because it is confusing when printing heterogenous commodities (such as currencies, stocks, etc.).

        BTC 0.1  Assets:Cryptocurrency:BTC wallet
     € 1,278.50  Assets:NL:ASN:Savings
1,100 NL2014-47  Assets:NL:BinckBank:Bonds
        € 40.60  Assets:NL:BinckBank:Cash
         5 HEIA  Assets:NL:BinckBank:Stocks
          $ 396  Assets:US:Interactive Brokers:Cash
         6 AAPL  Assets:US:Interactive Brokers:Stocks

This balance sheet matches our expectations but it isn’t giving us much extra information about each of the assets relative to each other—value-wise we’re comparing apples to oranges. Wouldn’t it be nice to have all the assets converted to one currency so we can compare apples to apples?

Implicit and explicit market prices

In order to compare values of assets we have to pick a base currency to convert them to. I’m carrying a Dutch passport so my usual pick is to convert everything to euros. But, as long as you supply Ledger the exchange rates, you could express the value of your assets, even your guitar if you’re so inclined, in whatever commodity you like—from Apple stock to real apples⁸. Obviously your tools shouldn’t stop you from expressing the value of your guitar in apples that you pick from the tree! The only thing Ledger needs is either an implicit or explicit market price.

We’ll discuss prices in a moment. Before, to see the value of our assets expressed in euros, we run the following command (adding --exchange €):

ledger -f postings3.dat b Assets --exchange € --no-total

Finally, we have a birds-eye view of all our assets’s value across different countries, accounts, and currencies:

€ 5,124.50  Assets
  € 561.00    Cryptocurrency:BTC wallet
€ 3,278.50    NL
€ 1,278.50      ASN:Savings
€ 2,000.00      BinckBank
€ 1,529.00        Bonds
   € 40.60        Cash
  € 430.40        Stocks
€ 1,285.00    US:Interactive Brokers
  € 339.65      Cash
  € 945.35      Stocks

How did Ledger convert everything to euros? Ledger keeps track of prices implicitly and also allows you to specify prices manually—explicitly. Let’s focus on the implicit part first, by asking Ledger for the prices that it stored so far:

ledger -f postings3.dat prices

With this command you peek into Ledger’s internal price database. The prices that you see were established by the postings in postings3.dat and are all implicit:

2018/07/01 €        $ 1.167315175097
2018/07/03 AAPL         $ 183.92
2018/07/04 "NL2014-47"       € 1.39
2018/07/04 HEIA          € 86.08
2018/07/05 BTC        € 5,610.00

As a matter of experiment, let’s say the price of Apple stock recently shot up. It rose to an extent that we’re now curious to see how much the value of our US brokerage account increased. To find out, we’re going to explicitly express Apple’s stock price in US dollars in a new file called prices.dat:

P 2018-08-03 AAPL $ 207.99

The single line in this file states: on 2018-08-03 the price for AAPL in $ was 207.99. Let’s make this file available to Ledger by specifying --price-db and querying assets in the US (in which Apple belongs) only (Assets:US):

ledger --file postings3.dat \
       balance Assets:US \
       --exchange € \
       --price-db prices.dat \
       --no-total

Indeed, we see the gains on Apple stock reflected by our increased total US assets value. Apple stock got converted to US dollars got converted to euros:

€ 1,408.72  Assets:US:Interactive Brokers
  € 339.65    Cash
€ 1,069.07    Stocks

You should add a line to prices.dat for every price that you want to track. I personally have more than a thousand lines in my prices file and retrieve some prices automatically using APIs (predominantly forex rates). The benefit of a high resolution like that is that graphical plots of my assets, liabilities, and net worth (using a daily interval on the x-axis) are less jumpy.

Conclusion

To summarize, we’ve just created our first postings, discovered the implicit exchange rates that Ledger keeps and added an Apple stock price explicitly. All along the way we were able to query our balance in two representations: in its original commodity and converted to one base currency.

In part 2 we’ll look at how you materialize an investment portfolio strategy and asset allocation using Ledger. Please leave your email address if you want a notification once it’s published! I’d also love to hear your feedback about this post and hear suggestions about topics that you’d like to see discussed in depth. Reach out to me on Twitter: @ppnlo. Or through email: replace the first dot in the domain name with an @.

Appendix

As always, this post is written in a literate programming⁹ style, which means that the code samples in it are reproducible and correct. Check out the Org-mode and Babel source code on GitHub: real-world-ledger-part-1.org.

Thank you Thomas Smolders, Pieter Levels, Arend Koopmans, Rik Helwegen and Nils Mackay for helping me with this post!

Footnotes

¹ https://www.ledger-cli.org/3.0/doc/ledger3.html

² https://www.investopedia.com/terms/p/portfolio.asp

³ Eerlijke Bankwijzer: ASN Bank

⁴ Interest rates for ABN Amro savings accounts, similar to other Dutch banks: https://www.abnamro.nl/en/personal/savings/spaarrente.html

⁵ CBS inflation

⁶ https://www.ledger-cli.org/3.0/doc/ledger3.html#Explicit-posting-costs

⁷ https://en.wikipedia.org/wiki/Net_worth

⁸ https://www.ledger-cli.org/3.0/doc/ledger3.html#Posting-costs

⁹ https://en.wikipedia.org/wiki/Literate_programming

Sidestepping Heroku's limits with PostgreSQL 9.5's UPSERT

Wed, 20 Apr 2016 00:00:00 -0300

My roommate needed help with building a web scraper. As a result of various constraints, we ended up using Heroku. There was, however, a database row limit that we had to work around. It turned out to be a nice opportunity to play with the new PostgreSQL 9.5 INSERT ... ON CONFLICT DO UPDATE, also known as UPSERT, functionality. The goal of this article is to introduce UPSERT to you via a practical example.

Introduction

My roommate, an aspiring programmer, recently asked me for advice on how to build a web scraper. The scraper should run on a machine that is constanly connected to the internet, because it should check at least every hour or so. Running the scraper on his laptop is not an option for my roommate, because he frequently takes it with him on the road. Neither is running it on a remote server, because he possesses none nor is he planning to spend money on one. We tried out Heroku; a free platform as a service provider that embraces all of these constraints.

Choosing the right storage

The Heroku Scheduler addon enabled us to run a script every 10 minutes (the highest available frequency) that scrapes the targeted website. With that requirement settled, we needed a way to compare the old state to the new state, so we could send ourselves an email as soon as the website changed. The simplest solution we could come up with involved saving the hashed HTML contents into a Heroku config variable. That solves the problem of notification, but will not allow us analyse the data later. For this reason, using a database made sense. PostgreSQL storage is cheap, free actually on Heroku, so we picked that one.

Heroku limitations

Normally, I would insert one row per scraping run with the scraped contents (or a foreign key to a unique piece of content) and a timestamp into the database. In this case, however, it was not a good idea because the free Heroku PostgreSQL database offering has certain limitations. You are allowed to have a maximum of 10.000 rows in the database. A scraper that scrapes every 10 minutes reaches the limit after 40 days of service. We needed a way to circumvent the limitation, since we were planning on scraping longer than that.

What if we scaled horizontally instead of vertically? Instead of adding rows, we could grow columns and consequently circumvent the row limit. One row per unique piece of scraped content with an ever expanding array of timestamps. PostgreSQL arrays seem to be a good fit for this. One could implement this using two queries. First, a SELECT establishes whether we have seen the content before. Second, an INSERT or UPDATE respectively inserts a new piece of content with the current time to the array or updates an existing row by appending the current time. Or, you use the new INSERT … ON CONFLICT DO UPDATE operation instead, like I will show you in this article.

Methods

In this article, I will only show you how to complete the task using UPSERT. If you would like to see how you could achieve similar behavior with PostgreSQL versions below 9.5, please check out this Stackoverflow answer.

The query

We built the plumbing of the scraper using Python; the source code is available on GitHub. The Python script executes the following query–the interesting part–every time it runs.

CREATE TABLE IF NOT EXISTS scraps (
    id serial PRIMARY KEY,
    hash bytea NOT NULL UNIQUE,
    body text NOT NULL,
    seen_at timestamptz[] NOT NULL
);

-- This extension yields the digest function that enables us to
-- hash the body and index it efficiently. Moreover, PostgreSQL
-- does not allow indexes on very large text columns. We expect to
-- store large HTML bodies so we definitely need the hashing.
CREATE EXTENSION IF NOT EXISTS pgcrypto;


-- Capture the script input (body) via a CTE (the WITH part) so we
-- can use it multiple times in the query. Once to save the body,
-- once to hash it.
WITH body AS (SELECT :body::text AS txt)

-- This is exactly the INSERT you would write without UPSERT.
INSERT INTO scraps (hash, body, seen_at)
    SELECT digest(txt, 'sha1'), txt, ARRAY[:ts::timestamptz]
    FROM body

-- Here it gets interesting; we utilize the UNIQUE index on hash
-- to yield a conflict if the body already exists. If that
-- happens, we append the new seen_at (via the special 'EXCLUDED'
-- table) to the seen_at array.
  ON CONFLICT (hash) DO UPDATE
    SET seen_at = scraps.seen_at || EXCLUDED.seen_at

-- The query returns a summary of the row so I can use it in my
-- blog post. This part will run, but the output is ignored, in
-- production.
RETURNING id, left(hash::text, 3) || '...' hash, body, seen_at;

The test

We supply the query above to the script below via the $QUERY variable. First, we initialize an empty database and run the query twice with the body variable set to content and ts set to the SQL function now(). After that, we simulate a change on the targeted website by running the query again after setting content to changed. In production, the body variable contains an HTML document. The output of the script is printed below it.

dropdb --if-exists scraper_test
createdb scraper_test

psql scraper_test --variable body="'content'" \
		  --variable ts='now()' \
		  <<EOF
$QUERY
$QUERY
EOF

echo \\nContent changed...\\n

psql scraper_test --variable body="'changed'" \
		  --variable ts='now()' \
		  <<EOF
$QUERY
EOF

CREATE TABLE
CREATE EXTENSION
 id |  hash  |  body   |             seen_at
----+--------+---------+----------------------------------
  1 | \x0... | content | {"2016-04-20 11:05:31.38508+02"}
(1 row)

INSERT 0 1
CREATE TABLE
CREATE EXTENSION
 id |  hash  |  body   |                             seen_at
----+--------+---------+------------------------------------------------------------------
  1 | \x0... | content | {"2016-04-20 11:05:31.38508+02","2016-04-20 11:05:31.386758+02"}
(1 row)

INSERT 0 1

Content changed...

CREATE TABLE
CREATE EXTENSION
 id |  hash  |  body   |              seen_at
----+--------+---------+-----------------------------------
  3 | \x3... | changed | {"2016-04-20 11:05:31.397775+02"}
(1 row)

INSERT 0 1

Results

The scraper has been running flawlessly in production for almost two weeks. Let’s see how many rows have been added during that time.

SELECT count(*) FROM scraps

After running every 10 minutes for almost two weeks, the scraper inserted just 24 records. Before we inspect the contents of the database, let’s make sure that we really are in compliance with the Heroku PostgreSQL maximum rows limitation:

We have to unnest the seen_at array to obtain the total count of scraper runs.

SELECT count(*) FROM (SELECT unnest(seen_at) FROM scraps) un

Let’s break the 2282 rows down by date and aggregate the count of checks and changes of content that occured that day. Please check out the appendix for the exact query that I used.

    day     | checks | changes
------------+--------+---------
 2016-04-04 |     87 |       0
 2016-04-05 |    144 |       1
 2016-04-06 |    144 |       0
 2016-04-07 |    139 |       3
 2016-04-08 |    144 |       4
 2016-04-09 |    144 |       0
 2016-04-10 |    144 |       0
 2016-04-11 |    141 |       2
 2016-04-12 |    144 |       4
 2016-04-13 |    144 |       1
 2016-04-14 |    138 |       3
 2016-04-15 |    139 |       0
 2016-04-16 |    144 |       0
 2016-04-17 |    144 |       0
 2016-04-18 |    143 |       4
 2016-04-19 |    144 |       2
 2016-04-20 |     55 |       0
 Total:     |   2282 |      24

The sums of checks and changes match the unnested and total counts above, respectively. Note: the amount of checks differs between days probably because of the following:

Scheduler is a best-effort service. There is no guarantee that jobs will execute at their scheduled time, or at all. Scheduler has a known issue whereby scheduled processes are occasionally skipped.

Source: Known issues and alternatives; Heroku Scheduler documentation

Discussion

This scraping method only works when the website is static and the content changes slowly, compared to dynamic websites with different HTML output on each request. For example, some websites return a different XSRF token at every visit. In that case, every scraping run inserts a new row into the database, negating the savings of our UPSERT horizontal expansion.

Conclusion

The HTML output of our targeted website only changed when the content of interest changed, leading to our high checks vs. changes ratio. Using UPSERT for scraping turned out to be a good fit for this website because it enables us to scrape for some time ahead, while logging all of the captured data.

Acknowledgements

Thank you people that made Org mode and Babel. :-)

Appendix

The source code of this article is available online.

Breakdown query

WITH base AS (SELECT
  date_trunc('day', seen_at)::date::text AS day,
  count(*) checks,
  count(DISTINCT body) - 1 changes
FROM (SELECT unnest(seen_at) seen_at, body FROM scraps) un
GROUP BY day)

SELECT * FROM base

UNION

SELECT 'Total:', sum(checks), sum(changes) FROM base

ORDER BY day

Software used

PostgreSQL

psql postgres --tuples-only -c 'SELECT version()'

PostgreSQL 9.5.2 on x86_64-apple-darwin15.4.0, compiled by Apple LLVM version 7.3.0 (clang-703.0.29), 64-bit

psql

psql --version

psql (PostgreSQL) 9.5.2

Visualizing 24 hours of medical students cramming anatomy

Tue, 13 Oct 2015 00:00:00 -0300

tl;dr: in the first year of medical school, I built an application that helps fellow students and myself with studying anatomy. The answers of the last exam, submitted by students while revising, have been visualized with Gource. Please check out the YouTube video for the result:

Introduction to the application

For medical students it is inevitable: you have to know all the anatomical terms by heart. The task is easy, but the amount of structures one has to learn is quite intimidating. I remember the feelings of despair that arose while staring at the latin words in the anatomy book. After one brave attempt, my attention span had decided: we needed a better way to study this (one that involves computers).

Building the visualization

This is the exam_date that we will be using for this visualization:

2015-09-21 08:30:00 +02:00

Retrieving the answers

We use one big PostgreSQL query that yields all the answers from the timeframe in the right format. No scripting needed!

-- We use trigram similarity to determine answer correctness
CREATE EXTENSION IF NOT EXISTS pg_trgm;

WITH
  translated_categories AS (
    SELECT
      id,
      CASE name
        WHEN 'Bovenarm' THEN 'Upper arm'
        WHEN 'Bovenbeen' THEN 'Upper leg'
        WHEN 'Elleboog' THEN 'Elbow'
        WHEN 'Enkel' THEN 'Ankle'
        WHEN 'Hals' THEN 'Neck'
        WHEN 'Heupgewricht' THEN 'Hip joint'
        WHEN 'Kniegewricht' THEN 'Knee joint'
        WHEN 'Onderarm' THEN 'Lower arm'
        WHEN 'Onderbeen' THEN 'Lower leg'
        WHEN 'Pols' THEN 'Wrist'
        WHEN 'Schouder' THEN 'Shoulder'
        WHEN 'Voet' THEN 'Foot'
        ELSE name
      END AS name
    FROM categories
  ),
  answer_colors AS (
    SELECT
      step::float / 10 AS similarity,
      CASE step
        WHEN 10 THEN '00FF00' -- Green: 100% correct answer
        WHEN 9 THEN '32FF00'
        WHEN 8 THEN '65FF00'
        WHEN 7 THEN '99FF00'
        WHEN 6 THEN 'CCFF00'
        WHEN 5 THEN 'FFFF00' -- Yellow: meh
        WHEN 4 THEN 'FFCC00'
        WHEN 3 THEN 'FF9900'
        WHEN 2 THEN 'FF6600'
        WHEN 1 THEN 'FF3200'
        WHEN 0 THEN 'FF0000' -- Red: wrong answer :-(
      END AS color
    FROM generate_series(10, 0, -1) step
  ),
  ranked_sessions AS (
    SELECT
      session_id,
      rank() OVER (ORDER BY min(created_at)) session_rank
    FROM answers
    WHERE session_id IS NOT NULL
      AND created_at
        BETWEEN timestamp :exam_date - interval '28 hours'
        AND :exam_date
    GROUP BY session_id
  ),
  plate_numbers AS (
    SELECT
      id,
      category_id,
      rank() OVER (PARTITION BY category_id ORDER BY plates.id)
    FROM plates
  )

SELECT
  round(extract(epoch from answers.created_at)),
  session_rank,
  -- Green 'beam' (A) when the answer is 100% correct
  CASE similarity(answers.input, structures.name)
    WHEN 1 THEN 'A'
    ELSE 'M'
  END,
  concat_ws('/',
    translated_categories.name,
    plate_numbers.rank,
    structures.name,
    rank() OVER (PARTITION BY structure_id ORDER BY answers)
  ),
  color
FROM answers
JOIN structures ON structures.id = structure_id
JOIN answer_colors
  ON answer_colors.similarity =
     round(similarity(input, structures.name)::numeric, 1)
JOIN plate_numbers ON plate_numbers.id = plate_id
JOIN ranked_sessions rs ON rs.session_id = answers.session_id
JOIN translated_categories ON translated_categories.id = category_id
WHERE answers.created_at
  BETWEEN timestamp with time zone :exam_date - interval '28 hours'
  AND timestamp with time zone :exam_date
ORDER BY answers

The results from the query seem to match Gource’s custom log format:

head -n 3 $ANSWERS_PATH
echo
tail -n 3 $ANSWERS_PATH

1442729360|1|A|Knee joint/1/meniscus medialis/1|00FF00
1442729371|1|A|Knee joint/1/lig. cruciatum posterior/1|00FF00
1442729377|1|A|Knee joint/1/meniscus lateralis/1|00FF00

1442815272|271|M|Upper leg/4/m. biceps femoris caput longum/75|32FF00
1442815283|271|A|Upper leg/4/m. vastus lateralis/75|00FF00
1442815322|271|M|Upper leg/4/m. peroneus longus/71|FF3200

How many answers do we have in total?

wc -l < $ANSWERS_PATH

Captions

tail -n 3 $CAPTIONS_PATH

1442809800|2 hours until exam
1442813400|1 hour until exam
1442817000|Exam begins at 08:30...

User images

Retrieving user agent data per session (rank)

SELECT
  rank() OVER (ORDER BY min(created_at)) session_rank,
  user_agent,
  min(id) first_id,
  min(created_at) session_start
FROM answers
WHERE session_id IS NOT NULL
AND answers.created_at
  BETWEEN timestamp with time zone :exam_date - interval '28 hours'
  AND timestamp with time zone :exam_date
GROUP BY session_id, user_agent

1|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12|1465360|2015-09-20 06:09:19.603637
2|Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0|1465384|2015-09-20 06:19:55.221907
3|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9|1465408|2015-09-20 06:28:14.890441

Linking the sessions to browser icons

ls -l $USER_IMAGES_PATH/{1,2,3}.png | cut -d/ -f4-

1.png -> /Users/pepijn/Desktop/browser_icons/Safari.png
2.png -> /Users/pepijn/Desktop/browser_icons/Firefox.png
3.png -> /Users/pepijn/Desktop/browser_icons/Safari.png

Putting it all together

time (gource -1280x720 \
	     --bloom-intensity 0.7 \
	     --caption-duration 15 \
	     --caption-file $CAPTIONS_PATH \
	     --caption-size 50 \
	     --dir-colour 00FFFF \
	     --dir-name-depth 2 \
	     --file-idle-time 10 \
	     --hide filenames \
	     --highlight-dirs \
	     --max-file-lag -1 \
	     --seconds-per-day 10000 \
	     --stop-at-end \
	     --title 'Answers from AMC/UvA (Amsterdam) 3rd year medical students revising online the day before their orthopaedics (course 3.1) anatomy exam' \
	     --user-image-dir $USER_IMAGES_PATH \
	     $ANSWERS_PATH 2>/dev/null) \
     2>&1

Building a GeoJSON travel log: an introduction to Org mode and Babel

Sun, 04 Oct 2015 00:00:00 -0300

Literate programming is a technique that caught my attention after recently stumbling upon Howard Abrams’ ‘Literate Devops with Emacs’ video. The intersection of watching this awesome video, reading about GeoJSON rendering on GitHub and returning from a road trip last summer led me to building my own travel log. In my first blog post I would like to show you how it works.

Creating a travel log means entering a lot of data like dates and locations; a job that can be tedious without a user-friendly interface. It shouldn’t be too hard to beat exisiting online travel log services, such as the well-known Dutch website ‘WaarBenJij.nu’, in this aspect. Besides, what happens to your data when the online travel log company goes bankrupt? These two shortcomings are easily addressed with common programmer tools like Emacs’ Org mode and distributed version control systems like Git. While most programmers are familiar with the features of Git and GitHub, those of Org mode are less-known:

“Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.” — http://orgmode.org

In addition to Org mode, we use the Babel extension to execute source code in various languages (in this case just Shell and Ruby) in between the blog post paragraphs. By using these blocks exclusively, we will create all the code necessary to: geocode the locations in the travel log to coordinates (and install a library that helps us do this), convert the travel log to a GeoJSON file, commit and push it into a new repository, and open GitHub in a browser at the right URL. In fact, you can copy & paste this blog post into Emacs, enter org-mode and execute everything (org-babel-execute-buffer) to reproduce my steps.

Enough talking, let’s build this thing! We begin by defining the travel log, and ‘store it in a variable’ called travel-log.

#+NAME: travel-log
#+RESULTS:
| Date             | Location                  |
|------------------+---------------------------|
| <2015-08-10 Mon> | Utrecht, The Netherlands  |
| <2015-08-10 Mon> | Kožná, Prague             |
| <2015-08-12 Wed> | Prenzlauer Berg, Berlin   |
| <2015-08-13 Thu> | A&O Hamburg City, Hamburg |
| <2015-08-14 Fri> | Utrecht, The Netherlands  |

The brackets (< and >) around the dates indicate an Org mode timestamp. We can easily add and manipulate dates by using the datepicker (C-c ., which, for those unfamiliar with Emacs, means pressing the Control and C keys simultaneously before hitting the dot) and use TAB to move through the table: a user-friendly interface.

In order to geocode the location names to coordinates we will use the geocoder Ruby gem. A shell source code block is an excellent way to install it, most importantly because the output displays the version that I used while writing the blog post, which improves reproducibility.

#+HEADER: :results output
#+BEGIN_SRC sh

gem install geocoder

#+END_SRC

#+RESULTS:
: Successfully installed geocoder-1.2.11
: Parsing documentation for geocoder-1.2.11
: Done installing documentation for geocoder after 1 seconds
: 1 gem installed

Now we will geocode the locations using the gem above. We don’t want to get rate-limited by the Google Maps API, so that’s why we create the geolocation-cache table with the distinct locations and their coordinates. For instance, Utrecht is listed twice in the travel log but only geocoded once. By the way, the #+HEADER: and #+BEGIN_SRC lines are instructions to Babel. I included them to enhance reproducibility.

#+HEADER: :var travel_log=travel-log
#+BEGIN_SRC ruby

locations = travel_log.map do |entry|
  # We only need the second column: the location
  _, location = entry
  location
end

distinct_locations = locations.uniq

require 'geocoder'
distinct_locations.map do |location|
  geo = Geocoder.search location
  coordinates = geo.first.geometry['location']

  [location, coordinates['lng'], coordinates['lat']]
end
#+END_SRC

#+NAME: geolocation-cache
#+RESULTS:
| Utrecht, The Netherlands  |  5.1214201 | 52.09073739999999 |
| Kožná, Prague             | 14.4213456 |        50.0862754 |
| Prenzlauer Berg, Berlin   |   13.44009 |          52.54114 |
| A&O Hamburg City, Hamburg |  9.9936818 |        53.5510846 |

Before we move on to the GeoJSON conversion, we have to specify a path where we can save the file.

#+NAME: geojson-file-path
#+RESULTS:
: /tmp/example_travel_log/my_trip.geojson

The following source block joins the travel-log with the geolocation-cache, builds a GeoJSON-formatted structure and saves it to the geojson-file-path.

#+HEADER: :results silent
#+HEADER: :var geojson_file_path=geojson-file-path
#+HEADER: :var geolocation_cache=geolocation-cache
#+HEADER: :var travel_log=travel-log
#+BEGIN_SRC ruby

coordinates = Hash[geolocation_cache.map do |entry|
  location, longitude, latitude = entry
  [location, [longitude, latitude]]
end]

require 'date'
geojson_features = []

# We use each_cons so we can draw lines between the locations
travel_log << nil
travel_log.each_cons(2) do |entry, next_entry|
  org_date, location = entry

  date = Date.parse org_date

  geojson_features << {
    type: 'Feature',
    geometry: {
      type: 'Point',
      coordinates: coordinates[location]
    },
    properties: {
      Location: location,
      Date: date
    }
  }

  next unless next_entry
  _, next_loc = next_entry
  geojson_features << {
    type: 'Feature',
    geometry: {
      type: 'LineString',
      coordinates: [coordinates[location], coordinates[next_loc]]
    }
  }
end

require 'fileutils'
repository_dir = File.dirname geojson_file_path
FileUtils.mkdir_p repository_dir

require 'json'
open(geojson_file_path, 'w') do |file|
  file.write JSON.pretty_generate(
    type: 'FeatureCollection',
    features: geojson_features
  )
end
#+END_SRC

Let’s verify the contents of the newly created file before we move on.

#+HEADER: :results output
#+HEADER: :var GEOJSON_FILE_PATH=geojson-file-path
#+BEGIN_SRC sh
head -n 33 '' $GEOJSON_FILE_PATH
echo etc...
#+END_SRC

#+RESULTS:
#+begin_example
==> /tmp/example_travel_log/my_trip.geojson <==
{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          5.1214201,
          52.09073739999999
        ]
      },
      "properties": {
        "Location": "Utrecht, The Netherlands",
        "Date": "2015-08-10"
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            5.1214201,
            52.09073739999999
          ],
          [
            14.4213456,
            50.0862754
          ]
        ]
      }
    },
etc...
#+end_example

The date, location and coordinates seem to match with the travel log we specified earlier. Let’s create a repository, commit, push and open GitHub to check it out!

#+HEADER: :results output silent
#+HEADER: :var GEOJSON_FILE_PATH=geojson-file-path
#+BEGIN_SRC sh
cd "$(dirname $GEOJSON_FILE_PATH)"

FILENAME="$(basename $GEOJSON_FILE_PATH)"

brew install hub
hub init
hub create
hub add $FILENAME
hub commit -m 'Update travel_log' $FILENAME
hub push origin master

GH_PATH_ROOT="$(hub remote -v | grep fetch | grep -oE '\w+\/\w+')"
hub browse "$GH_PATH_ROOT/blob/master/$FILENAME"
#+END_SRC

*Safari opens…*

It works; all the destinations from the travel log show up on the map with lines connecting them. Whenever we change the travel log or the code we can just rerun all the source code blocks and check out the new result; no slow switches between editors and command lines. All in all, the consequential higher speed of development and improved transparency of the program will most definitely make me pick Org mode and Babel for future projects.

For a more feature complete implementation that might actually be useful for real-life travel logging, please check out: https://github.com/pepijn/travel_log. Manipulating it (adding a destination) looks like this:

Happy hacking!

Pepijn Looije

Real World Ledger part 1: Weighing Eggs in Baskets

Introduction

Part 1: Weighing Eggs in Baskets

The basics: a single currency and a single asset class

Our first posting: adding our savings account

Our first mutation: interest from savings, and deposits and withdrawals

Going deeper: multiple currencies and asset classes

Diversifying into multiple assets

Implicit and explicit market prices

Conclusion

Feel free to subscribe: I'll notify you when publish the next part

Appendix

Footnotes

Sidestepping Heroku's limits with PostgreSQL 9.5's UPSERT

Introduction

Choosing the right storage

Heroku limitations

Methods

The query

The test

Results

Discussion

Conclusion

Acknowledgements

Appendix

Breakdown query

Software used

PostgreSQL

psql

Visualizing 24 hours of medical students cramming anatomy

Introduction to the application

Building the visualization

Retrieving the answers

Captions

User images

Retrieving user agent data per session (rank)

Linking the sessions to browser icons

Putting it all together

Building a GeoJSON travel log: an introduction to Org mode and Babel