Curl a Page

We share pages by fetching them in json format over the open internet. Here we will look into a few pages to get a better idea of how they are made.

We'll use unix command line tools to perform experiments. Curl fetches pages, jq parses json, wc counts words, sort sorts files, uniq counts duplicates.

A slug is a page title squeezed into mostly lower case. SLUG=about-federated-wiki

A page has a story made up of items of various types. Here we use paragraph for plain text and markdown for emphasis. Our versions of both use the same square bracket notation for links.

curl $SITE/$SLUG.json | \ jq '.story[].type'

"paragraph" "paragraph" "paragraph" "markdown" "paragraph" "paragraph" "markdown" "paragraph" "paragraph" "paragraph" "markdown" "paragraph" "paragraph" "paragraph" "paragraph"

A more complicated page has more various types. Reduce aggregates calculations from other pages.


"paragraph" "reduce" "paragraph" "paragraph" "html" "paragraph" "html" "html" "html" "pagefold" "paragraph" "paragraph" "paragraph" "pagefold" "paragraph" "html" "paragraph"

Each page has a journal that is made up of actions, also with types, but drawn from a much smaller and fixed set.

curl $SITE/$SLUG.json | \ jq '.jornal[].type'

"create" "add" "edit" "add" "edit" "edit" "edit" "edit" "add" "edit" "add" "add" "add" "add" "add" "add" "add" "edit" "move" "add" "edit" "move" "edit" "add" "move" "edit" "add" "edit" "add" "edit" "add" "edit" "add" "edit" "add" "edit" "edit" "edit" "edit" "remove" "edit" "edit" "add" "edit" "edit" "edit" "edit" "edit" "edit" "edit" "edit" "edit" "edit"

Here tallied with unix sort and uniq.

... | sort | uniq -c | sort -n

1 "create" 1 "remove" 3 "move" 18 "add" 30 "edit"

See Curl this Site to see how pages fit into a larger whole.