Building personal search infrastructure for your knowledge and code

Having information in the digital form, collecting and writing notes is incredibly valuable. Our brains are good at associations, pattern matching and creative thinking, not storing arrays of structured data, and external memory is one of the main thinking hacks computers aid us with.

However this information is not so useful if you can't access and search it quickly. Instant search changes the way you think. Ever got sense of flow while working through some problem, and trying different things from Stackoverflow or documentation?

These days, if you have decent connection, you are seconds away from finding almost any public knowledge in the internet. However, there is another aspect of information: personal and specific to your needs, work and hobbies. It's your todo list, your private notes, books you are reading. Of course, it's not that well integrated with the outside world, hence the tooling and experience of interacting with it is very different.

Some examples:

  • To find something from my Messenger history with a friend, I need to be online, open Facebook, navigate to search and use the interface Facebook's employees thought convenient (spoiler: it sucks)

    It's my information, something that came out from my brain. Why can't I have it available anywhere, anytime, presented the way I prefer?

  • To find something in my Kobo ebook, I need to reach my device physically and type the query using the virtual keyboard (yep, e-ink lag!). Not a very pleasant experience.

    It's something I own and have read. Why does it have to be so hard?

Such things are pretty frustrating to me, so I've been working on making them easier. Search has to be incremental, fast and as convenient to use as possible. I'll be sharing some of workflows, tricks and thoughts in this post.

The post is geared towards using Emacs and Org-mode, but hopefully you'll find some useful tricks for your current tools and workflow even if you don't. There is (almost) nothing inherently special about Emacs, I'm sure you can achieve similar workflows in other modern text editors given they are flexible enough.

Note: throughout the post I will link to my emacs config snippets. To prevent code references from staling, I use permalinks, but check master branch as well in case of patches or more comments in code.