How to build a Personal Document Management System (PDMS) and why you need it

My personal experience

Evernote

I’ve spent many years building a lifelong personal library. A couple of years back, the first time I tried to manage my Ebooks is using Evernote. I saved every single document into Evernote, from diaries to notes, from lecture notes to reference books.

However, when my notes number larger than 1000, it is very difficult for me to locate the documents.

Finally, I found Evernote really sucks on document management. It may be good for information storage but definitely not for document management. Even the advanced search feature still sucks, and the note size also has limitations.

File Explorer

Then I came back to the old but effective way, using the File Explorer and making all the documents nested inside the well-structured folder structure.

But wait, the problem is still there, once my documents getting more and more, how can I search them in a second? And most importantly, how can I manage them, classify them, and even reuse/reference them when I needed. Besides, How can I save the web pages for reading them later, how can I save the online videos and never lost them?

Let’s talk about Document Management

DMS (Document Management System)

Document management is very common in Business, there are also many Document management system (DMS) out there, the purpose of such systems are to receive, track, versions, manage and store documents and reduce paper.

PDMS (Personal Document Management System)

Personal documents have not much attention before, there may be some solutions, such as Evernote, Devonthink, etc. But they have many drawbacks:

1. You have to pay for using them
2. You may need to save your content to the software providers, no one can promise you they will help you to save them permanently.
3. Lots of limitations and rely on managing the software itself rather than managing our content.

Do you have some ideas but do not know how to structure them and make values from them? Do you want to save the documents in a single place and locate them as fast as possible? Do you have too many documents and have no way to manage them properly? Do you want to save the web pages permanently and read them without relying on internet services? Do you want to save your favorite online audio or video a life and manage them easily??

Then you need to build a Personal Document Management System.

In the section of Capturing, I will show you how to capturing and structuring your ideas with Mindmap and Org-mode.

Then, in the section of Archive, I will show you how to build your PDMS with some open sources tools, and you do not need to worry about the prices or how long you can keep on using them, since they are free for life, and all data are saved locally, permanently, easily be managed within a single index database.

Two Steps to Build a Personal Document Management System

Capturing

The first step is capturing the ideas generated from your life.

Mindmap

Mindmap is such a great tool to visually organize information and brainstorm.

Boeing used the mind map when designing the Boeing 747. According to Boeing people, if you use the usual method, it takes six years to design a large project like the Boeing 747. However, by using mind maps, their engineers completed the Boeing 747 design in just six months and saved $10 million…

Mindmapping is nothing complicated and just 5 steps, you just need a pen and a paper, then you can record and integration of divergent thinking at any time.

1. Start with a Central idea.
2. Draw four (or more) branches.
3. Draw (sub)branches clock-wise.
4. One short word/sentence one branch.
5. Color the branches outline.

I prefer drawing by hand, either with a real pen or e-pen on an iPad or asurface, since I really enjoy the “Coloring”, it is even a great therapy to make me peaceful, generating more ideas.

Here is a Mindmap I drew before — What a Raspberry Pi Can Do:

Outline and org-mode

Outline is another format of Mind-mapping, but from top to bottom. It is possible to keep the content of large files well-structured, visually cycling, and editing between trees. It is so simple to work with. And it also has all mindmap benefits, such s information organization, visual management, brainstorming, well-structured, etc.

If you want to make your ideas values, try org-mode, it is the best editing format out there, simple as Markdown, but way more powerful than that.

Here are some benefits:

1. org-mode can generate any formats that you want
2. org-mode is the best option for Writers/Editors/Researchers/Bloggers. Bloggers can use hugo and ox-hugo, to easily generate the whole blog website, without managing the web, just focus on the writing org files.
3. Run source codes alive with org-babel. Inside org-mode, you can insert source code blocks, run them inside your text file, and output the results in the same file. Unlike the jupyter-notebook, org-mode supports numerous possible languages (C, Java, Python, Ruby, R, …).
4. Mouse interaction is optional, it is so natural to just press a single “TAB” key, you can hide/show/cycle the all sections, focusing on what you are editing.
5. Images previewing, drag and drop.
6. Recursively search with rg.
7. Best Evernote/OneNote replacement, with deft, and cloud synchronization with any cloud providers.
8. Capturing ideas with org-capture, it provides a great tempting system, no panic of any interruption!
9. Tons of extensions.

Archive

Build a library index

The best way to manage the documents is to manage their metadata, in an index database.

You have a ton of resources you do not want to lose, for example,

1. Ebooks
2. Papers
3. Read It later web pages
4. Documents
5. Technical Documents
6. YouTube’s audios and videos

Building a library index is also simple with just some great open source tools.

• Ebooks/Documents

  The first solution you may know is calibre GUI, maybe you also try to use it, and found it is so frustrating.

  Here are some reasons:

  1. You may not know how to manage the documents properly.
  2. You may not realize the benefits of managing the ebooks and any other documents.
  3. Many front-ends alternatives, not just the official one.
  4. Tools are optional, the idea behind it is precious.

  Don’t be upset! It is such a great tool to provides us a refreshing way to see how our personal library should look like — An Index Table.

  Build a document index is the key of build a lifelong personal library.

  1. The index is small and portable to keep.
  2. The index is easy to be modified.
  3. The index is independent of the documents.
  4. You can find the documents in a second.
  5. You do not need to rely on internet service providers.
  6. Everything is free, you do not need to worry about the money.
  7. No privacy issue, all in local, within one index table.
  8. You can keep it a life.

  • calibre

    The most important thing to do is of course install calibre. It can help us to set up the Calibre Database and Calibre Library which we will work on it later without the calibre GUI.

    Calibre Database is a database file called metadata.db generated by calibre, the Calibre Library is the directory that holds your documents added into the Calibre Database.

    

  • calibre-web

    Setup calibre-web to access your Calibre Library at any time.

    PS: You can set up a telegram bot in Raspberry Pi or directly on your phone with Termux.

  • calibredb & calibredb.el

    Calibredb is the command-line tool that comes with Calibre, we will use it to create more possibilities.

    • Telegram Bot with calibredb

      I created a project — Pibot. You can send documents to it, it will save the documents to Calibre Library using calibredb in the background. The documents will be saved both on telegram apps and on your Calibre Library.

      

      

      PS: You can set up a telegram bot in Raspberry Pi or directly on your phone with Termux.

    • calibredb.el

      I recommend using calibredb.el, a front-end of calibredb.

      

      1. The dashboard is also the index table.
      2. It is inside Emacs, it works well with org-mode.
      3. You can search them in a second, in real-time.
      4. By building the virtual libraries with searching keywords.
      5. Generating an org file to include all file locations and you can just search within it.
      6. Add/remove/tagging/favorite/archive the items with just pressing a single key.

      Here is the live filter feature:

      

    • Librera Reader

      

      Librera Reader is an open-source E-Reader Application on Android. It has a great feature that it can scan your phones’ directory and show all your documents in the dashboard with preview images. Since Calibre Database is only an index of all your documents, all your documents will be saved under Calibre Library location, Librera Reader does a great job and finds out all documents in it! Of course, the first thing you need to do is to set up a synchronization bridge between your server’s Calibre Library and your phone.

• Web Pages (Read It Later)

  People read online. Sometimes, you find some interesting posts and want to save it forever, how to do it? You may know Pocket, but the contents are saved on Pocket’s server, we can only read our contents within Pocket. Not mention we need to pay for it for premium features, but what if somedays, Pocket is no longer operating, where our contents will go at that time? My purpose to build a lifelong solution.

  With some open-source tools, it becomes so simple.

  Normally, when you find something interesting, you must know its link. What we need to do, is just download the webpage into a EPUB/PDF, sometimes, convert it with pandoc to a preferred format, and save to our Calibre Library. In this way, we can benefit from managing just one index database as well, locally and permanently.

  I created a telegram bot called — Readit. If I find something interesting, I will send the link to Readit, it will automatically download the webpage with percollate, turn it into EPUB/PDF and add it into my Calibre Library. By the way, send back the downloaded file to me in the chat. Since my Calibre Library is synchronized to my phone, I can read the files with Librera Reader as well. Of course, I can read the document in my laptop with calibredb.el or calibre-web. All happen in the background, and all are saved locally, permanently, and of course, free of charge.

  PS: Readit will be open-sourced later. But you can build your own, just with some simple python.

• Audios/Videos

  • Telegram Bot with youtube-dl

    As I mentioned before — Pibot, actually, Pibot is not just for document archiving/searching, it can be also used as a youtube downloader (using youtube-dl), you can turn telegram into mp3 player and archive your favorite youtube music/video locally, forever.

  • Audio/Video Database: Plex Server

    Set up a Plex Server to scan your audio/video directory, you can access the audio/videos with your phones/browser anytime, anywhere.

    Once you send a link to Telegram Bot, pibot will download the audio/videos, and Plex will know audios/videos are added into the directory, then it will automatically be added into Plex Library, so that you can access them just like accessing youtube.

    

    PS: You can set up Plex in Raspberry Pi or with a NAS/router.

Tell me

If you find my ideas are interesting and want me to share more, please share it, tell me or ask me. I am always here to help you.