Ben Eisenbraun
Email: bene at klatsch dot org
<ramble>
I need a new brain. Or maybe just a bigger one. Or maybe one that can see connections between piles and piles of arbitrary input data that I wouldn't otherwise see. One that is almost as accessible as the one I currently have... What I need is some sort of personal digital librarian/archivist/researcher/statistician/assistant.
I've been using email for this purpose, but it's becoming clear that it's not up to the task. I sent about 6000 emails over the last two years: almost 1/2 of them were to myself. Email has some clear advantages for this purpose provided you're not quite so overloaded.
Pros:
- text file data storage (easy search and backup, good compression for longterm storage)
- accessible everywhere (with a little work--See the bottom of the page)
- powerful searching (if you've got a good mail client)
- easy/cheap/simple/low overhead (not sure how to describe this, but simple is probably best)
- a lot of the information I want to keep comes in via email (this is changing)
Cons:
- Stores data in only two ways: emails that come in and emails that I create (see Input Data)
- No easy way to publish select data
- All categorization/fuzzy matching is manual
- No markup/pretty printing for self-written stuff
I've been thinking about the requirements for this software in terms of data inputs: where the data is coming from. More and more I'm finding that I want something that will store data from all the multiple sources in my life: email, IRC/instant messaging, web and things I create. Those are the primary sources for me, but there are others (I used to be a Usenet junkie, but that has mostly stopped).
Input Data Requirements
- instant messaging/IRC
- knowledgebase/self-publishing (includes addressbook and similar collections)
- web proxy/bookmarks
Data Handling Requirements
- searching
- rebuilt-nightly type of dbs for fast common searches
- right-now text searches on keys with regexes
- backup ability
- paring function based on age/last access tunable by category
Access Requirements
- everywhere would be best, but...
- requirement for net connection is acceptable, provided there is some sort of offline synching capability
- favorite client, cheap client connection options (see my current setup below) == not being tied to a particular client for data input vector.
What I've Got Now
I have a colocated FreeBSD web/email/DNS/shell server. I run Postfix as the MTA and write emails to Maildir format. I use Mutt as my primary mail client (aka, my favorite client) and Squirrelmail as a web client (aka, a cheap client). Pretty much the only time I use Squirrelmail is when I don't have access to the shell, or occasionally to view an image file someone has sent me. I also have Courier-IMAP running, so when I'm anticipating being away from regular net access, I can sync Apple Mail on my iBook with the server and head off.
All in all, this works pretty well for access to email.
More to come.
Adam's done a lot of thinking about most of these things, but I think we have slightly different visions of what we want.
</ramble>
See also: DigitalPalace, PersonalDataRepository
InformationClients are one way to manage this. The big problem I see with IMAP is that the only way to inject new information into the system is via SMTP, which is pretty annoying. For searching it would be easy to setup a web based search engine for your Maildir mailboxes, in fact this was one of the reasons I migrated to Postfix/Maildir with the MausRoninMigration. The big hurdle in my mind is that replicatiion, multi-access (eg. web, shell, GUI) and intertwingling data are all vital to the end result. However that's also a pretty tall order and I've not come across any one thing that will do what I want, let alone do it well. I think I've pretty much come to the conclusion that no one is every going to write what I want and so I'm trying to get my goals very clear in my head and build small pieces of the solution in such a way that they can be intertwingled later. Thanks for bringing your thoughts ... I appreciate it 
-- AdamShand