See also: WikiSpace, DreamWiki, WikiAsPim, WikiToDrupal, PersonalDataRepository, AsWeMayThink, VannevarBush, MyLifeBits

I have been talking a lot to "Name Deleted" about his concept of a Digital Palace, or as he says "A place to keep your stuff". As we've been talking I've discovered that there are a lot of parallels between what he's talking about and what I've been quiety working towards with the WikiAsPim ideas and other such discussions all over the web.

Here's an email I recently composed to him about what my ideal system would look like, it's a first draft and hopefully I'll find the time to go through and organize it more fully in the future. As best as I can tell this is a really hard problem and no one has really invested a lot of resources into determining what the best technology to solve it is or what people actually want from such a solution (ie. I have a lot of ideas but as I've never gotten to actually implement any of them I could be way off base on what's actually useful or desirable).

An other big concern is obviously security. As you start to collect all your "stuff" into a single location you need to be careful about whose hands it could fall into, either accidentally or through deliberate subversion of your systems. This gets harder as you start to factor in off site backups, redudancy and reliability issues and especially when you start to share your data with other people.

Anyway, here's my inital "best guess" at what I view my requirements as (from an email). -- AdamShand

So to answer some of your other questions and to try and consolidate the emails.

The first thing to do is realize my personal goal. I'm not immediately after a secure solution that I can share with others. My current goal is a solution which will allow me to manage my own content in a way which isn't blatently *insecure* and helps me deal with my daily overdose of information (ie. it should make my life easier). I do not need, or particularly want to manage a digital vault right now.

Second, my interest in collobaration is mostly in the sense that getting other people involved in helps me to keep my data up to date and can provide a service to my friends. If it can provide a service to the public as well, so much the better, but that's not the immediate goal.

I have great interest in making my data as redundant as possible. This is why I want an online data source that I can sync with a PDA. This obviously limits the type of data I will be storing (it would be nice to store a/v data but it's *way* low on my priority list).

I care more about accessibilty/reliability then security. I would rather host this all on one server (that I control) and know that I can always get to my data via web/imap/shell/pda/??. I know that this is a potential point of risk and I'm willing to deal with that until a solution which meets my needs is available (maybe dPbridge will do that).

I am willing to accept net access as requirement to get to my data so long as options like syncing with a PDA are available for limited offline access.

I want all of my data going to this. All my email, all my contacts, all the web sites I browse etc. I believe I have the technology to implement this *now* ... I just need to do it, and prove to myself that it's reliable.

The ideal solution from a securiyt point of view should distribute the data across multiple computers, in an encrypted and redundant form. The data should still be searchable and retrievable via the net. Given that the ideal solution is very difficult to provision I would accept encrypted data on a single server with off site backups as a long term workable goal.

I am pretty much exclusely interested in an OpenSource solution for a variety of reasons.

I have no interest in having to manually maintain categories and I insist upon a flexible categorization scheme (eg. items should be able to belong to multiple categories and categories should be able, but not required, to be listed in a heirarchical manner).

Eventually authorization will be implicit in categorization. I like the way Drupal (http://www.drupal.org/) manages this with what it calls taxonomy. I could define a taxonomy like this (very simple): 

- security 
-- private
-- unprivate
-- public
- places
-- portland
-- anchorage
- people
-- graham
-- adam
-- teresa
- organizations
-- earthlight
-- internet alaska
-- pixelworks
-- personal telco

(Categories should also be able to be virtual and basically "link" to another category, eg in case an organization changes it's name).

Then when I create a new contact for my boss it might belong to the "unprivate, portland and pixelworks" taxonomy categories. You can then assign authorization rights to any of those categories so only someone who's personal login was in the Pixelworks category could view data in the pixelworks category (and only if the category was unprivate or lower).

Setting up a service like imap would just provide a different way of accessing data, especially accessing data online (just sync your folders). you use a user/pass to authenticate and the built in heirarchy of the taxonomy to authorize.

So there ... that's what i'm working on, what i consider acceptable and where i'm moving towards. ideally this would all combine with wireless access and a wearable computer so that the majority of the time my data is actually on my person with backups elsewhere. currently however i haven't found a computer that i'd actually be willing to wear.

I know this is pretty scattered and somewhat self contradictory but it's important to get as much as possible out on the table for discussion. The final vision I'd like to espouse is that really want I want is a protocol neutral data store which can categorize content with a system like Drupal's taxonomy. Once the data store exists everything else is just a matter of accessing the data. For example I would like to have all my contact information but I want to access it in multiple ways:

Hrm, it's starting to sound like what I want is an object database isn't it? (Note: Check out the Zope Object Database -- Adam 24 December 2002)

So this may not be useful to anyone else, but I think I just clarified what I want and I'll try and rewrite this in the next few days.

Adam.

Round two of thoughts is below. Again the idea of a taxonomy based data store with multiple interfaces seems crucial. I really need to go read up on what other people have researched and implemented now because I'm sure this is an old problem.

Now, on to it not being about the technology. I think that's a good point in theory and a horrible point in practice (though I'm willing to be wrong here). Either that or it again comes down to us both being after quite different things that happen to be confusingly similar :)

I am not after digitizing everything I own so I can have a digital home, I think it's an interesting goal and idea it's not something I have an overt interest in doing myself right now.

What I want is a way to manage my digital data. One of my specialties is knowing where to find information (see IntelligentAgent) and I have lots of little systems for keeping track of information that may be useful to myself or others later. The problem is that as I get older my brain is no longer able to keep up with the extent of the information that I need to categorize and my systems are starting to fail. Basically I have two options, stop worrying about data loss or build better tools.

Our conversation has been useful to me because you've thought extensively about parts of it (especially authorization and security) that I haven't really addressed yet because my focus right now is more on personal use and fuctionality because I need something *now* to solve my problem.

Not only do I need something soon to solve my problem, but I need a system that is highly automated. I don't have time to manually sort/filter process information. I want something which happens automatically in the background as much as possible, and when I do need to interface to it it's as easy as possible.

For example, here's are some types of data that I want to be able to store.

I want to be able to search, access, add/remove to all of the above via a web interface because it's the lowest common denominator that is always available.

I want to be able to search, access, add/remove my contact and email data from my email client (this probably means an imap and ldap interface to that data).

I need to be able to search, access, add/remove my bookmarks via a web browser, ideally via it's native bookmark management tools since web interfaces for this are typically pretty clunky. Further I'd like to improve on browser methods of managing bookmarks and snippets of information, for an idea see Queue.

I would like to be able to keep my writing in it as a blog or diary. Same for my word up posts, how do you reference articles of information that you've emailed out?

It should all be cross referenced automatically. For example I should be able to read an email from you and click on "More about Graham" and see all the contacts, pictures, bookmarks etc that I consider relevant to you, or do the same thing for all the info relevant to Portland or EarthLight. (Note: Zoe InterTwingle aready does a significant portion of this).

It should to be able to be sufficiently secure for me to feel comfortable about my privacy, it needs to be capable of being used off line (eg. sync to laptop or pda) etc.

So you see that what I'm after is more/different then a home for all my stuff. I'm after an agmentation to Adam's brain, primarily for use by Adam ... though it would be nice to have the ability to share with people parts that are relevant, especially if it could be done automatically (eg. you automatically have access to everything to everything I've tagged "Graham Perry" and "unprivate") thus you automatically gain access to everything I think you'd be interested in, and then you should further be able to categorize, enhance, annotate that data.

There's the potential that if everyone started to run such software and there were protocols for exchanging this type of data (XML-RPC might be a good choice) that information could flow between us automatically and that would become exponentially more powerful the more people that used it.

Anyway, maybe that clarifies some more of what I've been thinking.

Again, this became more of a brain dump then I'd originally intended so i'm putting it on the wiki (same page).

An interesting addition to this is a personal webproxy or wayback machine. There are definately some interesting security concerns once you start to track and store not only what you conciously decide to to log but are intentionly logging everything you do.

Recent updates about InterTwingle is amazing ... *this* is the way it should work ... but nothing really does.

CategoryWriting

DigitalPalace (last edited 2004-11-07 22:45:01 by AdamShand)