The sad state of personal data and infrastructure [see within blog graph]

• Why can't I search over all of my personal chat history with a friend, whether it's ICQ logs from 2005 or Whatsapp logs from 2019?
• Why can't I have incremental search over my tweets? Or browser bookmarks? Or over everything I've ever typed/read on the Internet?
• Why can't I search across my watched youtube videos, even though most of them have subtitles hence allowing for full text search?

• Why can't I see the places my friends recommended me on Google maps (or any other maps app)?

  It's just a matter of extracting links to maps from the chat apps and generating a list of coordinates. It's got to be trivial.

• Why can't my Google Home add shopping list items to Google Keep? Let alone other todo-list apps.

  Instead, it puts them in a completely separate product, Shopping list. If any of these had an API, any programmer could write a script to synchronize them in a few hours.

• Why can't I create a task in my todo list or calendar from a conversation on Facebook Messenger/Whatsapp/VK.com/Telegram?

  Often, a friend recommends a book to you so you want it to add to your reading list. Or they ask you for something and you want to schedule a reminder.

  Instead, these apps actively prevent me from using builtin Android share functions (because it means leaving the app presumably).

• Why do I have to lose all my browser history if I decide to switch browsers?

  Even when you switch between major ones like Chrome/Firefox. Let alone for less common alternatives.

• Why can't I see all the places I traveled to on a single map and photos alongside?

  I have location tracking and my photos have GPS and timestamps.

• Why can't I see what my heart rate (i.e. excitement) and speed were side by side with the video I recorded on GoPro while skiing?

  I've used HR tracking and location tracking, surely that's possible?

• Why can't I easily transfer all my books and metadata if I decide to switch from Kindle to PocketBook or vice versa?

• Why can't I see stuff I highlighted on Instapaper as an overlay on top of web page?

  Hypothes.is and plenty other services can do it, so it's a solved problem, right? Does every other annotation service really have to reimplement the algorithm from scratch?

• Why can't I have single 'read it later' list, unifying all things saved on Reddit/Hackernews/Pocket?

• Why can't I use my todo app instead of 'Watch later' playlist on youtube?

  'Watch later' is fine for short videos that I can watch over dinner or on my commute. Longer videos like talks and lectures need proper time commitment hence prioritizing.

• Why can't I 'follow' some user on Hackernews?

  Or any other service that doesn't have this feature implemnted. It's just a matter of regularly fetching new stories/comments by a person and showing new items, right?

• Why can't I see if I've run across a Youtube video because my friend sent me a link months ago?

  The links are there in the chat history, surely it's a trivial task to find it?

• Why can't I have uniform music listening stats based on my Last.fm/iTunes/Bandcamp/Spotify/Youtube?
• Why am I forced to use Spotify's music recommendation algorithm and don't have an option to try something else?
• Why can't I easily see what were the books/music/art recommended by my friends or some specific Twitter/Reddit/Hackernews users?
• [2019-12-15 05:35] Why my otherwise perfect hackernews app for Android doesn't share saved posts/comments with the website?

Aka #quantified-self.

• Why can't I tell if I was more sedentary than usual during the past week and whether I need to compensate by doing a bit more exercise?

  I have all my location (hence step data), so what's the issue?

• Why can't I see what's the impact of aerobic exercise on my resting HR?

  I use HR tracker and sleep tracker, so all the necessary data is there.

• Why can't I have a dashboard for all of my health: food, exercise and sleep to see baselines and trends?

  Why do I need to rely on some startup to implement this and trust them with my data?

• Why can't I see the impact of temperature or CO2 concentration in room on my sleep?

  My sensors have Bluetooth and Android apps, why can't they interact with my sleep data?

• Why can't I see how holidays (as in, not going to work) impact my stress levels?

  It's trivial to infer workdays by using my location data.

• Why can't I take my Headspace app data and see how/if meditation impacts my sleep?
• Why can't I run a short snippet of code and check some random health advice on the Internet against my health data.

• Why am I forced to manually copy transactions from different banking apps into a spreadsheet?
• Why can't I easily match my Amazon/Ebay orders with my bank transactions?

Aka #offline. On one hand it's less and less of an issue as the Internet gets more reliable. On the other if you start relying on it too much, it's becoming more and more of a single point of failure.

• Why when something like 'mind palace' is literally possible with VR technology, we don't see any in use?
• Why can't I easily convert select Instapaper highlights or new foreign words I encountered on my Kindle into Anki flashcards?

• Why do I have to suffer from poor management and design decisions in UI changes, even if the interface is not the main reason I'm using the product?

• Why can't I leave priorities and notes on my saved Reddit/Hackernews items?

  I've got too many saved things to read them linearly and I'll probably never read them all. I've also got other things to read and do in general, why can't I have a unified queue for consuming content?

• Why can't I leave private notes on Deliveroo restaurants/dishes, so I'd remember what to order/not to order next time?

• Why do people have to suffer from Google Inbox shutdown?

  Not to undervalue Inbox developers, but fundamentally it's just a different interface. I'm sure there are plenty of engineers who would happily support it in their spare time if only they had access to the APIs.

• Why can't I easily share my web or book highlights with a friend? Or just make highlights in select books public?
• Why can't I easily find out other person's expertise without interrogating them, just by looking what they read instead?

• Why do I have to think about it and actively invest time and effort?

  What about regular people who have no idea how unreliable computers can be and might find out the hard way?

I think all of this is pretty sad. Note that I haven't mentioned any mad science fiction stuff like tapping directly into the brain (as much as I wish it was possible). All these things are totally doable with the technology we already possess.

I wonder what computing pioneers like Douglas Engelbart (e.g. see Augmenting Human Intellect) or Alan Kay thought/think about it and if they'd share my disappointment. So many years have passed since the computing (and personal computers) spread, and we're still not quite there. And companies are actively promoting these silos.

Imagine if all of this was on your fingertips? If you didn't have to think about how and where to find information and could just access it and interact with it? If you could let computers handle the boring bits of your life and spend time on fun and creative things?

That's where it all starts with and it's a mess.

• easiest scenario: the service lets you generate an API token from its settings and you can just use it. Example: pinboard

• typical scenario: you need to do the whole Oauth thing.

  That involves creating a client app, getting client id, dealing with scopes and redirect urls, etc. Pretty tedious, and you certainly can't expect a nonprogrammer to be able to follow these steps.

  Examples: almost every service with an API out there: Twitter/Instapaper/Pocket/Github/etc.

• worst case scenario: the service doesn't even offer a public API. That also has different grades of horrible:

  • best worst: service uses a private API and you can spy on the token web app is using in browser dev tools.

    Not too bad, but a bit dubious.

    Example: Pocket API doesn't give you away highlights unless you mess with it.

  • typical worst: no private API, so you need to scrape the data. Sometimes you can grab the cookies from browser dev tools and use them to access your data.

    Scraping is orders of magnitude flakier, involves nasty parsing and obviously fragile. Some services might even actively prevent you from doing so by banning unusual user agents.

    Examples: Facebook Messenger, Blinkist

  • worst worst: you need to scrape the data and cookies don't work or expire often.

    Basically means you need to use your username/password. Bonus points if there is 2-factor auth involved.

    Potentially means you're going to store your password somewhere which is way less secure than using a token.

    Example: Google Takeout exports are not only asynchronous, but also don't have an API so you have to login in order to export.

  All the 'worst' scenarios are extremely flaky and basically impossible for nonprogrammers to use.

Whether you're using API or not, typically you'll have to retrieve multiple chunks of data and merge them after.

In principle, it's not hard to implement it on a one off basis, but unclear how to do it in some universal way because there is no common standard.

Pages might be addressed by page numbers and counts, offsets from start/end of data, before or after with respect to ids or timestamps, etc.

It's quite error prone: content might change under your feet, and if the API developers or you are not careful, you might end up with missing data or even some logical corruption.

If you simply start fetching a json and writing to disk, you'd very quickly end up with a corrupt file on the first network failure. You've gotta be really careful and ensure atomic writing and updating.

Even if you work around the atomicity issues, chances are you won't be able to guarantee atomic snapshotting as you're fetching your data within multiple requests, and the data is changing as you retrieve it.

No one likes their API hammered, fair enough. However, rate limits often vary from API endpoint to endpoint and are inherently tedious to get right.

If you're not using the API, you might get banned by DDOS prevention (e.g. Cloudflare) if you're not careful.

Overall, painful and not fun to implement.

Authorization, network, serializing, parsing, storing, synchronizing. There are among the most common error sources (as in, actual unrecoverable errors, not necessarily bugs) in software engineering. Generally, getting it right is required for reliably retrieving your data.

In addition, you want to be somewhat semi-defensive, and this is the hardest kind of error handling:

• you want to progress slowly but surely
• you want to make sure it only fails in completely unrecoverable scenarios, otherwise it's going to require constant tending
• and you want to somehow let user know of problems/suspicious data

If you want all your data, you have to look carefully through all the documentation and make sure you've got it all covered.

If the service adds some new endpoints, you might never find out.

For the most part not an issue, but some websites do not offer an API so you've got not choice but scraping and parsing HTML.

Notorious example: some Hackernews (!) endpoints like 'favorites' are not exposed via API.

Having raw export data (e.g. sqlite database/json file/etc) is nice, but to actually use it you need an abstract representation. You basically have to reinvent whatever the service developer does on the backend already.

Notable examples:

• unclear which data types to choose: nullable/non-nullable, string or integer for ids, float or integer for amounts
• timestamps: figuring out whether it was seconds or milliseconds, UTC or local timezone; and zillions of string formats which you need to parse (I had to do it so often that I even memorized the weird argument order in datetime.strptime)
• which situations are valid, e.g. can id be used as a dictionary key, can you assume that they are increasing, etc.

Sometimes you have no way to access your data at all:

• you are offline: nuff said

• app data on your phone

  Very few apps support data exports; even fewer support it in an automatic and regular way. Normally, internally, apps keep their data in sqlite databases which is even more convenient than plaintext/csv export.

  However, there are caveats: e.g. on Android, app data is in /data/data/ directory, which by default isn't accessible unless you rooted the phone.

• devices that have no means of synchronizing
  • Kobo doesn't seem to support cloud sync for annotations. I was considering syncing the database wirelessly, as there are some SSH modules for its firmware, but people report it may break wifi on it.
• Some devices/apps and formats are vendor locked

Now, remember when I said it was tedious for programmers? Constant boilerplate, shitty APIs (you're lucky if the service offers one at all), latency, flakiness, having to code defensively, etc.

Now think about ordinary people who have no clue what 'API' is. They deserve to use their data too.

• this might be hard to support for everyone

  On the other hand, service developers would have more control on data access patterns, so in a way it might work better.

  It would definitely be more efficient than third parties writing kludgy tools to export and backup data.

  In addition, for some services and scenarios, it would give better data locality and lower latencies.

• 'average' users often are not motivated enough to demand such things

  In particular, not everyone has or willing to set up necessary infrastructure to run all these things.

  However, if implemented properly, there is absolutely nothing preventing running a data mirror on your laptop or even phone. It really doesn't require much CPU or bandwidth if you support incremental updates.

• services have little motivation to promote this, silos benefit them

  Having a monopoly on the client interface (e.g. web UI) keeps users on your platform even if you suck.

  If anyone can implement a better interface, there would be little opportunity for stuff like ads, and the only way for the service to make money would be to collect a fee for data collection and hosting. (which I personally would be happy to pay)

Hopefully all of these issues would be solved by distributed/federated services, but we're pretty far from it.

• deleted content

  E.g. imagine you liked someone's post on Facebook, it got mirrored locally, and then the author removed the post.

  What's the right thing to do for the data mirror app? Should it erase just the post you liked from your data mirror? Should it keep the fact that you liked it at all?

  You may disagree with the way such a policy is imposed by the service, hence implement additional logic to keep more data, and at that point it seems like a matter for legal debate.

• synchronizing data

  If you want to access data from multiple devices, you either have to run multiple mirrors, which would be a bit of a hassle, or use some continuous sync service like Dropbox or Syncthing.

  That however might not be so atomic, depending on the way data is kept on the disk, since files might be pulled in random or lexicographic order, depending on sync configuration.

• protecting the data

  Even if you don't trust your average startup at securing your data, it might be even less safe on average user's disk.

• it's assumed that these tools/integrations are open source and running on computers you own.

  Realistically, closed source tools do exist and it's understandable when people want money for their efforts.

  From a user's perspective not everyone wants the hassle of running things locally either and many people are happy with online services for the most part.

This corresponds to the 'data mirror' bit.

I exported/scraped/reverse engineered pretty much my entire digital trace and figured out automation and infrastructure which works for me.

I've shared some of my personal export scripts and tools.

I also have some helper scripts to keep individual exporter's code as clean as possible while ensuring exports are reliable.

As I mentioned, I'll share all of this later in a separate post.

Each data exporter comes with minimal bindings that merely map json/sqlite export into simple datatypes and data classes. That way anyone who wishes to use data can kick off some reasonable representation, which is not overfitted to my specific needs.

Higher level querying and access, specific to myself is implemented in my. package.

my. package allows me to query my data from anywhere, enabling me to use familiar data processing, analysis and visualization tools, and various integrations.

As a nice byproduct I've also finally figured out a reliable and elegant way to deal with error handling in Python.

Finally, some tools and scripts I've implemented to make possible the interactions that I want:

• A personal search engine for quick incremental search of my data and digital trace
• orger: tool to convert data into org-mode views for instant and offline search and overview
  • also for prioritizing content consumption (e.g. processing Reddit saves)
  • also for populating my spaced repetition queue
  • also for creating TODOs straight from Telegram messages
• grasp, browser extension to clip links straight into my org-mode notes
• promnesia, a browser extension to escape silos by unifying annotations and browsing history from different data sources (still somewhat WIP and needs final touches, but planning to release soon)
• personal health, sleep and exercise dashboard, built from various data sources. I'm in the process of making it public, you can see some screenshots here

I wrote how each specific data source I export contributes to my personal infrastructure here.

I'm synchronizing everything across my computers with syncthing.

I'm simply using borg backup against exported data, whether it's kept in json files or sqlite databases.

• [2020-01-12] What data on myself I collect and why?
• [2020-01-18] In search of a friendlier scheduler
• [2020-03-15] Human Programming Interface