The Magic of Small Databases

[davidgasquez]

Found an awesome blog post about building/curating small databases. I think it'll resonate with people here.

I know @rufuspollock has been thinking about this for a while (Data Pinterest). The post mentions a similar analogy with Substack in the Market-making Small Databases section.

Imagine a Substack for databases - an easy tool for creating, maintaining and publishing databases with the ability to restrict parts or all of it behind a pay wall. Pair it with the ability to send email updates to your audience about changes and additions. Maybe even roll in some of the data-cleanup and data-collection magic that machine learning and web scraping offers.

I'd love to explore with you all how a community around this idea could be built and if there is room for a platform to help with this endeavour!

Edit: linking also a related Twitter thread with an interesting conversation.

There have been several well intentioned companies that made platforms for social sharing of data that moved toward commercial versions after the first year or two.

There's a tension between having individuals/groups working to develop databases according to their own interests & goals and making a product out of that.

Each repository on GitHub affords different community standards, there’s room for tools that makes it easier for collaborators to share and curate datasets while allowing different practices.

there’s so much $ to be made quickly in enterprise data that wonderful previous attempts [data.world and Enigma] have been gobbled up. Perhaps the best home/incubator for this is a side project or academia until it’s proven out

[davidgasquez]

In the article, there is a "Small Indexes" airtable which led me to find an awesome resource: Data is Plural, an newsletter covering interesting datasets.

The Data is Plural historical archive contains lots of datasets that might be interesting to index/curate.

[rufuspollock]

@davidgasquez great find David and really resonate.

Related but a bit distinct: https://blog.okfn.org/2013/04/22/forget-big-data-small-data-is-the-real-revolution/

PS: I wonder if we should link blog on datahub.io where we just post up interesting links with (optionally) a bit of commentary. This material would be perfect.