Yamanote is a bookmarklet-based bookmarking web app. It’s a web application so you need to run it on a computer, or get a friend to run it for you. When you decide you want to bookmark a page on the web, you click on a Yamanote bookmarklet in your browser’s bookmarks bar (works great on desktop, and in Safari on iOS) to tell the Yamanote server about it. Any text you’ve selected will be added as a “comment” to the bookmark by Yamanote. This is fun because as you read, you can select interesting snippets and keep clicking the bookmarklet to build a personalized list of excerpts. You can add additional commentary to the bookmark in Yamanote, either by editing one of the excerpts made from the bookmarklet or an entirely new comment with its own timestamp. Also, the first time you bookmark a URL, your browser will snapshot the entire webpage and send it to the Yamanote server as an archive (in technical terms, it’ll serialize the DOM). This is great for (1) paywalled content you had to log in to read, (2) Twitter, which makes it hard for things like Pinboard to archive, etc. The server will download any images—and optionally videos—in your bookmarked sites. You can browse Yamanote’s snapshot of the URL (it might look weird because we block custom JavaScript in the mirror and lots of sites these days look weird with just HTML and CSS—shocking I know). Nobody except you can see your bookmarks, comments, or archives.
WarcDB is a an SQLite-based file format that makes web crawl data easier to share and query. It is based on the standardized Web ARChive format, used by web archivers.
A Huginn agent for querying SQLite databases. Whatever it finds are emitted as events.
This gem provides two agents for Huginn that can read from and write to SQLite 3 databases.
For detailed instructions on their usage, please see the Markdown descriptions within the agents' source (which will also be displayed in your Huginn dashboard).
Note that this gem relies on the sqlite3 gem which itself requires SQLite3 development headers. If you're running Huginn on a regular server, satisfying this requirement may be simple.
WaveDB is SQLite with a HTTP interface.
It is a ~6MB (~2MB UPX-compressed) self-contained, zero-dependency executable that bundles SQLite 3.35.5 (2021-04-19) with JSON1, RTREE, FTS5, GEOPOLY, STAT4, and SOUNDEX.
If you are already a fan of SQLite, WaveDB acts as a thin HTTP-server wrapper that lets you access your SQLite databases over a network.
WaveDB can be used as a lightweight, cross-platform, installation-free companion SQL database for Wave apps. The h2o-wave package includes non-blocking async functions to access WaveDB.
Database files managed by WaveDB are 100% interoperable with SQLite, which means you can manage them with the sqlite3 CLI, backup/restore/transfer them as usual, or use Litestream for replication.
A proof-of-concept inspired and enabled by Hosting SQLite Databases on Github Pages and the ensuing Hacker News post. The compiled single-page app supports autocomplete for titles, automatic redirecting & other MediaWiki datasets like WikiQuote or Chinese Wikipedia. It makes no external API calls except to get Wikipedia's images.
Seems ideal for making offline copies of Wikipedia (or other Mediawiki installs, it's implied) available.
Search is disabled right now.
Github: https://github.com/segfall/static-wiki
SQLite copies of Wikipedia: https://www.kaggle.com/segfall/markdownlike-wikipedia-dumps-in-sqlite
Has instructions for turning an XML dump of Wikipedia into a SQLite database; unfortunately it uses node.js.
Dogsheep is a collection of tools for personal analytics using SQLite and Datasette.
Big internet companies know a lot about us. By exporting that data back out of them we can see what they know and maybe learn something interesting about ourselves.
minidb 2 makes it easy to store Python objects in a SQLite 3 database and work with the data in an easy way with concise syntax. Designed for embedded use (imported as a module) and not a stand-alone server. Supports SQL queries.
A self-hosted service that pings webhooks or other URLs on a user-defined schedule. Works a little bit like cron. Can even do things every X minutes or hours, like cron.
Written in PHP, uses SQLite.
If you don't want to set it up yourself: https://hookless.co/
A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LDJSON / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file. Can also pull data from supplied URLs.
Since Yahoo killed off their Where-On-Earth (WOEID) API service, it hasn't been possible to use it to get WOEID maprefs for certain APIs. Thankfully, some kind soul uploaded it to the Internet Archive.
The contents are five TSV (tab separated value) files, a Readme.txt file, and a license.txt file.
If you want to read them into a SQLite database (and you'll need to set a primary key on each table), do this:
user@host: sqlite3 geoplanet.sqlite
sqlite> .mode tabs
sqlite> PRAGMA foreign_keys=off;
sqlite> CREATE TABLE adjacencies (id INTEGER PRIMARY KEY, Place_WOE_ID TEXT, Place_ISO TEXT, Neighbour_WOE_ID TEXT, Neighbour_ISO TEXT);
sqlite> .import geoplanet_adjacencies_7.10.0.tsv temp_adjacencies
sqlite> INSERT INTO adjacencies(Place_WOE_ID, Place_ISO, Neighbour_WOE_ID, Neighbour_ISO) SELECT Place_WOE_ID, Place_ISO, Neighbour_WOE_ID, Neighbour_ISO from temp_adjacencies;
sqlite> drop table temp_adjacencies;
sqlite> CREATE TABLE admins (id INTEGER PRIMARY KEY, WOE_ID TEXT, iso TEXT, State TEXT, County TEXT, Local_Admin TEXT, Country TEXT, Continent TEXT);
sqlite> .import geoplanet_admins_7.10.0.tsv temp_admins
sqlite> INSERT INTO admins(WOE_ID, iso, State, County, Local_Admin, Country, Continent) SELECT WOE_ID, iso, State, County, Local_Admin, Country, Continent from temp_admins;
sqlite> drop table temp_admins;
sqlite> CREATE TABLE aliases (id INTEGER PRIMARY KEY, WOE_ID TEXT, Name TEXT, Name_Type TEXT, Language Text);
sqlite> .import geoplanet_aliases_7.10.0.tsv temp_aliases
sqlite> INSERT INTO aliases(WOE_ID, Name, Name_Type, Language) SELECT WOE_ID, Name, Name_Type, Language from temp_aliases;
sqlite> DROP TABLE temp_aliases;
sqlite> CREATE TABLE changes (id INTEGER PRIMARY KEY, Woe_id TEXT, Rep_id TEXT, Data_Version TEXT);
sqlite> .import geoplanet_changes_7.10.0.tsv temp_changes
sqlite> INSERT INTO changes (Woe_id, Rep_id, Data_Version) SELECT Woe_id, Rep_id, Data_Version from temp_changes;
sqlite> DROP TABLE temp_changes;
sqlite> CREATE TABLE places (id INTEGER PRIMARY KEY, WOE_ID TEXT, ISO TEXT, Name TEXT, Language TEXT, PlaceType TEXT, Parent_ID TEXT);
sqlite> .import geoplanet_places_7.10.0.tsv temp_places
sqlite> INSERT INTO places (WOE_ID, ISO, Name, Language, PlaceType, Parent_ID) SELECT WOE_ID, ISO, Name, Language, PlaceType, Parent_ID from temp_places;
sqlite> drop table temp_places;
sqlite> PRAGMA foreign_keys=on;
sqlite> .quit
I would also recommend cleaning up after yourself:
user@host: sqlite3 geoplanet.sqlite
sqlite> VACUUM;
sqlite> .quit
The Geoplanet database is licensed by Yahoo! Geoplanet as Creative Commons By Attribution v3.0:
http://creativecommons.org/licenses/by/3.0/us/
How to load JSON into a SQLite database all in one go using Python.
SQLite has a JSON datatype, so it's possible to load JSON objects into columns. There still needs to be a unique key for each entry, though.
Convert CSV files into a SQLite database. Designed for use with Datasette. Requires Python 3.
Datasette provides an instant, read-only JSON API for any SQLite database, even those applications on your system maintain. It also provides tools for packaging the database up as a Docker container and deploying that container to hosting providers such as Zeit Now. Got CSV data? Use csvs-to-sqlite to convert them to SQLite. Runs its own webserver on localhost:8001 so you don't need to write code to use it, just point it at a .sqlite3 file.
Requires Python 3.
A free, open source application that makes manipulating SQLite databases easier. Designed with both people who aren't skilled with databases and administrators in mind.
A single-file database administration utility written in PHP. Supports MySQL, PostgreSQL, SQLite, MS SQL, and Oracle database servers. Formerly phpMyAdmin. Seems to let you do everything you could do from a database shell. Session-aware. Has plugins for a number of popular web apps to help you fix their databases should you need to. Easily themeable, just drop a new CSS file into the directory Adminer is installed in.
A distributed relational database that uses SQLite as its back end. Eventual consensus of all nodes that relies upon quorum. Automatically elects leaders in the network of servers. More lightweight than MySQL or Postgres. Has a REST API built in. Fully transactional. Hot backups possible. Uses SQLite in "all in memory" mode by default for efficiency, but can maintain on-disk database files with a command line option (the on-disk transaction log can be used to repopulate the in-memory database on startup). Written in Go.
A map dataset in the public domain. Multiple scales of resolution available (1:10m, 1:50m, 1:110 million). Vector and raster images. Incorporates GIS attributes for mapmaking. Designed to make it easy to develop maps on your own. Comes as ArcMap MXD and QGIS documents. You can also download entire databases (even SQLite) all in one go.
A server-based chan like BBS written in C. No registration, just post. Supports tripcodes. futaba style. Aggressive content turnover. Requires lighttpd (probably) as its front-end, sqlite backend.
Kareha doesn't use a database, Wakaba does.