SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
An open-source distributed storage service you can self-host to fullfill many needs. Can do everything from hosting to working as a backup target. Designed to make it easy to have multiple replicas in widely geographically dispersed locations.
Comes as a single no-dependency executable. Written by sysadmins, so they understand how much you don't want to dick around with stuff until it works. Tries to be as resilient as possible. If you really want, you can run a copy on everything you own to build a storage field. Has very lightweight hardware requirements. Implements the S3 API.
A modern, scalable, and extensible web crawler designed for efficient distributed crawling and data extraction. Built using asynchronous I/O, robust logging, plugin architecture, and distributed task processing with Celery and Redis. Uses PostgreSQL (via SQLAlchemy) to store crawled pages efficiently with indexing and transactional support. A dedicated logging framework supports console output, rotating file logs, and SQLite logging for persistent diagnostics. Easily extend the crawler with custom plugins for processing, filtering, or transforming crawled data without modifying core code. Enhanced URL normalization removes trailing slashes and sorts query parameters to avoid duplicate processing.
The next iteration of Syncthing for Android. Available in the F-Droid repository. Tries to be a drop-in replacement for Syncthing.
Folder, device and overall sync progress can easily be read off the UI. "Syncthing Camera" - an optional feature where you can take pictures on two phones into one shared and private Syncthing folder. No cloud involved. (deprecated) "Sync every hour" to save even more battery. Individual sync conditions can be applied per device and per folder (for expert users). Changes to folder and device config can be made regardless if syncthing is running or not. UI explains why syncthing is running or not. Supports two-way synchronization on external sd cards since Android 11. Supports encrypted folders on untrusted devices.
Has migration instructions for the original Syncthing.
BarkVR is an open-source and decentralized social XR creativity tool, built upon the solid foundation of Godot 4.x.
The project is still in the very early stages. Please feel free to support the project however you wish, or not at all.
- Accessibility: Making BarkVR welcoming to everyone, regardless of background or ability.
- Decentralized Design: Giving users the power to shape BarkVR's world through decentralization.
- User Empowerment: Providing users with complete control over their creations for a sense of ownership.
- Open Design: Welcoming collaboration and transparency to evolve BarkVR together.
- VR FOSS: Leading the charge in open-source VR for limitless innovation.
- Social VR: Creating a space where connections thrive beyond physical limits, fostering shared experiences and boundless imagination.
For those eager to follow BarkVR's progress, the "dev" branch is where the latest changes are. Main will eventually be reserved for tagged releases.
Radicle is an open source, peer-to-peer code collaboration stack built on Git. Unlike centralized code hosting platforms, there is no single entity controlling the network. Repositories are replicated across peers in a decentralized manner, and users are in full control of their data and workflow.
The Radicle protocol leverages cryptographic identities for code and social artifacts, utilizes Git for efficient data transfer between peers, and employs a custom gossip protocol for exchanging repository metadata. All social artifacts are stored in Git, and signed using public-key cryptography. Radicle verifies the authenticity and authorship of all data for you. Radicle is local-first, providing always-available functionality even without internet access. Users own their data, making migration, backup, and access easy both online and offline. The Radicle Stack comes with a CLI, web interface and TUI, that are backed by the Radicle Node and HTTP Daemon. It’s modular, so any part can be swapped out and other clients can be developed.
Repo: https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5
In the AUR.
OpenSimulator is an open source multi-platform, multi-user 3D application server. It can be used to create a virtual environment (or world) which can be accessed through a variety of clients, on multiple protocols. The optional Hypergrid allow users to visit other OpenSimulator installations across the web from their 'home' installation or grid.
OpenSimulator allows virtual world developers to customize their worlds using the technologies they feel work best - we've designed the framework to be easily extensible. OpenSimulator is written in C#, running both on Windows with the .NET Framework and on Unix-like machines with Mono. The source code is released under a BSD License, a commercially friendly license to embed OpenSimulator in products.
Git repo: git://opensimulator.org/git/opensim
VeilidChat is a chat application written for the Veilid distributed application platform. It has a familiar and simple interface and is designed for private, and secure person-to-person communications.
Requires a local clone of the Veilid source code to compile.
For more information about VeilidChat: https://veilid.chat/
Quiet is an alternative to team chat apps like Slack, Discord, and Element that does not require trusting a central server or running one's own. In Quiet, all data syncs directly between a team's devices over Tor with no server required.
Quiet is written (mostly) in TypeScript, with Electron and React Native frontends.
While apps like Slack, Discord, and Signal use central servers, Quiet syncs messages directly between a team's devices, over Tor, with no server required.
Each group of people (Quiet calls them "communities") gets their own insular network, so that data from one community never touches the devices of Quiet users in other communities. Not even in encrypted form!
Message syncing is taken care of by a project called OrbitDB, which works like a mashup of Git, a gossip protocol, and BitTorrent; it broadcasts new messages, syncs the latest messages, and fetches files. Syncing means that users typically receive all messages sent while they were offline.
A system for running distributed code over multiple PI Picos connected to each other. The reference implementation has them all standing on end in a ring, resembling a Cray supercomputer.
Wire all of the picos in parallel connecting GPIO 10,11,12,13 and GND to all of the picos. Each pico needs powering separatly either from the USB bus or 3v3 pin. Make the I2C bus as short as possible and you will need one pair of 4k7 Ohm resistors to +3v3 on GPIO 10 and 11, ideally in the last processor in the chain.
The Code is the same for all Picos, to specify a controller add a wire from GPIO 22 to GND There must be (only) one controller in each cluster.
orbit-db is a distributed peer-to-peer database on IPFS. This project intends to provide a fully compatible port of the JavaScript version in Go. OrbitDB uses IPFS as its data storage and IPFS Pubsub to automatically sync databases with peers. Implements append-only logs, traversable feeds, key/value storage, JSON document storage, and even a basic counter.
USENET-inspired, uncensorable, decentralized internet discussion system running on IPFS and OrbitDB, with lots of 80's style synthewave puns. Aims to be censorship-resistant and distributed. Requires a local IPFS client to access the network.
Populus-Viewer is a tool for decentralized social annotation, built on pdfjs, wavesurfer.js and the Matrix protocol. You can use it to read PDFs, listen to audio, or watch videos, and have rich discussions in the margins, with your friends, classmates, or scholarly collaborators.
Each uploaded file is attached to a matrix space, and each annotation to the file becomes a room within that space. Populus-Viewer has been tested with synapse and dendrite, but should be compatible with any spec-compliant matrix server.
The metaverse is here: an immersive Web full of social virtual and augmented reality experiences. However, mega-corporations want to lock it down and wall it up to make another addictive, toxic ad-selling platform. We’re taking it in a different direction because we dream of a democratized new era of the Web where creators own their content, users own their data, and no single entity exhibits undue influence on the community as a whole.
Immers Space is a social networking application that runs alongside your Immersive Web experience. It provides account registration and login for your site, allows users to login to your site with their existing accounts from other immers, and publishes social updates for users on your site (e.g. sharing a link to invite friends to meet up at your site).
Github: https://github.com/immers-space/
Javascript, but it's supposed to run in a browser. What can you do?
NNCP (Node to Node copy) is a collection of utilities simplifying secure store-and-forward files, mail and command exchanging. This utilities are intended to help build up small size (dozens of nodes) ad-hoc friend-to-friend (F2F) statically routed darknet delay-tolerant networks for fire-and-forget secure reliable files, file requests, Internet mail and commands transmission. All packets are integrity checked, end-to-end encrypted, explicitly authenticated by known participants public keys. Onion encryption is applied to relayed packets. Each node acts both as a client and server, can use push and poll behaviour model.
Out-of-box offline sneakernet/floppynet, dead drops, sequential and append-only CD-ROM/tape storages, air-gapped computers support. But online TCP daemon with full-duplex resumable data transmission exists.
p2p IRC-inspired self-hosted web chat. Seems to be encrypted, or at least signed for identification (ECC keypairs). Uses WebRTC and Webtorrent. STUN and TURN enabled.
Lots and lots of Javascript so download a release. Only requires a static webhost, though.
Dendrite will be a Matrix server written in Go. Requires Kafka (if run as a cluster of microservices) or something called Naffka (an embedded in-process workalike) if run monolithically. Requires Postgres as its back end.
SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of relatively small files, and to serve those files fast. Implements an object store with O(1) disk seek and an optional filer with a POSIX interface. Metadata can be stored in one of several RDBMSes. Speaks HTTP(S). Supports multiple access APIs, including S3, HDFS, and WebDAV. Can automatically back itself up offsite. Supports multiple URI formats, with varying degrees of niceness. Large files are chunked transparently to the user.
Mitogen is a Python library for writing distributed self-replicating programs.
There is no requirement for installing packages, copying files around, writing shell snippets, upfront configuration, or providing any secondary link to a remote machine aside from an SSH connection. Due to its origins for use in managing potentially damaged infrastructure, the remote machine need not even have free disk space or a writeable filesystem.
It is not intended as a generic RPC framework; not intended for direct use by consumer software.
The focus is to centralize and perfect the intricate dance required to run Python code safely and efficiently on a remote machine, while avoiding temporary files or large chunks of error-prone shell scripts, and supporting common privilege escalation techniques like sudo, potentially in combination with exotic connection methods such as WMI, telnet, or console-over-IPMI.
Github: https://github.com/dw/mitogen
Notes and scripts for setting up (yet another) Raspberry Pi computing cluster. One master, at least one slave to do the actual work. The master implements a certain amount of infrastructure for the rest of the network. Includes greyprints for 3D printing a rack for the units. Uses k3s and Docker.