We have basically no cellular connectivity or internet access at our new home.

We will in the near future have a slightly better (but still slow) 4g connection thanks to our cell site booster, and DSL (about two miles from the DSLAM.)

I have invested a lot of time and energy in to offline first applications, but not in a while, so I'm re-investigating these things specifically as they relate to our new housing situation.

One tool that I have played with in the past is Kiwix.

Kiwix is an offline wikipedia reader, and it's pretty useful, especially when paired with Kiwix serve.

It seems like you should be able to abstract it in to a full fledged LAN to internet bridge, leveraging RSS support and maybe a couple of hacky hotsync jobs to download content when you have a network connection and then sync it with your LAN machine when you're on your LAN.

This doesn't work, though.

It has a few pretty severe limitations that relegate it to the world of Just Wikipedia, and not more generally useful for me, and I always forget about that until I try to build my own content for it.

Basically, it uses the file format from zim-wiki, a desktop wiki tool. It's really not great at serving content from multiple websites at one time (I can make it work, but it's needlessly complicated) and building content for it is an absolute nightmare.

So I'm looking for something better.

I want to cobble together a LAN internet in a box.

Something like the piratebox project, but that enables me to sync in content from outside my network.

I have some options.

Dat/Beaker always looks like it would be a solution to this problem, but it's node based and the last time I used it the command line functionality was limited. I'll re-investigate it.

More importantly Dat only solves the *sync*, not the capture or the serve.

Which brings me to the reason I started this thread in the first place:

What are your favorite LAN-first or offline first applications for desktops, servers, and mobile devices?

I would prefer to hear about applications with which you have firsthand experience, but I also wanna know about cool stuff that you've just heard about in passing.

Assume storage is free, assume local bandwidth is very high, assume that internet access is unavailable most of the time, but occasionally Very Fast.

@ajroach42 Hey @jgoerzen, I feel like you might have some valuable input here!

@elb @ajroach42 Oh yes, I can jump in. So, start with . I love it. I explored some possibilities with it at changelog.complete.org/archive . Syncthing is serverless sync, ala Dropbox, but also distributed. One possible scenario: a device (laptop or phone) is the "carrier". Syncs locally at the house, and to someplace on the Internet at a coffee shop / in town / whatever.

@elb @ajroach42 2/ I also looked into Dat and IPFS, but they are neither as capable nor as useful as Syncthing for personal synchronization.

@elb @ajroach42 3/ For, eg, downloading websites, archivebox.io could be very nice. Archive Team also has some wiki pages on how to do it with wget and httrack. There are also plugins like webrecorder that could help.

@elb @ajroach42 4/ What you are really after is more general asynchronous communication. I have a whole blog series about this, including and other tools: changelog.complete.org/archive will give you all the posts in the series. Many of them are somewhat focused on backups, but should give you some good ideas for other things also. NNCP can use things like USB sticks, serial links, regular Internet connections, Syncthing, etc. as transport.


@elb @ajroach42 5/ The NNCP page on use cases may give you some ideas (whether or not you use NNCP) nncpgo.org/Use-cases.html Their integration page nncpgo.org/Integration.html also is useful. nncpgo.org/WARCs.html describes downloading webpages.

@elb @ajroach42 6/ You talked about accessing web pages offline. I've tried that but mostly don't really bother. It is fairly painful (you frequently want to click on a link you don't have). In some cases, for things like larger articles, it can make good sense. But you might want to look into something more like rss2email . Email is already asynchronous and there are lots of ways to get asyncrhonous email across. NNCP is one and documents this workflow at nncpgo.org/Feeds.html

@elb @ajroach42 7/ Offline email is two separate problems: sending and receiving. Sending can go across BSMTP (delivered "somehow" via NNCP, UUCP, Syncthing, etc). I talk about Exim with NNCP as part of my series here changelog.complete.org/archive and the NNCP docs go over the Postfix setup. For incoming, you can use OfflineIMAP or an offline-capable mail reader. Alternatively, forwarding to an account you can receive via NNCP/etc to a local mailstore.

@elb @ajroach42 8/ So if you have a VPS or a machine "in town" or whatever, you can do some pretty nice things; take the photos you copied into the "to upload" Syncthing folder and upload them, then delete them out of there. Or a laptop can run those commands directly "in town"

@elb @ajroach42 9/ Finally the two best ways to improve your 4G signal are: 1) height, and 2) antenna. I got one of these smile.amazon.com/gp/product/B0 with a Nighthawk M1 awhile back. Tremendous difference. A booster can only boost what it can receive. A good antenna, mounted high, hardwired into the access point will almost certainly be better. That antenna has "gain", meaning it's directional, so figure out where your best towers are and point it at those.

@elb @ajroach42 10/ Also point-to-point wireless may help; if there's a good place you can get Internet and you have line-of-sight from your house, you may be able to work something out, even something surprisingly fast. For more challenging conditions, LoRA or XBee could work... but at 100Kbps or less. Not suitable for browsing but could work for email.

@elb @ajroach42 11/ Finally, don't understimate the utility of sshing to a VPS somewhere and reading email in text. My qualifications to anser: have lived in Internet-challenged areas for 20 years, frequently travel into no-Internet areas, have experience with modern communication over extremely low bandwidth links (1 to 100Kbps) including LoRA/XBee radio, AX.25 packet radio, and satellite.

@elb @ajroach42 Also, @joeyh lives off-grid and may have lots of ideas to contribute here.

@jgoerzen @elb I did some experimenting with LoRA and tried to do some work with XBee for keeping nodes of a distributed BBS in sync over multiple KM, but ultimately we just didn't have the mesh density, and I ended up building a solution that used a device I carried with me to rsync each location over wifi.

Using syncthing and a cellphone (and NNCP) seems like a more viable longterm solution.

@jgoerzen @elb Thick canopy and mountains, so there's only so much improving to do.

We'll stick an antenna up pretty high, and run that in to our little booster.

It'll get the job done (I'm posting now over that cellular connection. It works, when you're in the right location.)

@ajroach42 @jgoerzen @elb I still say we should try the trebuchet of usb flash drives thing at some point

@djsundog @ajroach42 @jgoerzen @elb question: what would be the bandwidth of 90 kg of flash drives launched 300 m?

How fast would that be travelling

@ajroach42 @djsundog @jgoerzen @elb I googled it myself. Top speed 70 m/s, apparently...someone do the math

@nev a USB drive on average weighs about 30 grams, according to the two sources I found that weighed USB drives.

Microcenter cells 256 GB flash drives. You could fit 3000 of them in a 90 KG payload.

So that's 768 terabytes.

A full sized trebuchet can launch a 90kg at roughly 70m/s or 156.586 MPH.

So you're sending 768 TB at 300 meters in 4.2 seconds for a transfer speed of 182.857143 terabytes TBps.

@nev Or, in more conventional terms, 1462857144 Mbps, or 1462857144000 Kbps or 1.462857144e+15 bits per second.

@ajroach42 thank you for this incredible contribution to modern science.

@ajroach42 @nev I thought 30g sounded like a /really/ heavy flashdrive, so I weighed the small jar of flashdrives on my desk. their weights: 10g, 9g, 4g, 8g, 16g. which would increase data throughput :) I can weigh some SD cards too I guess... nvm. microSD card doesn't make it up to 1g on my scales. 0.5g would be best guess. regular sized SD card is about 2 or 2.5g.

@epoch @ajroach42 okay so by my calculation, assuming a micro SD card is 0.5g and holds 1 TB, a 90kg payload contains 180 PB...

@nev @epoch and it'll travel 300 meters in 4.2 seconds.

So 180PB/4.2 seconds is 342 Petabits per second.

@jgoerzen @djsundog mentioned rss2email, which seems like a good choice to me (although, frankly, RSS to html files in syncthing is probably fine too.)

I think what I'm wanting in terms of capturing web pages is an interface where I provide, for example, a URL or a search term, and the next time a connected node has an internet connection, that web page and every page it links to directly, and every page each of those links to directly is captured for LAN transfer at a later date.

@ajroach42 @djsundog You could pretty easily do that with httrack on some remote system. nncp-exec sending a URL to that, it returns a tarball to you, or something similar, I'd think.

@jgoerzen @ajroach42 with a possible pit stop on a shared lan server that untars it long enough to index into elasticsearch or whatever along with a pointer to the tarball on lan storage for caching etc. yeah, this is more than doable.

@djsundog @ajroach42 Interestingly, Sergey just (literally within the past few days) added "multicast" support to nncpgo.org/Multicast.html . The Internet-connected machine could send the downloaded site to the LAN "area". Upon arriving at the LAN NNCP gateway, it would be copied both to the PC and to the indexer. Wouldn't require any extra scarce bandwidth, and also wouldn't require the indexer to be up in order for the data to reach the PC.

@djsundog @ajroach42 NNCP 7.x is still a bit buggy for me (I make pretty heavy use of it, sending thousands of packets, some of them hundreds of GB in size, across it daily) but I'm sure it will shake out shortly. In the meantime, 6.6.0 is rock solid

@jgoerzen @ajroach42 @elb

Reg. offline: you know scuttlebutt, do you? That sneakernet-compatible social network?

I think it's a pretty good idea that meshes direct connections to hubs/proxies and p2p connections to update offline data.

Sign in to participate in the conversation

For people who care about, support, or build Free, Libre, and Open Source Software (FLOSS).