Woman who would have been revered prophetess 4,000 years ago now relegated to clicking links, opening tabs

published: 2022-11-01

One of the funniest genres of emails I get in relation to my personal projects is the man (it's always been a man, every single damn time) who clearly has no idea how I manage to keep Let's Decentralize running and thinks he can improve on operations and fails immensely.

Nothing will top the Lokinet dev who came into my inbox unsolicited one day, looked at my site, and thought to himself, "Oh dear. An SVG of an outdated logo. You know how I can fix this? Being a condescending ass and then sending her the blurriest JPEG that's ever existed and demanding she use that as the logo instead." Needless to say, I did not change the logo. I instead put a note in the Lokinet section stating that the daemon doesn't seem to work when compiled from source (no sites will connect) and that the Lokinet developers haven't yet proven themselves trustworthy enough to not secretly be injecting other code into their precompiled binaries. "Chief" did not bother me again after that. (Despite the source code having been fixed for Debian since, the build instructions for FreeBSD conveniently omit that libtool and automake also need to be installed. And it doesn't compile. Maybe stop jerking off your shitcoin for a few minutes and fix that?)

I also occasionally get random scripts claiming to automate the task of checking if every individual site on each link list is still up. As if, if I wanted to use a script, I'm not perfectly capable of writing one myself. But the problem with these scripts, or any automation really, is that they invariably only check to see if the server returns the HTTP status 200 OK. There are a lot of failure modes that require action on my part to keep the list safe and organized that would still return 200 OK in a script:

A completely blank page.
A site that was innocuous when I added it to the list, like a search engine or an anonymous mail service, that has now sold out and become a CSAM distributor. I remove these the instant I find them.
An instance of a privacy frontend, like Nitter (for Twitter), becoming an instance for a different frontend that proxies a different service, like Invidious (for YouTube). I keep these, but move them to their new proper sections.
A message from the admin stating that they aren't going to be running the site anymore and that it should be removed from the user's bookmarks and link lists.
A misconfigured nginx server that used to serve a hidden service but doesn't any longer, but the hidden service is still configured in Tor. The request for the hidden service hits nginx, but since it's no longer a defined site, nginx redirects the request to whatever site has been configured as default, meaning the user is now browsing a clearnet site. The clearnet site then returns 200 OK, misleading the script into thinking the check was successful.

The difference between Let's Decentralize and other collectors of Tor links is that I individually click on every single link, every week, to check to see if the site is still what it was when I last checked it. I used to be able to do this in a single Sunday morning before I went to work, but now the Tor list is so damn long that I have to limit myself to an hour a day for my own sanity and start on Thursday or Friday so that I can still push the finalized list every Sunday. I don't know of any other link lists or Hidden Wiki clones that do this. They simply do what the people sending me the scripts want me to start doing and check for an HTTP 200 OK response. That's why you go onto sites like "Fresh Onions" and see a bunch of CSAM and porn and scam markets clogging everything up, or find a list that's organized but everything is horrifically outdated and half of the sites are dead and there's been no new additions since the list was originally uploaded.

Speaking of no new additions, I'm starting to hit the upper wall of services to add that aren't pornography or markets. You may have noticed that, over the course of this year, the Tor link list has more than tripled in size. This isn't due to a Cambrian explosion of new hidden services, but because my methods of finding them has gotten better. If I somehow, despite my myriad backups both online and offline, managed to completely lose the files to Let's Decentralize and had to bootstrap the list again from scratch, here's how I would go about finding hidden services:

Find one of the legion Hidden Wiki clones floating around. Most of the sites linked there are for cryptocurrencies or markets, but near the bottom is usually a small collection of blogs and personal pages. Add those personal pages to the list and check to see if they have any lists of their own; if they're on the darknet, they usually have at least a small handful.
Go to one of the "Fresh Onions" mirrors. (I can't link any because, as I stated before, these are flooded with links to CSAM.) Click on anything that looks interesting from the first few pages. As long as it's not porn or a market or displaying highly objectionable material, it goes on the list. Note that sometimes the links can be marked red for "not available" when really their web servers have just been configured to block requests from suspected web scrapers.
Go to Ahmia's list of known onion services and manually scrape the titles from all of them to try to find ones that look interesting. I picked Ahmia because it claims to remove URLs to sites that distribute CSAM. I wish all Tor hidden service search engines did that...
Sign up for a free Shodan account. Shodan lets you search by HTTP headers. Because the "Onion-Location" header is often used to signal that a site has a mirror on Tor, you can find hidden services (whose clearnet mirrors haven't blacklisted Shodan's IP ranges in iptables) by handing Shodan the following query in the search box:

onion-location -http.title:"Globaleaks" -http.title:"Sign in" -http.title:"Hack This Site" -http.title:"302 Found" -http.title:"Log in" -http.server:"GlobaLeaks"

Free accounts on Shodan can only view the first two pages of results. This can be bypassed, kind of, by going to the left sidebar and right-clicking each country code to open it in a new tab. The query above also filters out by title certain sites that have lots of duplicate mirrors or are just blank pages. If Shodan didn't restrict free accounts to only the first two pages of a search, this wouldn't be a problem. You can theoretically pony up $50 one time to get access to twenty pages per search, but that seems a bit pricey considering that finding Tor hidden services is the only thing I'd use it for.

Lurk in places like r/onions and the technology boards on imageboards and other places online where people discuss Tor. Invariably there will be threads where people ask for links to hidden services or show off ones that they've made themselves.

Personally, given the amount of effort I've put into Let's Decentralize since I broke it off from MayVaneDay a few years ago, I'd rather just trust my backups. Although I suppose I will die one day, maybe sooner than later, and someone else will inherit all this mess. If you, the reader, think you can do this better than me, are you prepared to take up all the effort detailed above and relegate yourself to clicking links and opening tabs?