Updated README

README.md

@@ -19,18 +19,53 @@ or use these lists for other applications like selektor.
 So we make two files that are structured in YAML:
 ```
 /etc/tor/yaml/torrc-goodnodes.yaml
+
+---
 GoodNodes:
+  EntryNodes: []
   Relays:
-    IntroductionPoints:
-      - NODEFINGERPRINT
-  ...
+  # ExitNodes will be overwritten by this program
+    ExitNodes: []
+    IntroductionPoints: []
+  # use the Onions section to list onion services you want the
+  # Introduction Points whitelisted - these points may change daily
+  # Look in tor's notice.log for 'Every introduction point for service'
+  Onions: []
+  # use the Services list to list elays you want the whitelisted
+  # Look in tor's notice.log for 'Wanted to contact directory mirror'
+  Services: []
+
+
 By default all sections of the goodnodes.yaml are used as a whitelist.
 
+Use the GoodNodes/Onions list to list onion services you want the
+Introduction Points whitelisted - these points may change daily
+Look in tor's notice.log for warnings of 'Every introduction point for service'
+
+```--hs_dir``` ```default='/var/lib/tor'``` will make the program 
+parse  the files named ```hostname``` below this dir to find 
+Hidden Services to whitelist.
+
+The Introduction Points can change during the day, so you may want to
+rerun this program to freshen the list of Introduction Points. A full run
+that processes all the relays from stem can take 30 minutes, or run with:
+
+```--saved_only``` will run the program with just cached information
+on the relats, but will update the Introduction Points from the Services.
+
 /etc/tor/yaml/torrc-badnodes.yaml
+
 BadNodes:
-  ExcludeExitNodes:
-    BadExit:
-      # $0000000000000000000000000000000000000007
+  # list the internet domains you know are bad so you don't
+  # waste time trying to download contacts from them.
+  ExcludeDomains: []
+  ExcludeNodes:
+  # BadExit will be overwritten by this program
+    BadExit: []
+  # list MyBadExit in --bad_sections if you want it used, to exclude nodes
+  # or any others as a list separated by comma(,)
+    MyBadExit: []
+
 ```
 That part requires [PyYAML](https://pyyaml.org/wiki/PyYAML)
 https://github.com/yaml/pyyaml/ or ```ruamel```: do
@@ -39,7 +74,7 @@ the advantage of the former is that it preserves comments.
 
 (You may have to run this as the Tor user to get RW access to
 /run/tor/control, in which case the directory for the YAML files must
-be group Tor writeable, and its parents group Tor RX.)
+be group Tor writeable, and its parent's directories group Tor RX.)
 
 Because you don't want to exclude the introduction points to any onion
 you want to connect to, ```--white_onions``` should whitelist the
@@ -47,6 +82,13 @@ introduction points to a comma sep list of onions; we fixed stem to do this:
 * https://github.com/torproject/stem/issues/96
 * https://gitlab.torproject.org/legacy/trac/-/issues/25417
 
+Use the GoodNodes/Onions list in goodnodes.yaml to list onion services
+you want the Introduction Points whitelisted - these points may change daily.
+Look in tor's notice.log for 'Every introduction point for service'
+
+```notice_log``` will parse the notice log for warnings about relays and 
+services that will then  be whitelisted.
+
 ```--torrc_output``` will write the torrc ExcludeNodes configuration to a file.
 
 ```--good_contacts``` will write the contact info as a ciiss dictionary
@@ -71,7 +113,7 @@ list of fingerprints to ```ExitNodes```, a whitelist of relays to use as exits.
 3. clean relays that don't have "good' contactinfo. (implies 1)
    ```=Empty,NoEmail,NotGood```
 
-The default is ```=Empty,NotGood``` ; ```NoEmail``` is inherently imperfect
+The default is ```Empty,NoEmail,NotGood``` ; ```NoEmail``` is inherently imperfect
 in that many of the contact-as-an-email are obfuscated, but we try anyway.
 
 To be "good" the ContactInfo must:
@@ -80,81 +122,20 @@ To be "good" the ContactInfo must:
 3. must support getting the file with a valid SSL cert from a recognized authority
 4. (not in the spec but added by Python) must use a TLS SSL > v1
 5. must have a fingerprint list in the file
-6. must have the FP that got us the contactinfo in the fingerprint list in the file,
+6. must have the FP that got us the contactinfo in the fingerprint list in the file.
 
-For usage, do ```python3 exclude_badExits.py --help`
+```--wait_boot``` is the number of seconds to wait for Tor to booststrap
 
+```--wellknown_output``` will make the program write the well-known files
+(```/.well-known/tor-relay/rsa-fingerprint.txt```) to a directory.
 
+```--torrc_output``` will write a file of the commands that it sends to
+the Tor controller, so you can include it in a ```/etc/toc/torrc```.
 
-## Usage 
-```
+```--relays_output write the download relays in json to a file. The relays
+are downloaded from https://onionoo.torproject.org/details
 
-usage: exclude_badExits.py [-h] [--https_cafile HTTPS_CAFILE]
-                           [--proxy_host PROXY_HOST] [--proxy_port PROXY_PORT]
-                           [--proxy_ctl PROXY_CTL] [--torrc TORRC]
-                           [--timeout TIMEOUT] [--good_nodes GOOD_NODES]
-                           [--bad_nodes BAD_NODES] [--bad_on BAD_ON]
-                           [--bad_contacts BAD_CONTACTS]
-                           [--strict_nodes {0,1}] [--wait_boot WAIT_BOOT]
-                           [--points_timeout POINTS_TIMEOUT]
-                           [--log_level LOG_LEVEL]
-                           [--bad_sections BAD_SECTIONS]
-                           [--white_onions WHITE_ONIONS]
-                           [--torrc_output TORRC_OUTPUT]
-                           [--relays_output RELAYS_OUTPUT]
-                           [--good_contacts GOOD_CONTACTS]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --https_cafile HTTPS_CAFILE
-                        Certificate Authority file (in PEM)
-  --proxy_host PROXY_HOST, --proxy-host PROXY_HOST
-                        proxy host
-  --proxy_port PROXY_PORT, --proxy-port PROXY_PORT
-                        proxy control port
-  --proxy_ctl PROXY_CTL, --proxy-ctl PROXY_CTL
-                        control socket - or port
-  --torrc TORRC         torrc to check for suggestions
-  --timeout TIMEOUT     proxy download connect timeout
-  --good_nodes GOOD_NODES
-                        Yaml file of good info that should not be excluded
-  --bad_nodes BAD_NODES
-                        Yaml file of bad nodes that should also be excluded
-  --bad_on BAD_ON       comma sep list of conditions - Empty,NoEmail,NotGood
-  --bad_contacts BAD_CONTACTS
-                        Yaml file of bad contacts that bad FPs are using
-  --strict_nodes {0,1}  Set StrictNodes: 1 is less anonymous but more secure,
-                        although some sites may be unreachable
-  --wait_boot WAIT_BOOT
-                        Seconds to wait for Tor to booststrap
-  --points_timeout POINTS_TIMEOUT
-                        Timeout for getting introduction points - must be long
-                        >120sec. 0 means disabled looking for IPs
-  --log_level LOG_LEVEL
-                        10=debug 20=info 30=warn 40=error
-  --bad_sections BAD_SECTIONS
-                        sections of the badnodes.yaml to use, comma separated,
-                        '' BROKEN
-  --white_onions WHITE_ONIONS
-                        comma sep. list of onions to whitelist their
-                        introduction points - BROKEN
-  --torrc_output TORRC_OUTPUT
-                        Write the torrc configuration to a file
-  --relays_output RELAYS_OUTPUT
-                        Write the download relays in json to a file
-  --good_contacts GOOD_CONTACTS
-                        Write the proof data of the included nodes to a YAML
-                        file
-
-This extends nusenu's basic idea of using the stem library to dynamically
-exclude nodes that are likely to be bad by putting them on the ExcludeNodes or
-ExcludeExitNodes setting of a running Tor. *
-https://github.com/nusenu/noContactInfo_Exit_Excluder *
-https://github.com/TheSmashy/TorExitRelayExclude The basic idea is to exclude
-Exit nodes that do not have ContactInfo: *
-https://github.com/nusenu/ContactInfo-Information-Sharing-Specification That
-can be extended to relays that do not have an email in the contact, or to
-relays that do not have ContactInfo that is verified to include them.
+For usage, do ```python3 exclude_badExits.py --help`
+See [exclude_badExits.txt](./exclude_badExits.txt)
 
-```
 

exclude_badExits.bash

@@ -3,24 +3,37 @@
 
 PROG=exclude_badExits.py
 SOCKS_PORT=9050
+SOCKS_HOST=127.0.0.1
 CAFILE=/etc/ssl/certs/ca-certificates.crt
+# you may have a special python for installed packages
+EXE=`which python3.bash`
 
+$EXE exclude_badExits.py --help > exclude_badExits.txt &
+$EXE -c 'from exclude_badExits import __doc__; print(__doc__)' >exclude_badExits.md
 # an example of running exclude_badExits with full debugging
-# expected to take an hour or so
+# expected to 20 minutes or so
 declare -a LARGS
 LARGS=(
+    #    --saved_only
+    #    --strict_nodes 1
+    --points_timeout 150
     --log_level 10
-    )
-# you may have a special python for installed packages
-EXE=`which python3.bash`
+    --https_cafile $CAFILE
+)
+[ -z "$socks_proxy" ] || \
 LARGS+=(
-    --strict_nodes 1
-    --points_timeout 120
-    --proxy-host 127.0.0.1
+    --proxy-host $SOCKS_HOST
     --proxy-port $SOCKS_PORT
-    --https_cafile $CAFILE
 )
 
+if [ -f /var/lib/tor/.SelekTOR/3xx/cache/9050/notice.log ] ; then
+    LARGS+=(--notice_log /var/lib/tor/.SelekTOR/3xx/cache/9050/notice.log)
+fi
+
+if [ -d /var/lib/tor/hs ] ; then
+    LARGS+=( --hs_dir /var/lib/tor/hs )
+fi
+
 if [ -f '/run/tor/control' ] ; then
     LARGS+=(--proxy-ctl '/run/tor/control' )
 else
@@ -34,8 +47,9 @@ LARGS+=( --white_onions $ddg )
 # you may need to be the tor user to read /run/tor/control
 grep -q ^debian-tor /etc/group && TORU=debian-tor || {
     grep -q ^tor /etc/group && TORU=tor
-}
-sudo -u $TORU $EXE exclude_badExits.py "${LARGS[@]}" \
+    }
+#     --saved_only
+sudo -u $TORU $EXE exclude_badExits.py "${LARGS[@]}" "$@" \
    2>&1|tee exclude_badExits6.log
 
 # The DEBUG statements contain the detail of why the relay was considered bad.

exclude_badExits.txt

@@ -0,0 +1,76 @@
+usage: exclude_badExits.py [-h] [--https_cafile HTTPS_CAFILE]
+                           [--proxy_host PROXY_HOST] [--proxy_port PROXY_PORT]
+                           [--proxy_ctl PROXY_CTL] [--torrc TORRC]
+                           [--timeout TIMEOUT] [--good_nodes GOOD_NODES]
+                           [--bad_nodes BAD_NODES] [--bad_on BAD_ON]
+                           [--bad_contacts BAD_CONTACTS] [--saved_only]
+                           [--strict_nodes {0,1}] [--wait_boot WAIT_BOOT]
+                           [--points_timeout POINTS_TIMEOUT]
+                           [--log_level LOG_LEVEL]
+                           [--bad_sections BAD_SECTIONS]
+                           [--white_onions WHITE_ONIONS]
+                           [--torrc_output TORRC_OUTPUT] [--hs_dir HS_DIR]
+                           [--notice_log NOTICE_LOG]
+                           [--relays_output RELAYS_OUTPUT]
+                           [--wellknown_output WELLKNOWN_OUTPUT]
+                           [--good_contacts GOOD_CONTACTS]
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --https_cafile HTTPS_CAFILE
+                        Certificate Authority file (in PEM)
+  --proxy_host PROXY_HOST, --proxy-host PROXY_HOST
+                        proxy host
+  --proxy_port PROXY_PORT, --proxy-port PROXY_PORT
+                        proxy control port
+  --proxy_ctl PROXY_CTL, --proxy-ctl PROXY_CTL
+                        control socket - or port
+  --torrc TORRC         torrc to check for suggestions
+  --timeout TIMEOUT     proxy download connect timeout
+  --good_nodes GOOD_NODES
+                        Yaml file of good info that should not be excluded
+  --bad_nodes BAD_NODES
+                        Yaml file of bad nodes that should also be excluded
+  --bad_on BAD_ON       comma sep list of conditions - Empty,NoEmail,NotGood
+  --bad_contacts BAD_CONTACTS
+                        Yaml file of bad contacts that bad FPs are using
+  --saved_only          Just use the info in the last *.yaml files without
+                        querying the Tor controller
+  --strict_nodes {0,1}  Set StrictNodes: 1 is less anonymous but more secure,
+                        although some onion sites may be unreachable
+  --wait_boot WAIT_BOOT
+                        Seconds to wait for Tor to booststrap
+  --points_timeout POINTS_TIMEOUT
+                        Timeout for getting introduction points - must be long
+                        >120sec. 0 means disabled looking for IPs
+  --log_level LOG_LEVEL
+                        10=debug 20=info 30=warn 40=error
+  --bad_sections BAD_SECTIONS
+                        sections of the badnodes.yaml to use, in addition to
+                        BadExit, comma separated
+  --white_onions WHITE_ONIONS
+                        comma sep. list of onions to whitelist their
+                        introduction points - BROKEN
+  --torrc_output TORRC_OUTPUT
+                        Write the torrc configuration to a file
+  --hs_dir HS_DIR       Parse the files name hostname below this dir to find
+                        Hidden Services to whitelist
+  --notice_log NOTICE_LOG
+                        Parse the notice log for relays and services
+  --relays_output RELAYS_OUTPUT
+                        Write the download relays in json to a file
+  --wellknown_output WELLKNOWN_OUTPUT
+                        Write the well-known files to a directory
+  --good_contacts GOOD_CONTACTS
+                        Write the proof data of the included nodes to a YAML
+                        file
+
+This extends nusenu's basic idea of using the stem library to dynamically
+exclude nodes that are likely to be bad by putting them on the ExcludeNodes or
+ExcludeExitNodes setting of a running Tor. *
+https://github.com/nusenu/noContactInfo_Exit_Excluder *
+https://github.com/TheSmashy/TorExitRelayExclude The basic idea is to exclude
+Exit nodes that do not have ContactInfo: *
+https://github.com/nusenu/ContactInfo-Information-Sharing-Specification That
+can be extended to relays that do not have an email in the contact, or to
+relays that do not have ContactInfo that is verified to include them.

support_onions.py

@@ -33,18 +33,39 @@ bHAVE_TORR = shutil.which('tor-resolve')
 # in the wild we'll keep a copy here so we can avoid restesting
 yKNOWN_NODNS = """
 ---
-  - a9.wtf
+  - 0x0.is 
+  - a9.wtf 
+  - apt96.com
+  - axims.net 
+  - backup.spekadyon.org 
+  - dfri.se 
+  - dotsrc.org
+  - dtf.contact
+  - ezyn.de 
+  - for-privacy.net 
+  - galtland.network 
   - heraldonion.org
+  - interfesse.net 
+  - kryptonit.org 
   - linkspartei.org
+  - mkg20001.io 
+  - nicdex.com
+  - nx42.de 
   - pineapple.cx
   - privacylayer.xyz
+  - privacysvcs.net 
   - prsv.ch
+  - sebastian-elisa-pfeifer.eu
   - thingtohide.nl
-  - tor-exit-2.aa78i2efsewr0neeknk.xyz
-  - tor-exit-3.aa78i2efsewr0neeknk.xyz
+  - tor-exit-2.aa78i2efsewr0neeknk.xyz 
+  - tor-exit-3.aa78i2efsewr0neeknk.xyz 
   - tor.dlecan.com
+  - tor.skankhunt42.pw
+  - transliberation.today 
   - tuxli.org
+  - unzane.com 
   - verification-for-nusenu.net
+  - www.defcon.org
 """
 #  - 0x0.is
 #  - aklad5.com
@@ -220,7 +241,8 @@ def lIntroductionPoints(controller=None, lOnions=[], itimeout=120, log_level=10)
             l += lp
         except (Empty, Timeout,) as e: # noqa
             LOG.warn(f"Timed out getting introduction points for {elt}")
-            continue
+        except stem.DescriptorUnavailable as e:
+          LOG.error(e)
         except Exception as e:
           LOG.exception(e)
     return l