Compare commits
3 commits
204a6adc48
...
7a1999f117
| Author | SHA1 | Date | |
|---|---|---|---|
|
7a1999f117 | ||
|
|
ec7c600d85 | ||
|
|
d08b34fd57 |
5 changed files with 591 additions and 347 deletions
145
README.md
145
README.md
|
|
@ -19,18 +19,53 @@ or use these lists for other applications like selektor.
|
|||
So we make two files that are structured in YAML:
|
||||
```
|
||||
/etc/tor/yaml/torrc-goodnodes.yaml
|
||||
|
||||
---
|
||||
GoodNodes:
|
||||
EntryNodes: []
|
||||
Relays:
|
||||
IntroductionPoints:
|
||||
- NODEFINGERPRINT
|
||||
...
|
||||
# ExitNodes will be overwritten by this program
|
||||
ExitNodes: []
|
||||
IntroductionPoints: []
|
||||
# use the Onions section to list onion services you want the
|
||||
# Introduction Points whitelisted - these points may change daily
|
||||
# Look in tor's notice.log for 'Every introduction point for service'
|
||||
Onions: []
|
||||
# use the Services list to list elays you want the whitelisted
|
||||
# Look in tor's notice.log for 'Wanted to contact directory mirror'
|
||||
Services: []
|
||||
|
||||
|
||||
By default all sections of the goodnodes.yaml are used as a whitelist.
|
||||
|
||||
Use the GoodNodes/Onions list to list onion services you want the
|
||||
Introduction Points whitelisted - these points may change daily
|
||||
Look in tor's notice.log for warnings of 'Every introduction point for service'
|
||||
|
||||
```--hs_dir``` ```default='/var/lib/tor'``` will make the program
|
||||
parse the files named ```hostname``` below this dir to find
|
||||
Hidden Services to whitelist.
|
||||
|
||||
The Introduction Points can change during the day, so you may want to
|
||||
rerun this program to freshen the list of Introduction Points. A full run
|
||||
that processes all the relays from stem can take 30 minutes, or run with:
|
||||
|
||||
```--saved_only``` will run the program with just cached information
|
||||
on the relats, but will update the Introduction Points from the Services.
|
||||
|
||||
/etc/tor/yaml/torrc-badnodes.yaml
|
||||
|
||||
BadNodes:
|
||||
ExcludeExitNodes:
|
||||
BadExit:
|
||||
# $0000000000000000000000000000000000000007
|
||||
# list the internet domains you know are bad so you don't
|
||||
# waste time trying to download contacts from them.
|
||||
ExcludeDomains: []
|
||||
ExcludeNodes:
|
||||
# BadExit will be overwritten by this program
|
||||
BadExit: []
|
||||
# list MyBadExit in --bad_sections if you want it used, to exclude nodes
|
||||
# or any others as a list separated by comma(,)
|
||||
MyBadExit: []
|
||||
|
||||
```
|
||||
That part requires [PyYAML](https://pyyaml.org/wiki/PyYAML)
|
||||
https://github.com/yaml/pyyaml/ or ```ruamel```: do
|
||||
|
|
@ -39,7 +74,7 @@ the advantage of the former is that it preserves comments.
|
|||
|
||||
(You may have to run this as the Tor user to get RW access to
|
||||
/run/tor/control, in which case the directory for the YAML files must
|
||||
be group Tor writeable, and its parents group Tor RX.)
|
||||
be group Tor writeable, and its parent's directories group Tor RX.)
|
||||
|
||||
Because you don't want to exclude the introduction points to any onion
|
||||
you want to connect to, ```--white_onions``` should whitelist the
|
||||
|
|
@ -47,6 +82,13 @@ introduction points to a comma sep list of onions; we fixed stem to do this:
|
|||
* https://github.com/torproject/stem/issues/96
|
||||
* https://gitlab.torproject.org/legacy/trac/-/issues/25417
|
||||
|
||||
Use the GoodNodes/Onions list in goodnodes.yaml to list onion services
|
||||
you want the Introduction Points whitelisted - these points may change daily.
|
||||
Look in tor's notice.log for 'Every introduction point for service'
|
||||
|
||||
```notice_log``` will parse the notice log for warnings about relays and
|
||||
services that will then be whitelisted.
|
||||
|
||||
```--torrc_output``` will write the torrc ExcludeNodes configuration to a file.
|
||||
|
||||
```--good_contacts``` will write the contact info as a ciiss dictionary
|
||||
|
|
@ -71,7 +113,7 @@ list of fingerprints to ```ExitNodes```, a whitelist of relays to use as exits.
|
|||
3. clean relays that don't have "good' contactinfo. (implies 1)
|
||||
```=Empty,NoEmail,NotGood```
|
||||
|
||||
The default is ```=Empty,NotGood``` ; ```NoEmail``` is inherently imperfect
|
||||
The default is ```Empty,NoEmail,NotGood``` ; ```NoEmail``` is inherently imperfect
|
||||
in that many of the contact-as-an-email are obfuscated, but we try anyway.
|
||||
|
||||
To be "good" the ContactInfo must:
|
||||
|
|
@ -80,81 +122,20 @@ To be "good" the ContactInfo must:
|
|||
3. must support getting the file with a valid SSL cert from a recognized authority
|
||||
4. (not in the spec but added by Python) must use a TLS SSL > v1
|
||||
5. must have a fingerprint list in the file
|
||||
6. must have the FP that got us the contactinfo in the fingerprint list in the file,
|
||||
6. must have the FP that got us the contactinfo in the fingerprint list in the file.
|
||||
|
||||
```--wait_boot``` is the number of seconds to wait for Tor to booststrap
|
||||
|
||||
```--wellknown_output``` will make the program write the well-known files
|
||||
(```/.well-known/tor-relay/rsa-fingerprint.txt```) to a directory.
|
||||
|
||||
```--torrc_output``` will write a file of the commands that it sends to
|
||||
the Tor controller, so you can include it in a ```/etc/toc/torrc```.
|
||||
|
||||
```--relays_output write the download relays in json to a file. The relays
|
||||
are downloaded from https://onionoo.torproject.org/details
|
||||
|
||||
For usage, do ```python3 exclude_badExits.py --help`
|
||||
See [exclude_badExits.txt](./exclude_badExits.txt)
|
||||
|
||||
|
||||
|
||||
## Usage
|
||||
```
|
||||
|
||||
usage: exclude_badExits.py [-h] [--https_cafile HTTPS_CAFILE]
|
||||
[--proxy_host PROXY_HOST] [--proxy_port PROXY_PORT]
|
||||
[--proxy_ctl PROXY_CTL] [--torrc TORRC]
|
||||
[--timeout TIMEOUT] [--good_nodes GOOD_NODES]
|
||||
[--bad_nodes BAD_NODES] [--bad_on BAD_ON]
|
||||
[--bad_contacts BAD_CONTACTS]
|
||||
[--strict_nodes {0,1}] [--wait_boot WAIT_BOOT]
|
||||
[--points_timeout POINTS_TIMEOUT]
|
||||
[--log_level LOG_LEVEL]
|
||||
[--bad_sections BAD_SECTIONS]
|
||||
[--white_onions WHITE_ONIONS]
|
||||
[--torrc_output TORRC_OUTPUT]
|
||||
[--relays_output RELAYS_OUTPUT]
|
||||
[--good_contacts GOOD_CONTACTS]
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
--https_cafile HTTPS_CAFILE
|
||||
Certificate Authority file (in PEM)
|
||||
--proxy_host PROXY_HOST, --proxy-host PROXY_HOST
|
||||
proxy host
|
||||
--proxy_port PROXY_PORT, --proxy-port PROXY_PORT
|
||||
proxy control port
|
||||
--proxy_ctl PROXY_CTL, --proxy-ctl PROXY_CTL
|
||||
control socket - or port
|
||||
--torrc TORRC torrc to check for suggestions
|
||||
--timeout TIMEOUT proxy download connect timeout
|
||||
--good_nodes GOOD_NODES
|
||||
Yaml file of good info that should not be excluded
|
||||
--bad_nodes BAD_NODES
|
||||
Yaml file of bad nodes that should also be excluded
|
||||
--bad_on BAD_ON comma sep list of conditions - Empty,NoEmail,NotGood
|
||||
--bad_contacts BAD_CONTACTS
|
||||
Yaml file of bad contacts that bad FPs are using
|
||||
--strict_nodes {0,1} Set StrictNodes: 1 is less anonymous but more secure,
|
||||
although some sites may be unreachable
|
||||
--wait_boot WAIT_BOOT
|
||||
Seconds to wait for Tor to booststrap
|
||||
--points_timeout POINTS_TIMEOUT
|
||||
Timeout for getting introduction points - must be long
|
||||
>120sec. 0 means disabled looking for IPs
|
||||
--log_level LOG_LEVEL
|
||||
10=debug 20=info 30=warn 40=error
|
||||
--bad_sections BAD_SECTIONS
|
||||
sections of the badnodes.yaml to use, comma separated,
|
||||
'' BROKEN
|
||||
--white_onions WHITE_ONIONS
|
||||
comma sep. list of onions to whitelist their
|
||||
introduction points - BROKEN
|
||||
--torrc_output TORRC_OUTPUT
|
||||
Write the torrc configuration to a file
|
||||
--relays_output RELAYS_OUTPUT
|
||||
Write the download relays in json to a file
|
||||
--good_contacts GOOD_CONTACTS
|
||||
Write the proof data of the included nodes to a YAML
|
||||
file
|
||||
|
||||
This extends nusenu's basic idea of using the stem library to dynamically
|
||||
exclude nodes that are likely to be bad by putting them on the ExcludeNodes or
|
||||
ExcludeExitNodes setting of a running Tor. *
|
||||
https://github.com/nusenu/noContactInfo_Exit_Excluder *
|
||||
https://github.com/TheSmashy/TorExitRelayExclude The basic idea is to exclude
|
||||
Exit nodes that do not have ContactInfo: *
|
||||
https://github.com/nusenu/ContactInfo-Information-Sharing-Specification That
|
||||
can be extended to relays that do not have an email in the contact, or to
|
||||
relays that do not have ContactInfo that is verified to include them.
|
||||
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -3,23 +3,36 @@
|
|||
|
||||
PROG=exclude_badExits.py
|
||||
SOCKS_PORT=9050
|
||||
SOCKS_HOST=127.0.0.1
|
||||
CAFILE=/etc/ssl/certs/ca-certificates.crt
|
||||
|
||||
# an example of running exclude_badExits with full debugging
|
||||
# expected to take an hour or so
|
||||
declare -a LARGS
|
||||
LARGS=(
|
||||
--log_level 10
|
||||
)
|
||||
# you may have a special python for installed packages
|
||||
EXE=`which python3.bash`
|
||||
LARGS+=(
|
||||
--strict_nodes 1
|
||||
--points_timeout 120
|
||||
--proxy-host 127.0.0.1
|
||||
--proxy-port $SOCKS_PORT
|
||||
|
||||
$EXE exclude_badExits.py --help > exclude_badExits.txt &
|
||||
$EXE -c 'from exclude_badExits import __doc__; print(__doc__)' >exclude_badExits.md
|
||||
# an example of running exclude_badExits with full debugging
|
||||
# expected to 20 minutes or so
|
||||
declare -a LARGS
|
||||
LARGS=(
|
||||
# --saved_only
|
||||
# --strict_nodes 1
|
||||
--points_timeout 150
|
||||
--log_level 10
|
||||
--https_cafile $CAFILE
|
||||
)
|
||||
[ -z "$socks_proxy" ] || \
|
||||
LARGS+=(
|
||||
--proxy-host $SOCKS_HOST
|
||||
--proxy-port $SOCKS_PORT
|
||||
)
|
||||
|
||||
if [ -f /var/lib/tor/.SelekTOR/3xx/cache/9050/notice.log ] ; then
|
||||
LARGS+=(--notice_log /var/lib/tor/.SelekTOR/3xx/cache/9050/notice.log)
|
||||
fi
|
||||
|
||||
if [ -d /var/lib/tor/hs ] ; then
|
||||
LARGS+=( --hs_dir /var/lib/tor/hs )
|
||||
fi
|
||||
|
||||
if [ -f '/run/tor/control' ] ; then
|
||||
LARGS+=(--proxy-ctl '/run/tor/control' )
|
||||
|
|
@ -35,7 +48,8 @@ LARGS+=( --white_onions $ddg )
|
|||
grep -q ^debian-tor /etc/group && TORU=debian-tor || {
|
||||
grep -q ^tor /etc/group && TORU=tor
|
||||
}
|
||||
sudo -u $TORU $EXE exclude_badExits.py "${LARGS[@]}" \
|
||||
# --saved_only
|
||||
sudo -u $TORU $EXE exclude_badExits.py "${LARGS[@]}" "$@" \
|
||||
2>&1|tee exclude_badExits6.log
|
||||
|
||||
# The DEBUG statements contain the detail of why the relay was considered bad.
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
76
exclude_badExits.txt
Normal file
76
exclude_badExits.txt
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
usage: exclude_badExits.py [-h] [--https_cafile HTTPS_CAFILE]
|
||||
[--proxy_host PROXY_HOST] [--proxy_port PROXY_PORT]
|
||||
[--proxy_ctl PROXY_CTL] [--torrc TORRC]
|
||||
[--timeout TIMEOUT] [--good_nodes GOOD_NODES]
|
||||
[--bad_nodes BAD_NODES] [--bad_on BAD_ON]
|
||||
[--bad_contacts BAD_CONTACTS] [--saved_only]
|
||||
[--strict_nodes {0,1}] [--wait_boot WAIT_BOOT]
|
||||
[--points_timeout POINTS_TIMEOUT]
|
||||
[--log_level LOG_LEVEL]
|
||||
[--bad_sections BAD_SECTIONS]
|
||||
[--white_onions WHITE_ONIONS]
|
||||
[--torrc_output TORRC_OUTPUT] [--hs_dir HS_DIR]
|
||||
[--notice_log NOTICE_LOG]
|
||||
[--relays_output RELAYS_OUTPUT]
|
||||
[--wellknown_output WELLKNOWN_OUTPUT]
|
||||
[--good_contacts GOOD_CONTACTS]
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
--https_cafile HTTPS_CAFILE
|
||||
Certificate Authority file (in PEM)
|
||||
--proxy_host PROXY_HOST, --proxy-host PROXY_HOST
|
||||
proxy host
|
||||
--proxy_port PROXY_PORT, --proxy-port PROXY_PORT
|
||||
proxy control port
|
||||
--proxy_ctl PROXY_CTL, --proxy-ctl PROXY_CTL
|
||||
control socket - or port
|
||||
--torrc TORRC torrc to check for suggestions
|
||||
--timeout TIMEOUT proxy download connect timeout
|
||||
--good_nodes GOOD_NODES
|
||||
Yaml file of good info that should not be excluded
|
||||
--bad_nodes BAD_NODES
|
||||
Yaml file of bad nodes that should also be excluded
|
||||
--bad_on BAD_ON comma sep list of conditions - Empty,NoEmail,NotGood
|
||||
--bad_contacts BAD_CONTACTS
|
||||
Yaml file of bad contacts that bad FPs are using
|
||||
--saved_only Just use the info in the last *.yaml files without
|
||||
querying the Tor controller
|
||||
--strict_nodes {0,1} Set StrictNodes: 1 is less anonymous but more secure,
|
||||
although some onion sites may be unreachable
|
||||
--wait_boot WAIT_BOOT
|
||||
Seconds to wait for Tor to booststrap
|
||||
--points_timeout POINTS_TIMEOUT
|
||||
Timeout for getting introduction points - must be long
|
||||
>120sec. 0 means disabled looking for IPs
|
||||
--log_level LOG_LEVEL
|
||||
10=debug 20=info 30=warn 40=error
|
||||
--bad_sections BAD_SECTIONS
|
||||
sections of the badnodes.yaml to use, in addition to
|
||||
BadExit, comma separated
|
||||
--white_onions WHITE_ONIONS
|
||||
comma sep. list of onions to whitelist their
|
||||
introduction points - BROKEN
|
||||
--torrc_output TORRC_OUTPUT
|
||||
Write the torrc configuration to a file
|
||||
--hs_dir HS_DIR Parse the files name hostname below this dir to find
|
||||
Hidden Services to whitelist
|
||||
--notice_log NOTICE_LOG
|
||||
Parse the notice log for relays and services
|
||||
--relays_output RELAYS_OUTPUT
|
||||
Write the download relays in json to a file
|
||||
--wellknown_output WELLKNOWN_OUTPUT
|
||||
Write the well-known files to a directory
|
||||
--good_contacts GOOD_CONTACTS
|
||||
Write the proof data of the included nodes to a YAML
|
||||
file
|
||||
|
||||
This extends nusenu's basic idea of using the stem library to dynamically
|
||||
exclude nodes that are likely to be bad by putting them on the ExcludeNodes or
|
||||
ExcludeExitNodes setting of a running Tor. *
|
||||
https://github.com/nusenu/noContactInfo_Exit_Excluder *
|
||||
https://github.com/TheSmashy/TorExitRelayExclude The basic idea is to exclude
|
||||
Exit nodes that do not have ContactInfo: *
|
||||
https://github.com/nusenu/ContactInfo-Information-Sharing-Specification That
|
||||
can be extended to relays that do not have an email in the contact, or to
|
||||
relays that do not have ContactInfo that is verified to include them.
|
||||
|
|
@ -33,18 +33,39 @@ bHAVE_TORR = shutil.which('tor-resolve')
|
|||
# in the wild we'll keep a copy here so we can avoid restesting
|
||||
yKNOWN_NODNS = """
|
||||
---
|
||||
- 0x0.is
|
||||
- a9.wtf
|
||||
- apt96.com
|
||||
- axims.net
|
||||
- backup.spekadyon.org
|
||||
- dfri.se
|
||||
- dotsrc.org
|
||||
- dtf.contact
|
||||
- ezyn.de
|
||||
- for-privacy.net
|
||||
- galtland.network
|
||||
- heraldonion.org
|
||||
- interfesse.net
|
||||
- kryptonit.org
|
||||
- linkspartei.org
|
||||
- mkg20001.io
|
||||
- nicdex.com
|
||||
- nx42.de
|
||||
- pineapple.cx
|
||||
- privacylayer.xyz
|
||||
- privacysvcs.net
|
||||
- prsv.ch
|
||||
- sebastian-elisa-pfeifer.eu
|
||||
- thingtohide.nl
|
||||
- tor-exit-2.aa78i2efsewr0neeknk.xyz
|
||||
- tor-exit-3.aa78i2efsewr0neeknk.xyz
|
||||
- tor.dlecan.com
|
||||
- tor.skankhunt42.pw
|
||||
- transliberation.today
|
||||
- tuxli.org
|
||||
- unzane.com
|
||||
- verification-for-nusenu.net
|
||||
- www.defcon.org
|
||||
"""
|
||||
# - 0x0.is
|
||||
# - aklad5.com
|
||||
|
|
@ -220,7 +241,8 @@ def lIntroductionPoints(controller=None, lOnions=[], itimeout=120, log_level=10)
|
|||
l += lp
|
||||
except (Empty, Timeout,) as e: # noqa
|
||||
LOG.warn(f"Timed out getting introduction points for {elt}")
|
||||
continue
|
||||
except stem.DescriptorUnavailable as e:
|
||||
LOG.error(e)
|
||||
except Exception as e:
|
||||
LOG.exception(e)
|
||||
return l
|
||||
|
|
|
|||
Loading…
Reference in a new issue