Merge pull request #4988

973403bc Adding initial support for broadcasting transactions over Tor (Lee Clagett)

ANONYMITY_NETWORKS.md

@@ -0,0 +1,179 @@
+# Anonymity Networks with Monero
+
+Currently only Tor has been integrated into Monero. Providing support for
+Kovri/I2P should be minimal, but has not yet been attempted. The usage of
+these networks is still considered experimental - there are a few pessimistic
+cases where privacy is leaked. The design is intended to maximize privacy of
+the source of a transaction by broadcasting it over an anonymity network, while
+relying on IPv4 for the remainder of messages to make surrounding node attacks
+(via sybil) more difficult.
+
+
+## Behavior
+
+If _any_ anonymity network is enabled, transactions being broadcast that lack
+a valid "context" (i.e. the transaction did not come from a p2p connection),
+will only be sent to peers on anonymity networks. If an anonymity network is
+enabled but no peers over an anonymity network are available, an error is
+logged and the transaction is kept for future broadcasting over an anonymity
+network. The transaction will not be broadcast unless an anonymity connection
+is made or until `monerod` is shutdown and restarted with only public
+connections enabled.
+
+
+## P2P Commands
+
+Only handshakes, peer timed syncs, and transaction broadcast messages are
+supported over anonymity networks. If one `--add-exclusive-node` onion address
+is specified, then no syncing will take place and only transaction broadcasting
+can occur. It is therefore recommended that `--add-exclusive-node` be combined
+with additional exclusive IPv4 address(es).
+
+
+## Usage
+
+Anonymity networks have no seed nodes (the feature is still considered
+experimental), so a user must specify an address. If configured properly,
+additional peers can be found through typical p2p peerlist sharing.
+
+### Outbound Connections
+
+Connecting to an anonymous address requires the command line option
+`--proxy` which tells `monerod` the ip/port of a socks proxy provided by a
+separate process. On most systems the configuration will look like:
+
+> `--proxy tor,127.0.0.1:9050,10`
+> `--proxy i2p,127.0.0.1:9000`
+
+which tells `monerod` that ".onion" p2p addresses can be forwarded to a socks
+proxy at IP 127.0.0.1 port 9050 with a max of 10 outgoing connections and
+".i2p" p2p addresses can be forwarded to a socks proxy at IP 127.0.0.1 port 9000
+with the default max outgoing connections. Since there are no seed nodes for
+anonymity connections, peers must be manually specified:
+
+> `--add-exclusive-node rveahdfho7wo4b2m.onion:28083`
+> `--add-peer rveahdfho7wo4b2m.onion:28083`
+
+Either option can be listed multiple times, and can specify any mix of Tor,
+I2P, and IPv4 addresses. Using `--add-exclusive-node` will prevent the usage of
+seed nodes on ALL networks, which will typically be undesireable.
+
+### Inbound Connections
+
+Receiving anonymity connections is done through the option
+`--anonymous-inbound`. This option tells `monerod` the inbound address, network
+type, and max connections:
+
+> `--anonymous-inbound rveahdfho7wo4b2m.onion:28083,127.0.0.28083,25`
+> `--anonymous-inbound foobar.i2p:5000,127.0.0.1:30000`
+
+which tells `monerod` that a max of 25 inbound Tor connections are being
+received at address "rveahdfho7wo4b2m.onion:28083" and forwarded to `monerod`
+localhost port 28083, and a default max I2P connections are being received at
+address "foobar.i2p:5000" and forwarded to `monerod` localhost port 30000.
+These addresses will be shared with outgoing peers, over the same network type,
+otherwise the peer will not be notified of the peer address by the proxy.
+
+### Network Types
+
+#### Tor
+
+Options `--add-exclusive-node` and `--add-peer` recognize ".onion" addresses,
+and will properly forward those addresses to the proxy provided with
+`--proxy tor,...`.
+
+Option `--anonymous-inbound` also recognizes ".onion" addresses, and will
+automatically be sent out to outgoing Tor connections so the peer can
+distribute the address to its other peers.
+
+##### Configuration
+
+Tor must be configured for hidden services. An example configuration ("torrc")
+might look like:
+
+> HiddenServiceDir /var/lib/tor/data/monero
+> HiddenServicePort 28083 127.0.0.1:28083
+
+This will store key information in `/var/lib/tor/data/monero` and will forward
+"Tor port" 28083 to port 28083 of ip 127.0.0.1. The file
+`/usr/lib/tor/data/monero/hostname` will contain the ".onion" address for use
+with `--anonymous-inbound`.
+
+#### Kovri/I2P
+
+Support for this network has not been implemented. Using ".i2p" addresses or
+specifying "i2p" will currently generate an error.
+
+
+## Privacy Limitations
+
+There are currently some techniques that could be used to _possibly_ identify
+the machine that broadcast a transaction over an anonymity network.
+
+### Timestamps
+
+The peer timed sync command sends the current time in the message. This value
+can be used to link an onion address to an IPv4/IPv6 address. If a peer first
+sees a transaction over Tor, it could _assume_ (possibly incorrectly) that the
+transaction originated from the peer. If both the Tor connection and an
+IPv4/IPv6 connection have timestamps that are approximately close in value they
+could be used to link the two connections. This is less likely to happen if the
+system clock is fairly accurate - many peers on the Monero network should have
+similar timestamps.
+
+#### Mitigation
+
+Keep the system clock accurate so that fingerprinting is more difficult. In
+the future a random offset might be applied to anonymity networks so that if
+the system clock is noticeably off (and therefore more fingerprintable),
+linking the public IPv4/IPv6 connections with the anonymity networks will be
+more difficult.
+
+### Bandwidth Usage
+
+An ISP can passively monitor `monerod` connections from a node and observe when
+a transaction is sent over a Tor/Kovri connection via timing analysis + size of
+data sent during that timeframe. Kovri should provide better protection against
+this attack - its connections are not circuit based. However, if a node is
+only using Kovri for broadcasting Monero transactions, the total aggregate of
+Kovri/I2P data would also leak information.
+
+#### Mitigation
+
+There is no current mitigation for the user right now. This attack is fairly
+sophisticated, and likely requires support from the internet host of a Monero
+user.
+
+In the near future, "whitening" the amount of data sent over anonymity network
+connections will be performed. An attempt will be made to make a transaction
+broadcast indistinguishable from a peer timed sync command.
+
+### Intermittent Monero Syncing
+
+If a user only runs `monerod` to send a transaction then quit, this can also
+be used by an ISP to link a user to a transaction.
+
+#### Mitigation
+
+Run `monerod` as often as possible to conceal when transactions are being sent.
+Future versions will also have peers that first receive a transaction over an
+anonymity network delay the broadcast to public peers by a randomized amount.
+This will not completetely mitigate a user who syncs up sends then quits, in
+part because this rule is not enforceable, so this mitigation strategy is
+simply a best effort attempt.
+
+### Active Bandwidth Shaping
+
+An attacker could attempt to bandwidth shape traffic in an attempt to determine
+the source of a Tor/Kovri/I2P connection. There isn't great mitigation against
+this, but Kovri/I2P should provide better protection against this attack since
+the connections are not circuit based.
+
+#### Mitigation
+
+The best mitigiation is to use Kovri/I2P instead of Tor. However, Kovri/I2P
+has a smaller set of users (less cover traffic) and academic reviews, so there
+is a tradeoff in potential isses. Also, anyone attempting this strategy really
+wants to uncover a user, it seems unlikely that this would be performed against
+every Tor/Kovri/I2P user.
+

README.md

@@ -615,6 +615,12 @@ See [README.i18n.md](README.i18n.md).
 
 ## Using Tor
 
+> There is a new, still experimental, [integration with Tor](ANONYMITY_NETWORKS.md). The
+> feature allows connecting over IPv4 and Tor simulatenously - IPv4 is used for
+> relaying blocks and relaying transactions received by peers whereas Tor is
+> used solely for relaying transactions received over local RPC. This provides
+> privacy and better protection against surrounding node (sybil) attacks.
+
 While Monero isn't made to integrate with Tor, it can be used wrapped with torsocks, by
 setting the following configuration parameters and environment variables:
 

contrib/epee/include/net/abstract_tcp_server2.h

@@ -41,6 +41,7 @@
 #include <boost/noncopyable.hpp>
 #include <boost/shared_ptr.hpp>
 #include <atomic>
+#include <cassert>
 #include <map>
 #include <memory>
 
@@ -87,14 +88,25 @@ namespace net_utils
   {
   public:
     typedef typename t_protocol_handler::connection_context t_connection_context;
-    /// Construct a connection with the given io_service.
 
+    struct shared_state : socket_stats
+    {
+      shared_state()
+        : socket_stats(), pfilter(nullptr), config()
+      {}
+
+      i_connection_filter* pfilter;
+      typename t_protocol_handler::config_type config;
+    };
+
+    /// Construct a connection with the given io_service.
     explicit connection( boost::asio::io_service& io_service,
-			typename t_protocol_handler::config_type& config, 
+                        boost::shared_ptr<shared_state> state,
-			std::atomic<long> &ref_sock_count,  // the ++/-- counter 
+			t_connection_type connection_type);
-			std::atomic<long> &sock_number, // the only increasing ++ number generator
+
-			i_connection_filter * &pfilter
+    explicit connection( boost::asio::ip::tcp::socket&& sock,
-			,t_connection_type connection_type);
+                        boost::shared_ptr<shared_state> state,
+			t_connection_type connection_type);
 
     virtual ~connection() noexcept(false);
     /// Get the socket associated with the connection.
@@ -103,6 +115,9 @@ namespace net_utils
     /// Start the first asynchronous operation for the connection.
     bool start(bool is_income, bool is_multithreaded);
 
+    // `real_remote` is the actual endpoint (if connection is to proxy, etc.)
+    bool start(bool is_income, bool is_multithreaded, network_address real_remote);
+
     void get_context(t_connection_context& context_){context_ = context;}
 
     void call_back_starter();
@@ -148,7 +163,6 @@ namespace net_utils
     //boost::array<char, 1024> buffer_;
 
     t_connection_context context;
-    i_connection_filter* &m_pfilter;
 
 	// TODO what do they mean about wait on destructor?? --rfree :
     //this should be the last one, because it could be wait on destructor, while other activities possible on other threads
@@ -210,7 +224,9 @@ namespace net_utils
     /// Stop the server.
     void send_stop_signal();
 
-    bool is_stop_signal_sent();
+    bool is_stop_signal_sent() const noexcept { return m_stop_signal_sent; };
+
+    const std::atomic<bool>& get_stop_signal() const noexcept { return m_stop_signal_sent; }
 
     void set_threads_prefix(const std::string& prefix_name);
 
@@ -220,17 +236,28 @@ namespace net_utils
 
     void set_connection_filter(i_connection_filter* pfilter);
 
+    void set_default_remote(epee::net_utils::network_address remote)
+    {
+      default_remote = std::move(remote);
+    }
+
+    bool add_connection(t_connection_context& out, boost::asio::ip::tcp::socket&& sock, network_address real_remote);
     bool connect(const std::string& adr, const std::string& port, uint32_t conn_timeot, t_connection_context& cn, const std::string& bind_ip = "0.0.0.0");
     template<class t_callback>
     bool connect_async(const std::string& adr, const std::string& port, uint32_t conn_timeot, const t_callback &cb, const std::string& bind_ip = "0.0.0.0");
 
-    typename t_protocol_handler::config_type& get_config_object(){return m_config;}
+    typename t_protocol_handler::config_type& get_config_object()
+    {
+      assert(m_state != nullptr); // always set in constructor
+      return m_state->config;
+    }
 
     int get_binded_port(){return m_port;}
 
     long get_connections_count() const
     {
-      auto connections_count = (m_sock_count > 0) ? (m_sock_count - 1) : 0; // Socket count minus listening socket
+      assert(m_state != nullptr); // always set in constructor
+      auto connections_count = m_state->sock_count > 0 ? (m_state->sock_count - 1) : 0; // Socket count minus listening socket
       return connections_count;
     }
 
@@ -292,9 +319,6 @@ namespace net_utils
       return true;
     }
 
-  protected:
-    typename t_protocol_handler::config_type m_config;
-
   private:
     /// Run the server's io_service loop.
     bool worker_thread();
@@ -303,21 +327,21 @@ namespace net_utils
 
     bool is_thread_worker();
 
+    const boost::shared_ptr<typename connection<t_protocol_handler>::shared_state> m_state;
+
     /// The io_service used to perform asynchronous operations.
     std::unique_ptr<boost::asio::io_service> m_io_service_local_instance;
     boost::asio::io_service& io_service_;    
 
     /// Acceptor used to listen for incoming connections.
     boost::asio::ip::tcp::acceptor acceptor_;
+    epee::net_utils::network_address default_remote;
 
     std::atomic<bool> m_stop_signal_sent;
     uint32_t m_port;
-	std::atomic<long> m_sock_count;
-	std::atomic<long> m_sock_number;
     std::string m_address;
     std::string m_thread_name_prefix; //TODO: change to enum server_type, now used
     size_t m_threads_count;
-    i_connection_filter* m_pfilter;
     std::vector<boost::shared_ptr<boost::thread> > m_threads;
     boost::thread::id m_main_thread_id;
     critical_section m_threads_lock;

contrib/epee/include/net/abstract_tcp_server2.inl

@@ -40,6 +40,7 @@
 #include <boost/asio/deadline_timer.hpp>
 #include <boost/date_time/posix_time/posix_time.hpp> // TODO
 #include <boost/thread/condition_variable.hpp> // TODO
+#include <boost/make_shared.hpp>
 #include "warnings.h"
 #include "string_tools.h"
 #include "misc_language.h"
@@ -62,6 +63,13 @@ namespace epee
 {
 namespace net_utils
 {
+  template<typename T>
+  T& check_and_get(boost::shared_ptr<T>& ptr)
+  {
+    CHECK_AND_ASSERT_THROW_MES(bool(ptr), "shared_state cannot be null");
+    return *ptr;
+  }
+
   /************************************************************************/
   /*                                                                      */
   /************************************************************************/
@@ -69,25 +77,31 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   connection<t_protocol_handler>::connection( boost::asio::io_service& io_service,
-  	typename t_protocol_handler::config_type& config, 
+                boost::shared_ptr<shared_state> state,
-		std::atomic<long> &ref_sock_count,  // the ++/-- counter 
+		t_connection_type connection_type
-		std::atomic<long> &sock_number, // the only increasing ++ number generator
+	)
-		i_connection_filter* &pfilter
+	: connection(boost::asio::ip::tcp::socket{io_service}, std::move(state), connection_type)
-		,t_connection_type connection_type
+  {
+  }
+
+  template<class t_protocol_handler>
+  connection<t_protocol_handler>::connection( boost::asio::ip::tcp::socket&& sock,
+                boost::shared_ptr<shared_state> state,
+		t_connection_type connection_type
 	)
 	: 
-		connection_basic(io_service, ref_sock_count, sock_number), 
+		connection_basic(std::move(sock), state),
-		m_protocol_handler(this, config, context),
+		m_protocol_handler(this, check_and_get(state).config, context),
-		m_pfilter( pfilter ),
 		m_connection_type( connection_type ),
 		m_throttle_speed_in("speed_in", "throttle_speed_in"),
 		m_throttle_speed_out("speed_out", "throttle_speed_out"),
-		m_timer(io_service),
+		m_timer(socket_.get_io_service()),
 		m_local(false),
 		m_ready_to_close(false)
   {
     MDEBUG("test, connection constructor set m_connection_type="<<m_connection_type);
   }
+
 PRAGMA_WARNING_DISABLE_VS(4355)
   //---------------------------------------------------------------------------------
   template<class t_protocol_handler>
@@ -127,34 +141,44 @@ PRAGMA_WARNING_DISABLE_VS(4355)
   {
     TRY_ENTRY();
 
+    boost::system::error_code ec;
+    auto remote_ep = socket_.remote_endpoint(ec);
+    CHECK_AND_NO_ASSERT_MES(!ec, false, "Failed to get remote endpoint: " << ec.message() << ':' << ec.value());
+    CHECK_AND_NO_ASSERT_MES(remote_ep.address().is_v4(), false, "IPv6 not supported here");
+
+    const unsigned long ip_{boost::asio::detail::socket_ops::host_to_network_long(remote_ep.address().to_v4().to_ulong())};
+    return start(is_income, is_multithreaded, ipv4_network_address{uint32_t(ip_), remote_ep.port()});
+    CATCH_ENTRY_L0("connection<t_protocol_handler>::start()", false);
+  }
+  //---------------------------------------------------------------------------------
+  template<class t_protocol_handler>
+  bool connection<t_protocol_handler>::start(bool is_income, bool is_multithreaded, network_address real_remote)
+  {
+    TRY_ENTRY();
+
     // Use safe_shared_from_this, because of this is public method and it can be called on the object being deleted
     auto self = safe_shared_from_this();
     if(!self)
       return false;
 
     m_is_multithreaded = is_multithreaded;
-
+    m_local = real_remote.is_loopback() || real_remote.is_local();
-    boost::system::error_code ec;
-    auto remote_ep = socket_.remote_endpoint(ec);
-    CHECK_AND_NO_ASSERT_MES(!ec, false, "Failed to get remote endpoint: " << ec.message() << ':' << ec.value());
-    CHECK_AND_NO_ASSERT_MES(remote_ep.address().is_v4(), false, "IPv6 not supported here");
-
-    auto local_ep = socket_.local_endpoint(ec);
-    CHECK_AND_NO_ASSERT_MES(!ec, false, "Failed to get local endpoint: " << ec.message() << ':' << ec.value());
-
-    context = boost::value_initialized<t_connection_context>();
-    const unsigned long ip_{boost::asio::detail::socket_ops::host_to_network_long(remote_ep.address().to_v4().to_ulong())};
-    m_local = epee::net_utils::is_ip_loopback(ip_) || epee::net_utils::is_ip_local(ip_);
 
     // create a random uuid, we don't need crypto strength here
     const boost::uuids::uuid random_uuid = boost::uuids::random_generator()();
 
-    context.set_details(random_uuid, epee::net_utils::ipv4_network_address(ip_, remote_ep.port()), is_income);
+    context = t_connection_context{};
+    context.set_details(random_uuid, std::move(real_remote), is_income);
+
+    boost::system::error_code ec;
+    auto local_ep = socket_.local_endpoint(ec);
+    CHECK_AND_NO_ASSERT_MES(!ec, false, "Failed to get local endpoint: " << ec.message() << ':' << ec.value());
+
     _dbg3("[sock " << socket_.native_handle() << "] new connection from " << print_connection_context_short(context) <<
       " to " << local_ep.address().to_string() << ':' << local_ep.port() <<
-      ", total sockets objects " << m_ref_sock_count);
+      ", total sockets objects " << get_stats().sock_count);
 
-    if(m_pfilter && !m_pfilter->is_remote_host_allowed(context.m_remote_address))
+    if(static_cast<shared_state&>(get_stats()).pfilter && !static_cast<shared_state&>(get_stats()).pfilter->is_remote_host_allowed(context.m_remote_address))
     {
       _dbg2("[sock " << socket_.native_handle() << "] host denied " << context.m_remote_address.host_str() << ", shutdowning connection");
       close();
@@ -279,7 +303,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
     }
     MDEBUG(" connection type " << to_string( m_connection_type ) << " "
         << socket_.local_endpoint().address().to_string() << ":" << socket_.local_endpoint().port()
-        << " <--> " << address << ":" << port);
+        << " <--> " << context.m_remote_address.str() << " (via " << address << ":" << port << ")");
   }
   //---------------------------------------------------------------------------------
   template<class t_protocol_handler>
@@ -784,12 +808,14 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   boosted_tcp_server<t_protocol_handler>::boosted_tcp_server( t_connection_type connection_type ) :
+    m_state(boost::make_shared<typename connection<t_protocol_handler>::shared_state>()),
     m_io_service_local_instance(new boost::asio::io_service()),
     io_service_(*m_io_service_local_instance.get()),
     acceptor_(io_service_),
+    default_remote(),
     m_stop_signal_sent(false), m_port(0), 
-	m_sock_count(0), m_sock_number(0), m_threads_count(0), 
+    m_threads_count(0),
-	m_pfilter(NULL), m_thread_index(0),
+    m_thread_index(0),
 		m_connection_type( connection_type ),
     new_connection_()
   {
@@ -799,11 +825,13 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   boosted_tcp_server<t_protocol_handler>::boosted_tcp_server(boost::asio::io_service& extarnal_io_service, t_connection_type connection_type) :
+    m_state(boost::make_shared<typename connection<t_protocol_handler>::shared_state>()),
     io_service_(extarnal_io_service),
     acceptor_(io_service_),
+    default_remote(),
     m_stop_signal_sent(false), m_port(0),
-		m_sock_count(0), m_sock_number(0), m_threads_count(0), 
+    m_threads_count(0),
-		m_pfilter(NULL), m_thread_index(0),
+    m_thread_index(0),
 		m_connection_type(connection_type),
     new_connection_()
   {
@@ -844,7 +872,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
     boost::asio::ip::tcp::endpoint binded_endpoint = acceptor_.local_endpoint();
     m_port = binded_endpoint.port();
     MDEBUG("start accept");
-    new_connection_.reset(new connection<t_protocol_handler>(io_service_, m_config, m_sock_count, m_sock_number, m_pfilter, m_connection_type));
+    new_connection_.reset(new connection<t_protocol_handler>(io_service_, m_state, m_connection_type));
     acceptor_.async_accept(new_connection_->socket(),
       boost::bind(&boosted_tcp_server<t_protocol_handler>::handle_accept, this,
       boost::asio::placeholders::error));
@@ -922,7 +950,8 @@ POP_WARNINGS
   template<class t_protocol_handler>
   void boosted_tcp_server<t_protocol_handler>::set_connection_filter(i_connection_filter* pfilter)
   {
-    m_pfilter = pfilter;
+    assert(m_state != nullptr); // always set in constructor
+    m_state->pfilter = pfilter;
   }
   //---------------------------------------------------------------------------------
   template<class t_protocol_handler>
@@ -1030,12 +1059,6 @@ POP_WARNINGS
   }
   //---------------------------------------------------------------------------------
   template<class t_protocol_handler>
-  bool boosted_tcp_server<t_protocol_handler>::is_stop_signal_sent()
-  {
-    return m_stop_signal_sent;
-  }
-  //---------------------------------------------------------------------------------
-  template<class t_protocol_handler>
   void boosted_tcp_server<t_protocol_handler>::handle_accept(const boost::system::error_code& e)
   {
     MDEBUG("handle_accept");
@@ -1048,7 +1071,7 @@ POP_WARNINGS
 			new_connection_->setRpcStation(); // hopefully this is not needed actually
 		}
 		connection_ptr conn(std::move(new_connection_));
-      new_connection_.reset(new connection<t_protocol_handler>(io_service_, m_config, m_sock_count, m_sock_number, m_pfilter, m_connection_type));
+      new_connection_.reset(new connection<t_protocol_handler>(io_service_, m_state, m_connection_type));
       acceptor_.async_accept(new_connection_->socket(),
         boost::bind(&boosted_tcp_server<t_protocol_handler>::handle_accept, this,
         boost::asio::placeholders::error));
@@ -1056,7 +1079,10 @@ POP_WARNINGS
       boost::asio::socket_base::keep_alive opt(true);
       conn->socket().set_option(opt);
 
+      if (default_remote.get_type_id() == net_utils::address_type::invalid)
         conn->start(true, 1 < m_threads_count);
+      else
+        conn->start(true, 1 < m_threads_count, default_remote);
       conn->save_dbg_log();
       return;
     }
@@ -1071,20 +1097,41 @@ POP_WARNINGS
     }
 
     // error path, if e or exception
-    _erro("Some problems at accept: " << e.message() << ", connections_count = " << m_sock_count);
+    assert(m_state != nullptr); // always set in constructor
+    _erro("Some problems at accept: " << e.message() << ", connections_count = " << m_state->sock_count);
     misc_utils::sleep_no_w(100);
-    new_connection_.reset(new connection<t_protocol_handler>(io_service_, m_config, m_sock_count, m_sock_number, m_pfilter, m_connection_type));
+    new_connection_.reset(new connection<t_protocol_handler>(io_service_, m_state, m_connection_type));
     acceptor_.async_accept(new_connection_->socket(),
       boost::bind(&boosted_tcp_server<t_protocol_handler>::handle_accept, this,
       boost::asio::placeholders::error));
   }
   //---------------------------------------------------------------------------------
   template<class t_protocol_handler>
+  bool boosted_tcp_server<t_protocol_handler>::add_connection(t_connection_context& out, boost::asio::ip::tcp::socket&& sock, network_address real_remote)
+  {
+    if(std::addressof(get_io_service()) == std::addressof(sock.get_io_service()))
+    {
+      connection_ptr conn(new connection<t_protocol_handler>(std::move(sock), m_state, m_connection_type));
+      if(conn->start(false, 1 < m_threads_count, std::move(real_remote)))
+      {
+        conn->get_context(out);
+        conn->save_dbg_log();
+        return true;
+      }
+    }
+    else
+    {
+	MWARNING(out << " was not added, socket/io_service mismatch");
+    }
+    return false;
+  }
+  //---------------------------------------------------------------------------------
+  template<class t_protocol_handler>
   bool boosted_tcp_server<t_protocol_handler>::connect(const std::string& adr, const std::string& port, uint32_t conn_timeout, t_connection_context& conn_context, const std::string& bind_ip)
   {
     TRY_ENTRY();
 
-    connection_ptr new_connection_l(new connection<t_protocol_handler>(io_service_, m_config, m_sock_count, m_sock_number, m_pfilter, m_connection_type) );
+    connection_ptr new_connection_l(new connection<t_protocol_handler>(io_service_, m_state, m_connection_type) );
     connections_mutex.lock();
     connections_.insert(new_connection_l);
     MDEBUG("connections_ size now " << connections_.size());
@@ -1187,7 +1234,8 @@ POP_WARNINGS
     }
     else
     {
-      _erro("[sock " << new_connection_l->socket().native_handle() << "] Failed to start connection, connections_count = " << m_sock_count);
+      assert(m_state != nullptr); // always set in constructor
+      _erro("[sock " << new_connection_l->socket().native_handle() << "] Failed to start connection, connections_count = " << m_state->sock_count);
     }
     
 	new_connection_l->save_dbg_log();
@@ -1201,7 +1249,7 @@ POP_WARNINGS
   bool boosted_tcp_server<t_protocol_handler>::connect_async(const std::string& adr, const std::string& port, uint32_t conn_timeout, const t_callback &cb, const std::string& bind_ip)
   {
     TRY_ENTRY();    
-    connection_ptr new_connection_l(new connection<t_protocol_handler>(io_service_, m_config, m_sock_count, m_sock_number, m_pfilter, m_connection_type) );
+    connection_ptr new_connection_l(new connection<t_protocol_handler>(io_service_, m_state, m_connection_type) );
     connections_mutex.lock();
     connections_.insert(new_connection_l);
     MDEBUG("connections_ size now " << connections_.size());

contrib/epee/include/net/connection_basic.hpp

@@ -55,6 +55,15 @@ namespace epee
 {
 namespace net_utils
 {
+  struct socket_stats
+  {
+    socket_stats()
+      : sock_count(0), sock_number(0)
+    {}
+
+    std::atomic<long> sock_count;
+    std::atomic<long> sock_number;
+  };
 
   /************************************************************************/
   /*                                                                      */
@@ -72,6 +81,8 @@ class connection_basic_pimpl; // PIMPL for this class
   std::string to_string(t_connection_type type);
 
 class connection_basic { // not-templated base class for rapid developmet of some code parts
+    // beware of removing const, net_utils::connection is sketchily doing a cast to prevent storing ptr twice
+    const boost::shared_ptr<socket_stats> m_stats;
 	public:
 		std::unique_ptr< connection_basic_pimpl > mI; // my Implementation
 
@@ -86,13 +97,15 @@ class connection_basic { // not-templated base class for rapid developmet of som
     /// Socket for the connection.
     boost::asio::ip::tcp::socket socket_;
 
-		std::atomic<long> &m_ref_sock_count; // reference to external counter of existing sockets that we will ++/--
 	public:
 		// first counter is the ++/-- count of current sockets, the other socket_number is only-increasing ++ number generator
-		connection_basic(boost::asio::io_service& io_service, std::atomic<long> &ref_sock_count, std::atomic<long> &sock_number);
+		connection_basic(boost::asio::ip::tcp::socket&& socket, boost::shared_ptr<socket_stats> stats);
 
 		virtual ~connection_basic() noexcept(false);
 
+                //! \return `socket_stats` object passed in construction (ptr never changes).
+		socket_stats& get_stats() noexcept { return *m_stats; /* verified in constructor */ }
+
 		// various handlers to be called from connection class:
 		void do_send_handler_write(const void * ptr , size_t cb);
 		void do_send_handler_write_from_queue(const boost::system::error_code& e, size_t cb , int q_len); // from handle_write, sending next part

contrib/epee/include/net/enums.h

@@ -0,0 +1,65 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/utility/string_ref.hpp>
+#include <cstdint>
+
+namespace epee
+{
+namespace net_utils
+{
+	enum class address_type : std::uint8_t
+	{
+		// Do not change values, this will break serialization
+		invalid = 0,
+		ipv4 = 1,
+		ipv6 = 2,
+		i2p = 3,
+		tor = 4
+	};
+
+	enum class zone : std::uint8_t
+	{
+		invalid = 0,
+		public_ = 1, // public is keyword
+		i2p = 2,
+		tor = 3
+	};
+
+	// implementations in src/net_utils_base.cpp
+
+	//! \return String name of zone or "invalid" on error.
+	const char* zone_to_string(zone value) noexcept;
+
+	//! \return `zone` enum of `value` or `zone::invalid` on error.
+	zone zone_from_string(boost::string_ref value) noexcept;
+} // net_utils
+} // epee
+

contrib/epee/include/net/net_utils_base.h

@@ -33,6 +33,7 @@
 #include <boost/asio/io_service.hpp>
 #include <typeinfo>
 #include <type_traits>
+#include "enums.h"
 #include "serialization/keyvalue_serialization.h"
 #include "misc_log_ex.h"
 
@@ -43,6 +44,11 @@
 #define MAKE_IP( a1, a2, a3, a4 )	(a1|(a2<<8)|(a3<<16)|(a4<<24))
 #endif
 
+namespace net
+{
+	class tor_address;
+}
+
 namespace epee
 {
 namespace net_utils
@@ -53,6 +59,10 @@ namespace net_utils
 		uint16_t m_port;
 
 	public:
+		constexpr ipv4_network_address() noexcept
+			: ipv4_network_address(0, 0)
+		{}
+
 		constexpr ipv4_network_address(uint32_t ip, uint16_t port) noexcept
 			: m_ip(ip), m_port(port) {}
 
@@ -67,9 +77,10 @@ namespace net_utils
 		std::string host_str() const;
 		bool is_loopback() const;
 		bool is_local() const;
-		static constexpr uint8_t get_type_id() noexcept { return ID; }
+		static constexpr address_type get_type_id() noexcept { return address_type::ipv4; }
+		static constexpr zone get_zone() noexcept { return zone::public_; }
+		static constexpr bool is_blockable() noexcept { return true; }
 
-		static const uint8_t ID = 1;
 		BEGIN_KV_SERIALIZE_MAP()
 			KV_SERIALIZE(m_ip)
 			KV_SERIALIZE(m_port)
@@ -103,7 +114,9 @@ namespace net_utils
 			virtual std::string host_str() const = 0;
 			virtual bool is_loopback() const = 0;
 			virtual bool is_local() const = 0;
-			virtual uint8_t get_type_id() const = 0;
+			virtual address_type get_type_id() const = 0;
+			virtual zone get_zone() const = 0;
+			virtual bool is_blockable() const = 0;
 		};
 
 		template<typename T>
@@ -131,7 +144,9 @@ namespace net_utils
 			virtual std::string host_str() const override { return value.host_str(); }
 			virtual bool is_loopback() const override { return value.is_loopback(); }
 			virtual bool is_local() const override { return value.is_local(); }
-			virtual uint8_t get_type_id() const override { return value.get_type_id(); }
+			virtual address_type get_type_id() const override { return value.get_type_id(); }
+			virtual zone get_zone() const override { return value.get_zone(); }
+			virtual bool is_blockable() const override { return value.is_blockable(); }
 		};
 
 		std::shared_ptr<interface> self;
@@ -146,6 +161,23 @@ namespace net_utils
 				throw std::bad_cast{};
 			return static_cast<implementation<Type_>*>(self_)->value;
 		}
+
+		template<typename T, typename t_storage>
+		bool serialize_addr(std::false_type, t_storage& stg, typename t_storage::hsection hparent)
+		{
+			T addr{};
+			if (!epee::serialization::selector<false>::serialize(addr, stg, hparent, "addr"))
+				return false;
+			*this = std::move(addr);
+			return true;
+		}
+
+		template<typename T, typename t_storage>
+		bool serialize_addr(std::true_type, t_storage& stg, typename t_storage::hsection hparent) const
+		{
+			return epee::serialization::selector<true>::serialize(as<T>(), stg, hparent, "addr");
+		}
+
 	public:
 		network_address() : self(nullptr) {}
 		template<typename T>
@@ -158,43 +190,32 @@ namespace net_utils
 		std::string host_str() const { return self ? self->host_str() : "<none>"; }
 		bool is_loopback() const { return self ? self->is_loopback() : false; }
 		bool is_local() const { return self ? self->is_local() : false; }
-		uint8_t get_type_id() const { return self ? self->get_type_id() : 0; }
+		address_type get_type_id() const { return self ? self->get_type_id() : address_type::invalid; }
+		zone get_zone() const { return self ? self->get_zone() : zone::invalid; }
+		bool is_blockable() const { return self ? self->is_blockable() : false; }
 		template<typename Type> const Type &as() const { return as_mutable<const Type>(); }
 
 		BEGIN_KV_SERIALIZE_MAP()
-			uint8_t type = is_store ? this_ref.get_type_id() : 0;
+			// need to `#include "net/tor_address.h"` when serializing `network_address`
+			static constexpr std::integral_constant<bool, is_store> is_store_{};
+
+			std::uint8_t type = std::uint8_t(is_store ? this_ref.get_type_id() : address_type::invalid);
 			if (!epee::serialization::selector<is_store>::serialize(type, stg, hparent_section, "type"))
 				return false;
-			switch (type)
+
+			switch (address_type(type))
 			{
-				case ipv4_network_address::ID:
+				case address_type::ipv4:
-				{
+					return this_ref.template serialize_addr<ipv4_network_address>(is_store_, stg, hparent_section);
-					if (!is_store)
+				case address_type::tor:
-					{
+					return this_ref.template serialize_addr<net::tor_address>(is_store_, stg, hparent_section);
-						const_cast<network_address&>(this_ref) = ipv4_network_address{0, 0};
+				case address_type::invalid:
-						auto &addr = this_ref.template as_mutable<ipv4_network_address>();
+				default:
-						if (epee::serialization::selector<is_store>::serialize(addr, stg, hparent_section, "addr"))
-							MDEBUG("Found as addr: " << this_ref.str());
-						else if (epee::serialization::selector<is_store>::serialize(addr, stg, hparent_section, "template as<ipv4_network_address>()"))
-							MDEBUG("Found as template as<ipv4_network_address>(): " << this_ref.str());
-						else if (epee::serialization::selector<is_store>::serialize(addr, stg, hparent_section, "template as_mutable<ipv4_network_address>()"))
-							MDEBUG("Found as template as_mutable<ipv4_network_address>(): " << this_ref.str());
-						else
-						{
-							MWARNING("Address not found");
-							return false;
-						}
-					}
-					else
-					{
-						auto &addr = this_ref.template as_mutable<ipv4_network_address>();
-						if (!epee::serialization::selector<is_store>::serialize(addr, stg, hparent_section, "addr"))
-							return false;
-					}
 					break;
 			}
-				default: MERROR("Unsupported network address type: " << (unsigned)type); return false;
+
-			}
+			MERROR("Unsupported network address type: " << (unsigned)type);
+			return false;
 		END_KV_SERIALIZE_MAP()
 	};
 
@@ -211,8 +232,6 @@ namespace net_utils
 	inline bool operator>=(const network_address& lhs, const network_address& rhs)
 	{ return !lhs.less(rhs); }
 
-	bool create_network_address(network_address &address, const std::string &string, uint16_t default_port = 0);
-
 	/************************************************************************/
 	/*                                                                      */
 	/************************************************************************/
@@ -250,7 +269,7 @@ namespace net_utils
     {}
 
     connection_context_base(): m_connection_id(),
-                               m_remote_address(ipv4_network_address{0,0}),
+                               m_remote_address(),
                                m_is_income(false),
                                m_started(time(NULL)),
                                m_last_recv(0),

contrib/epee/src/connection_basic.cpp

@@ -103,7 +103,7 @@ namespace net_utils
 // connection_basic_pimpl
 // ================================================================================================
 	
-connection_basic_pimpl::connection_basic_pimpl(const std::string &name) : m_throttle(name) { }
+connection_basic_pimpl::connection_basic_pimpl(const std::string &name) : m_throttle(name), m_peer_number(0) { }
 
 // ================================================================================================
 // connection_basic
@@ -113,27 +113,31 @@ connection_basic_pimpl::connection_basic_pimpl(const std::string &name) : m_thro
 int connection_basic_pimpl::m_default_tos;
 
 // methods:
-connection_basic::connection_basic(boost::asio::io_service& io_service, std::atomic<long> &ref_sock_count, std::atomic<long> &sock_number)
+connection_basic::connection_basic(boost::asio::ip::tcp::socket&& socket, boost::shared_ptr<socket_stats> stats)
 	:
+	m_stats(std::move(stats)),
 	mI( new connection_basic_pimpl("peer") ),
-	strand_(io_service),
+	strand_(socket.get_io_service()),
-	socket_(io_service),
+	socket_(std::move(socket)),
 	m_want_close_connection(false), 
-	m_was_shutdown(false),
+	m_was_shutdown(false)
-	m_ref_sock_count(ref_sock_count)
 {
-	++ref_sock_count; // increase the global counter
+	// add nullptr checks if removed
-	mI->m_peer_number = sock_number.fetch_add(1); // use, and increase the generated number
+	CHECK_AND_ASSERT_THROW_MES(bool(m_stats), "stats shared_ptr cannot be null");
+
+	++(m_stats->sock_count); // increase the global counter
+	mI->m_peer_number = m_stats->sock_number.fetch_add(1); // use, and increase the generated number
 
 	std::string remote_addr_str = "?";
 	try { boost::system::error_code e; remote_addr_str = socket_.remote_endpoint(e).address().to_string(); } catch(...){} ;
 
-	_note("Spawned connection p2p#"<<mI->m_peer_number<<" to " << remote_addr_str << " currently we have sockets count:" << m_ref_sock_count);
+	_note("Spawned connection p2p#"<<mI->m_peer_number<<" to " << remote_addr_str << " currently we have sockets count:" << m_stats->sock_count);
 }
 
 connection_basic::~connection_basic() noexcept(false) {
+	--(m_stats->sock_count);
+
 	std::string remote_addr_str = "?";
-	m_ref_sock_count--;
 	try { boost::system::error_code e; remote_addr_str = socket_.remote_endpoint(e).address().to_string(); } catch(...){} ;
 	_note("Destructing connection p2p#"<<mI->m_peer_number << " to " << remote_addr_str);
 }

contrib/epee/src/net_utils_base.cpp

@@ -8,8 +8,6 @@
 
 namespace epee { namespace net_utils
 {
-	const uint8_t ipv4_network_address::ID;
-
 	bool ipv4_network_address::equal(const ipv4_network_address& other) const noexcept
 	{ return is_same_host(other) && port() == other.port(); }
 
@@ -58,20 +56,6 @@ namespace epee { namespace net_utils
 		return self_->is_same_host(*other_self);
 	}
 
-	bool create_network_address(network_address &address, const std::string &string, uint16_t default_port)
-	{
-		uint32_t ip;
-		uint16_t port;
-		if (epee::string_tools::parse_peer_from_string(ip, port, string))
-		{
-			if (default_port && !port)
-				port = default_port;
-			address = ipv4_network_address{ip, port};
-			return true;
-		}
-		return false;
-	}
-
   std::string print_connection_context(const connection_context_base& ctx)
   {
     std::stringstream ss;
@@ -86,5 +70,31 @@ namespace epee { namespace net_utils
     return ss.str();
   }
 
+  const char* zone_to_string(zone value) noexcept
+  {
+    switch (value)
+    {
+    case zone::public_:
+      return "public";
+    case zone::i2p:
+      return "i2p";
+    case zone::tor:
+      return "tor";
+    default:
+      break;
+    }
+    return "invalid";
+  }
+
+  zone zone_from_string(const boost::string_ref value) noexcept
+  {
+    if (value == "public")
+      return zone::public_;
+    if (value == "i2p")
+      return zone::i2p;
+    if (value == "tor")
+      return zone::tor;
+    return zone::invalid;
+  }
 }}
 

src/CMakeLists.txt

@@ -110,6 +110,7 @@ add_subdirectory(checkpoints)
 add_subdirectory(cryptonote_basic)
 add_subdirectory(cryptonote_core)
 add_subdirectory(multisig)
+add_subdirectory(net)
 if(NOT IOS)
   add_subdirectory(blockchain_db)
 endif()

src/cryptonote_config.h

@@ -108,6 +108,7 @@
 #define P2P_DEFAULT_PACKET_MAX_SIZE                     50000000     //50000000 bytes maximum packet size
 #define P2P_DEFAULT_PEERS_IN_HANDSHAKE                  250
 #define P2P_DEFAULT_CONNECTION_TIMEOUT                  5000       //5 seconds
+#define P2P_DEFAULT_TOR_CONNECT_TIMEOUT                 20         // seconds
 #define P2P_DEFAULT_PING_CONNECTION_TIMEOUT             2000       //2 seconds
 #define P2P_DEFAULT_INVOKE_TIMEOUT                      60*2*1000  //2 minutes
 #define P2P_DEFAULT_HANDSHAKE_INVOKE_TIMEOUT            5000       //5 seconds

src/cryptonote_protocol/cryptonote_protocol_handler.h

@@ -173,15 +173,6 @@ namespace cryptonote
         //handler_response_blocks_now(blob.size()); // XXX
         return m_p2p->invoke_notify_to_peer(t_parameter::ID, epee::strspan<uint8_t>(blob), context);
       }
-
-      template<class t_parameter>
-      bool relay_post_notify(typename t_parameter::request& arg, cryptonote_connection_context& exclude_context)
-      {
-        LOG_PRINT_L2("[" << epee::net_utils::print_connection_context_short(exclude_context) << "] post relay " << typeid(t_parameter).name() << " -->");
-        std::string arg_buff;
-        epee::serialization::store_t_to_binary(arg, arg_buff);
-        return m_p2p->relay_notify_to_all(t_parameter::ID, epee::strspan<uint8_t>(arg_buff), exclude_context);
-      }
   };
 
 } // namespace

src/cryptonote_protocol/cryptonote_protocol_handler.inl

@@ -226,7 +226,7 @@ namespace cryptonote
       cnx.host = cntxt.m_remote_address.host_str();
       cnx.ip = "";
       cnx.port = "";
-      if (cntxt.m_remote_address.get_type_id() == epee::net_utils::ipv4_network_address::ID)
+      if (cntxt.m_remote_address.get_type_id() == epee::net_utils::ipv4_network_address::get_type_id())
       {
         cnx.ip = cnx.host;
         cnx.port = std::to_string(cntxt.m_remote_address.as<epee::net_utils::ipv4_network_address>().port());
@@ -333,6 +333,13 @@ namespace cryptonote
       return true;
     }
 
+    // No chain synchronization over hidden networks (tor, i2p, etc.)
+    if(context.m_remote_address.get_zone() != epee::net_utils::zone::public_)
+    {
+      context.m_state = cryptonote_connection_context::state_normal;
+      return true;
+    }
+
     if (hshd.current_height > target)
     {
     /* As I don't know if accessing hshd from core could be a good practice,
@@ -2058,20 +2065,20 @@ skip:
     fluffy_arg.b.txs = fluffy_txs;
 
     // sort peers between fluffy ones and others
-    std::list<boost::uuids::uuid> fullConnections, fluffyConnections;
+    std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> fullConnections, fluffyConnections;
     m_p2p->for_each_connection([this, &exclude_context, &fullConnections, &fluffyConnections](connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)
     {
-      if (peer_id && exclude_context.m_connection_id != context.m_connection_id)
+      if (peer_id && exclude_context.m_connection_id != context.m_connection_id && context.m_remote_address.get_zone() == epee::net_utils::zone::public_)
       {
         if(m_core.fluffy_blocks_enabled() && (support_flags & P2P_SUPPORT_FLAG_FLUFFY_BLOCKS))
         {
           LOG_DEBUG_CC(context, "PEER SUPPORTS FLUFFY BLOCKS - RELAYING THIN/COMPACT WHATEVER BLOCK");
-          fluffyConnections.push_back(context.m_connection_id);
+          fluffyConnections.push_back({context.m_remote_address.get_zone(), context.m_connection_id});
         }
         else
         {
           LOG_DEBUG_CC(context, "PEER DOESN'T SUPPORT FLUFFY BLOCKS - RELAYING FULL BLOCK");
-          fullConnections.push_back(context.m_connection_id);
+          fullConnections.push_back({context.m_remote_address.get_zone(), context.m_connection_id});
         }
       }
       return true;
@@ -2082,13 +2089,13 @@ skip:
     {
       std::string fluffyBlob;
       epee::serialization::store_t_to_binary(fluffy_arg, fluffyBlob);
-      m_p2p->relay_notify_to_list(NOTIFY_NEW_FLUFFY_BLOCK::ID, epee::strspan<uint8_t>(fluffyBlob), fluffyConnections);
+      m_p2p->relay_notify_to_list(NOTIFY_NEW_FLUFFY_BLOCK::ID, epee::strspan<uint8_t>(fluffyBlob), std::move(fluffyConnections));
     }
     if (!fullConnections.empty())
     {
       std::string fullBlob;
       epee::serialization::store_t_to_binary(arg, fullBlob);
-      m_p2p->relay_notify_to_list(NOTIFY_NEW_BLOCK::ID, epee::strspan<uint8_t>(fullBlob), fullConnections);
+      m_p2p->relay_notify_to_list(NOTIFY_NEW_BLOCK::ID, epee::strspan<uint8_t>(fullBlob), std::move(fullConnections));
     }
 
     return true;
@@ -2097,6 +2104,12 @@ skip:
   template<class t_core>
   bool t_cryptonote_protocol_handler<t_core>::relay_transactions(NOTIFY_NEW_TRANSACTIONS::request& arg, cryptonote_connection_context& exclude_context)
   {
+    const bool hide_tx_broadcast =
+      1 < m_p2p->get_zone_count() && exclude_context.m_remote_address.get_zone() == epee::net_utils::zone::invalid;
+
+    if (hide_tx_broadcast)
+      MDEBUG("Attempting to conceal origin of tx via anonymity network connection(s)");
+
     // no check for success, so tell core they're relayed unconditionally
     const bool pad_transactions = m_core.pad_transactions();
     size_t bytes = pad_transactions ? 9 /* header */ + 4 /* 1 + 'txs' */ + tools::get_varint_data(arg.txs.size()).size() : 0;
@@ -2131,7 +2144,30 @@ skip:
       // if the size of _ moved enough, we might lose byte in size encoding, we don't care
     }
 
-    return relay_post_notify<NOTIFY_NEW_TRANSACTIONS>(arg, exclude_context);
+    std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections;
+    m_p2p->for_each_connection([hide_tx_broadcast, &exclude_context, &connections](connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)
+    {
+      const epee::net_utils::zone current_zone = context.m_remote_address.get_zone();
+      const bool broadcast_to_peer =
+        peer_id &&
+        (hide_tx_broadcast != bool(current_zone == epee::net_utils::zone::public_)) &&
+	exclude_context.m_connection_id != context.m_connection_id;
+
+      if (broadcast_to_peer)
+        connections.push_back({current_zone, context.m_connection_id});
+
+      return true;
+    });
+
+    if (connections.empty())
+      MERROR("Transaction not relayed - no" << (hide_tx_broadcast ? " privacy": "") << " peers available");
+    else
+    {
+      std::string fullBlob;
+      epee::serialization::store_t_to_binary(arg, fullBlob);
+      m_p2p->relay_notify_to_list(NOTIFY_NEW_TRANSACTIONS::ID, epee::strspan<uint8_t>(fullBlob), std::move(connections));
+    }
+    return true;
   }
   //------------------------------------------------------------------------------------------------------------------------
   template<class t_core>

src/net/CMakeLists.txt

@@ -0,0 +1,34 @@
+# Copyright (c) 2018, The Monero Project
+#
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+set(net_sources error.cpp parse.cpp socks.cpp tor_address.cpp)
+set(net_headers error.h parse.h socks.h tor_address.h)
+
+monero_add_library(net ${net_sources} ${net_headers})
+target_link_libraries(net epee ${Boost_ASIO_LIBRARY})
+

src/net/error.cpp

@@ -0,0 +1,92 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "error.h"
+
+#include <string>
+
+namespace
+{
+    struct net_category : std::error_category
+    {
+        net_category() noexcept
+          : std::error_category()
+        {}
+
+        const char* name() const noexcept override
+        {
+            return "net::error_category";
+        }
+
+        std::string message(int value) const override
+        {
+            switch (net::error(value))
+            {
+            case net::error::expected_tld:
+                return "Expected top-level domain";
+            case net::error::invalid_host:
+                return "Host value is not valid";
+            case net::error::invalid_i2p_address:
+                return "Invalid I2P address";
+            case net::error::invalid_port:
+                return "Invalid port value (expected 0-65535)";
+            case net::error::invalid_tor_address:
+                return "Invalid Tor address";
+            case net::error::unsupported_address:
+                return "Network address not supported";
+            default:
+                break;
+            }
+
+            return "Unknown net::error";
+        }
+
+        std::error_condition default_error_condition(int value) const noexcept override
+        {
+            switch (net::error(value))
+            {
+            case net::error::invalid_port:
+                return std::errc::result_out_of_range;
+            case net::error::expected_tld:
+            case net::error::invalid_tor_address:
+            default:
+                break;
+            }
+            return std::error_condition{value, *this};
+        }
+    };
+} // anonymous
+
+namespace net
+{
+    std::error_category const& error_category() noexcept
+    {
+        static const net_category instance{};
+        return instance;
+    }
+}

src/net/error.h

@@ -0,0 +1,64 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <system_error>
+#include <type_traits>
+
+namespace net
+{
+    //! General net errors
+    enum class error : int
+    {
+        // 0 reserved for success (as per expect<T>)
+        expected_tld = 1,   //!< Expected a tld
+        invalid_host,       //!< Hostname is not valid
+        invalid_i2p_address,
+        invalid_port,       //!< Outside of 0-65535 range
+        invalid_tor_address,//!< Invalid base32 or length
+        unsupported_address //!< Type not supported by `get_network_address`
+    };
+
+    //! \return `std::error_category` for `net` namespace.
+    std::error_category const& error_category() noexcept;
+
+    //! \return `net::error` as a `std::error_code` value.
+    inline std::error_code make_error_code(error value) noexcept
+    {
+        return std::error_code{int(value), error_category()};
+    }
+}
+
+namespace std
+{
+    template<>
+    struct is_error_code_enum<::net::error>
+      : true_type
+    {};
+}

src/net/fwd.h

@@ -0,0 +1,45 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <cstdint>
+
+namespace net
+{
+    enum class error : int;
+    class tor_address;
+
+    namespace socks
+    {
+        class client;
+        template<typename> class connect_handler;
+        enum class error : int;
+        enum class version : std::uint8_t;
+    }
+}

src/net/parse.cpp

@@ -0,0 +1,60 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "parse.h"
+
+#include "net/tor_address.h"
+#include "string_tools.h"
+
+namespace net
+{
+    expect<epee::net_utils::network_address>
+    get_network_address(const boost::string_ref address, const std::uint16_t default_port)
+    {
+        const boost::string_ref host = address.substr(0, address.rfind(':'));
+
+        if (host.empty())
+            return make_error_code(net::error::invalid_host);
+        if (host.ends_with(".onion"))
+            return tor_address::make(address, default_port);
+        if (host.ends_with(".i2p"))
+            return make_error_code(net::error::invalid_i2p_address); // not yet implemented (prevent public DNS lookup)
+
+        std::uint16_t port = default_port;
+        if (host.size() < address.size())
+        {
+            if (!epee::string_tools::get_xtype_from_string(port, std::string{address.substr(host.size() + 1)}))
+                return make_error_code(net::error::invalid_port);
+        }
+
+        std::uint32_t ip = 0;
+        if (epee::string_tools::get_ip_int32_from_string(ip, std::string{host}))
+            return {epee::net_utils::ipv4_network_address{ip, port}};
+        return make_error_code(net::error::unsupported_address);
+    }
+}

src/net/parse.h

@@ -0,0 +1,54 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/utility/string_ref.hpp>
+#include <cstdint>
+
+#include "common/expect.h"
+#include "net/net_utils_base.h"
+
+namespace net
+{
+    /*!
+      Identifies onion and IPv4 addresses and returns them as a generic
+      `network_address`. If the type is unsupported, it might be a hostname,
+      and `error() == net::error::kUnsupportedAddress` is returned.
+
+      \param address An onion address, ipv4 address or hostname. Hostname
+          will return an error.
+      \param default_port If `address` does not specify a port, this value
+          will be used.
+
+      \return A tor or IPv4 address, else error.
+    */
+    expect<epee::net_utils::network_address>
+        get_network_address(boost::string_ref address, std::uint16_t default_port);
+}
+

src/net/socks.cpp

@@ -0,0 +1,308 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "socks.h"
+
+#include <algorithm>
+#include <boost/asio/buffer.hpp>
+#include <boost/asio/read.hpp>
+#include <boost/asio/write.hpp>
+#include <boost/endian/arithmetic.hpp>
+#include <boost/endian/conversion.hpp>
+#include <cstring>
+#include <limits>
+#include <string>
+
+#include "net/net_utils_base.h"
+#include "net/tor_address.h"
+
+namespace net
+{
+namespace socks
+{
+    namespace
+    {
+        constexpr const unsigned v4_reply_size = 8;
+        constexpr const std::uint8_t v4_connect_command = 1;
+        constexpr const std::uint8_t v4tor_resolve_command = 0xf0;
+        constexpr const std::uint8_t v4_request_granted = 90;
+
+        struct v4_header
+        {
+            std::uint8_t version;
+            std::uint8_t command_code;
+            boost::endian::big_uint16_t port;
+            boost::endian::big_uint32_t ip;
+        };
+
+        std::size_t write_domain_header(epee::span<std::uint8_t> out, const std::uint8_t command, const std::uint16_t port, const boost::string_ref domain)
+        {
+            if (std::numeric_limits<std::size_t>::max() - sizeof(v4_header) - 2 < domain.size())
+                return 0;
+
+            const std::size_t buf_size = sizeof(v4_header) + domain.size() + 2;
+            if (out.size() < buf_size)
+                return 0;
+
+            // version 4, 1 indicates invalid ip for domain extension
+            const v4_header temp{4, command, port, std::uint32_t(1)};
+            std::memcpy(out.data(), std::addressof(temp), sizeof(temp));
+            out.remove_prefix(sizeof(temp));
+
+            *(out.data()) = 0;
+            out.remove_prefix(1);
+
+            std::memcpy(out.data(), domain.data(), domain.size());
+            out.remove_prefix(domain.size());
+
+            *(out.data()) = 0;
+            return buf_size;
+        }
+
+        struct socks_category : boost::system::error_category
+        {
+            explicit socks_category() noexcept
+              : boost::system::error_category()
+            {}
+
+            const char* name() const noexcept override
+            {
+                return "net::socks::error_category";
+            }
+
+            virtual std::string message(int value) const override
+            {
+                switch (socks::error(value))
+                {
+                case socks::error::rejected:
+                    return "Socks request rejected or failed";
+                case socks::error::identd_connection:
+                    return "Socks request rejected because server cannot connect to identd on the client";
+                case socks::error::identd_user:
+                    return "Socks request rejected because the client program and identd report different user-ids";
+
+                case socks::error::bad_read:
+                    return "Socks boost::async_read read fewer bytes than expected";
+                case socks::error::bad_write:
+                    return "Socks boost::async_write wrote fewer bytes than expected";
+                case socks::error::unexpected_version:
+                    return "Socks server returned unexpected version in reply";
+
+                default:
+                    break;
+                }
+                return "Unknown net::socks::error";
+            }
+
+            boost::system::error_condition default_error_condition(int value) const noexcept override
+            {
+                switch (socks::error(value))
+                {
+                case socks::error::bad_read:
+                case socks::error::bad_write:
+                    return boost::system::errc::io_error;
+                case socks::error::unexpected_version:
+                    return boost::system::errc::protocol_error;
+                default:
+                    break;
+                };
+                if (1 <= value && value <= 256)
+                    return boost::system::errc::protocol_error;
+
+                return boost::system::error_condition{value, *this};
+            }
+        };
+    }
+
+    const boost::system::error_category& error_category() noexcept
+    {
+        static const socks_category instance{};
+        return instance;
+    }
+
+    struct client::completed
+    {
+        std::shared_ptr<client> self_;
+
+        void operator()(const boost::system::error_code error, const std::size_t bytes) const
+        {
+            static_assert(1 < sizeof(self_->buffer_), "buffer too small for v4 response");
+
+            if (self_)
+            {
+                client& self = *self_;
+                self.buffer_size_ = std::min(bytes, sizeof(self.buffer_));
+
+                if (error)
+                    self.done(error, std::move(self_));
+                else if (self.buffer().size() < sizeof(v4_header))
+                    self.done(socks::error::bad_read, std::move(self_));
+                else if (self.buffer_[0] != 0) // response version
+                    self.done(socks::error::unexpected_version, std::move(self_));
+                else if (self.buffer_[1] != v4_request_granted)
+                    self.done(socks::error(int(self.buffer_[1]) + 1), std::move(self_));
+                else
+                    self.done(boost::system::error_code{}, std::move(self_));
+            }
+        }
+    };
+
+    struct client::read
+    {
+        std::shared_ptr<client> self_;
+
+        static boost::asio::mutable_buffers_1 get_buffer(client& self) noexcept
+        {
+            static_assert(sizeof(v4_header) <= sizeof(self.buffer_), "buffer too small for v4 response");
+            return boost::asio::buffer(self.buffer_, sizeof(v4_header));
+        }
+
+        void operator()(const boost::system::error_code error, const std::size_t bytes)
+        {
+            if (self_)
+            {
+                client& self = *self_;
+                if (error)
+                    self.done(error, std::move(self_));
+                else if (bytes < self.buffer().size())
+                    self.done(socks::error::bad_write, std::move(self_));
+                else
+                    boost::asio::async_read(self.proxy_, get_buffer(self), completed{std::move(self_)});
+            }
+        }
+    };
+
+    struct client::write
+    {
+        std::shared_ptr<client> self_;
+
+        static boost::asio::const_buffers_1 get_buffer(client const& self) noexcept
+        {
+            return boost::asio::buffer(self.buffer_, self.buffer_size_);
+        }
+
+        void operator()(const boost::system::error_code error)
+        {
+            if (self_)
+            {
+                client& self = *self_;
+                if (error)
+                    self.done(error, std::move(self_));
+                else
+                    boost::asio::async_write(self.proxy_, get_buffer(self), read{std::move(self_)});
+            }
+        }
+    };
+
+    client::client(stream_type::socket&& proxy, socks::version ver)
+      : proxy_(std::move(proxy)), buffer_size_(0), buffer_(), ver_(ver)
+    {}
+
+    client::~client() {}
+
+    bool client::set_connect_command(const epee::net_utils::ipv4_network_address& address)
+    {
+        switch (socks_version())
+        {
+        case version::v4:
+        case version::v4a:
+        case version::v4a_tor:
+            break;
+        default:
+            return false;
+        }
+
+        static_assert(sizeof(v4_header) < sizeof(buffer_), "buffer size too small for request");
+        static_assert(0 < sizeof(buffer_), "buffer size too small for null termination"); 
+
+        // version 4
+        const v4_header temp{4, v4_connect_command, address.port(), boost::endian::big_to_native(address.ip())};
+        std::memcpy(std::addressof(buffer_), std::addressof(temp), sizeof(temp));
+        buffer_[sizeof(temp)] = 0;
+        buffer_size_ = sizeof(temp) + 1;
+
+        return true;
+    }
+
+    bool client::set_connect_command(const boost::string_ref domain, std::uint16_t port)
+    {
+        switch (socks_version())
+        {
+        case version::v4a:
+        case version::v4a_tor:
+            break;
+
+        default:
+            return false;
+        }
+
+        const std::size_t buf_used = write_domain_header(buffer_, v4_connect_command, port, domain);
+        buffer_size_ = buf_used;
+        return buf_used != 0;
+    }
+
+    bool client::set_connect_command(const net::tor_address& address)
+    {
+        if (!address.is_unknown())
+            return set_connect_command(address.host_str(), address.port());
+        return false;
+    }
+
+    bool client::set_resolve_command(boost::string_ref domain)
+    {
+        if (socks_version() != version::v4a_tor)
+            return false;
+
+        const std::size_t buf_used = write_domain_header(buffer_, v4tor_resolve_command, 0, domain);
+        buffer_size_ = buf_used;
+        return buf_used != 0;
+    }
+
+    bool client::connect_and_send(std::shared_ptr<client> self, const stream_type::endpoint& proxy_address)
+    {
+        if (self && !self->buffer().empty())
+        {
+            client& alias = *self;
+            alias.proxy_.async_connect(proxy_address, write{std::move(self)});
+            return true;
+        }
+        return false;
+    }
+
+    bool client::send(std::shared_ptr<client> self)
+    {
+        if (self && !self->buffer().empty())
+        {
+            client& alias = *self;
+            boost::asio::async_write(alias.proxy_, write::get_buffer(alias), read{std::move(self)});
+            return true;
+        }
+        return false;
+    }
+} // socks
+} // net

src/net/socks.h

@@ -0,0 +1,225 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <cstdint>
+#include <boost/asio/ip/tcp.hpp>
+#include <boost/asio/io_service.hpp>
+#include <boost/system/error_code.hpp>
+#include <boost/type_traits/integral_constant.hpp>
+#include <boost/utility/string_ref.hpp>
+#include <memory>
+#include <utility>
+
+#include "net/fwd.h"
+#include "span.h"
+
+namespace epee
+{
+namespace net_utils
+{
+    class ipv4_network_address;
+}
+}
+
+namespace net
+{
+namespace socks
+{
+    //! Supported socks variants.
+    enum class version : std::uint8_t
+    {
+        v4 = 0,
+        v4a,
+        v4a_tor  //!< Extensions defined in Tor codebase
+    };
+
+    //! Possible errors with socks communication. Defined in https://www.openssh.com/txt/socks4.protocol
+    enum class error : int
+    {
+        // 0 is reserved for success value
+        // 1-256 -> reserved for error values from socks server (+1 from wire value).
+        rejected = 92,
+        identd_connection,
+        identd_user,
+        // Specific to application
+        bad_read = 257,
+        bad_write,
+        unexpected_version
+    };
+
+    /* boost::system::error_code is extended for easier compatibility with
+       boost::asio errors. If std::error_code is needed (with expect<T> for
+       instance), then upgrade to boost 1.65+ or use conversion code in
+       develop branch at boost/system/detail/std_interoperability.hpp */
+
+    //! \return boost::system::error_category for net::socks namespace
+    const boost::system::error_category& error_category() noexcept;
+
+    //! \return net::socks::error as a boost::system::error_code.
+    inline boost::system::error_code make_error_code(error value) noexcept
+    {
+        return boost::system::error_code{int(value), socks::error_category()};
+    }
+
+    //! Client support for socks connect and resolve commands.
+    class client
+    {
+        boost::asio::ip::tcp::socket proxy_;
+        std::uint16_t buffer_size_;
+        std::uint8_t buffer_[1024];
+        socks::version ver_;
+
+        /*!
+            Only invoked after `*send(...)` function completes or fails.
+            `bool(error) == false` indicates success; `self.get()` is always
+            `this` and allows implementations to skip
+            `std::enable_shared_from_this<T>` (ASIO callbacks need shared_ptr).
+            The design saves space and reduces cycles (everything uses moves,
+            so no atomic operations are ever necessary).
+
+            \param error when processing last command (if any).
+            \param self `shared_ptr<client>` handle to `this`.
+         */
+        virtual void done(boost::system::error_code error, std::shared_ptr<client> self) = 0;
+
+    public:
+        using stream_type = boost::asio::ip::tcp;
+
+        // defined in cpp
+        struct write;
+        struct read;
+        struct completed;
+
+        /*!
+            \param proxy ownership is passed into `this`. Does not have to be
+                in connected state.
+            \param ver socks version for the connection.
+        */
+        explicit client(stream_type::socket&& proxy, socks::version ver);
+
+        client(const client&) = delete;
+        virtual ~client();
+        client& operator=(const client&) = delete;
+
+        //! \return Ownership of socks client socket object.
+        stream_type::socket take_socket()
+        {
+            return stream_type::socket{std::move(proxy_)};
+        }
+
+        //! \return Socks version.
+        socks::version socks_version() const noexcept { return ver_; }
+
+        //! \return Contents of internal buffer.
+        epee::span<const std::uint8_t> buffer() const noexcept
+        {
+            return {buffer_, buffer_size_};
+        }
+
+        //! \post `buffer.empty()`.
+        void clear_command() noexcept { buffer_size_ = 0; }
+
+        //! Try to set `address` as remote connection request.
+        bool set_connect_command(const epee::net_utils::ipv4_network_address& address);
+
+        //! Try to set `domain` + `port` as remote connection request.
+        bool set_connect_command(boost::string_ref domain, std::uint16_t port);
+
+        //! Try to set `address` as remote Tor hidden service connection request.
+        bool set_connect_command(const net::tor_address& address);
+
+        //! Try to set `domain` as remote DNS A record lookup request.
+        bool set_resolve_command(boost::string_ref domain);
+
+        /*!
+            Asynchronously connect to `proxy_address` then issue command in
+            `buffer()`. The `done(...)` method will be invoked upon completion
+            with `self` and potential `error`s.
+
+            \note Must use one of the `self->set_*_command` calls before using
+                this function.
+
+            \param self ownership of object is given to function.
+            \param proxy_address of the socks server.
+            \return False if `self->buffer().empty()` (no command set).
+         */
+        static bool connect_and_send(std::shared_ptr<client> self, const stream_type::endpoint& proxy_address);
+
+        /*!
+            Assume existing connection to proxy server; asynchronously issue
+            command in `buffer()`. The `done(...)` method will be invoked
+            upon completion with `self` and potential `error`s.
+
+            \note Must use one of the `self->set_*_command` calls before using
+                the function.
+
+            \param self ownership of object is given to function.
+            \return False if `self->buffer().empty()` (no command set).
+        */
+        static bool send(std::shared_ptr<client> self);
+    };
+
+    template<typename Handler>
+    class connect_client : public client
+    {
+        Handler handler_;
+
+        virtual void done(boost::system::error_code error, std::shared_ptr<client>) override
+        {
+            handler_(error, take_socket());
+        }
+
+    public:
+        explicit connect_client(stream_type::socket&& proxy, socks::version ver, Handler&& handler)
+          : client(std::move(proxy), ver), handler_(std::move(handler))
+        {}
+
+        virtual ~connect_client() override {}
+    };
+
+    template<typename Handler>
+    inline std::shared_ptr<client>
+    make_connect_client(client::stream_type::socket&& proxy, socks::version ver, Handler handler)
+    {
+        return std::make_shared<connect_client<Handler>>(std::move(proxy), ver, std::move(handler));
+    }
+} // socks
+} // net
+
+namespace boost
+{
+namespace system
+{
+    template<>
+    struct is_error_code_enum<net::socks::error>
+      : true_type
+    {};
+} // system
+} // boost

src/net/tor_address.cpp

@@ -0,0 +1,203 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "tor_address.h"
+
+#include <algorithm>
+#include <boost/spirit/include/karma_generate.hpp>
+#include <boost/spirit/include/karma_uint.hpp>
+#include <cassert>
+#include <cstring>
+#include <limits>
+
+#include "net/error.h"
+#include "serialization/keyvalue_serialization.h"
+#include "storages/portable_storage.h"
+#include "string_tools.h"
+
+namespace net
+{
+    namespace
+    {
+        constexpr const char tld[] = u8".onion";
+        constexpr const char unknown_host[] = "<unknown tor host>";
+
+        constexpr const unsigned v2_length = 16;
+        constexpr const unsigned v3_length = 56;
+
+        constexpr const char base32_alphabet[] =
+            u8"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz234567";
+
+        expect<void> host_check(boost::string_ref host) noexcept
+        {
+            if (!host.ends_with(tld))
+                return {net::error::expected_tld};
+
+            host.remove_suffix(sizeof(tld) - 1);
+
+            //! \TODO v3 has checksum, base32 decoding is required to verify it
+            if (host.size() != v2_length && host.size() != v3_length)
+                return {net::error::invalid_tor_address};
+            if (host.find_first_not_of(base32_alphabet) != boost::string_ref::npos)
+                return {net::error::invalid_tor_address};
+
+            return success();
+        }
+
+        struct tor_serialized
+        {
+            std::string host;
+            std::uint16_t port;
+
+            BEGIN_KV_SERIALIZE_MAP()
+                KV_SERIALIZE(host)
+                KV_SERIALIZE(port)
+            END_KV_SERIALIZE_MAP()
+        };
+    }
+
+    tor_address::tor_address(const boost::string_ref host, const std::uint16_t port) noexcept
+      : port_(port)
+    {
+        // this is a private constructor, throw if moved to public
+        assert(host.size() < sizeof(host_));
+
+        const std::size_t length = std::min(sizeof(host_) - 1, host.size());
+        std::memcpy(host_, host.data(), length);
+        std::memset(host_ + length, 0, sizeof(host_) - length);
+    }
+
+    const char* tor_address::unknown_str() noexcept
+    {
+        return unknown_host;
+    }
+
+    tor_address::tor_address() noexcept
+      : port_(0)
+    {
+        static_assert(sizeof(unknown_host) <= sizeof(host_), "bad buffer size");
+        std::memcpy(host_, unknown_host, sizeof(unknown_host));
+        std::memset(host_ + sizeof(unknown_host), 0, sizeof(host_) - sizeof(unknown_host));
+    }
+
+    expect<tor_address> tor_address::make(const boost::string_ref address, const std::uint16_t default_port)
+    {
+        boost::string_ref host = address.substr(0, address.rfind(':'));
+        const boost::string_ref port =
+            address.substr(host.size() + (host.size() == address.size() ? 0 : 1));
+
+        MONERO_CHECK(host_check(host));
+
+        std::uint16_t porti = default_port;
+        if (!port.empty() && !epee::string_tools::get_xtype_from_string(porti, std::string{port}))
+            return {net::error::invalid_port};
+
+        static_assert(v2_length <= v3_length, "bad internal host size");
+        static_assert(v3_length + sizeof(tld) == sizeof(tor_address::host_), "bad internal host size");
+        return tor_address{host, porti};
+    }
+
+    bool tor_address::_load(epee::serialization::portable_storage& src, epee::serialization::section* hparent)
+    {
+        tor_serialized in{};
+        if (in._load(src, hparent) && in.host.size() < sizeof(host_) && (in.host == unknown_host || !host_check(in.host).has_error()))
+        {
+            std::memcpy(host_, in.host.data(), in.host.size());
+            std::memset(host_ + in.host.size(), 0, sizeof(host_) - in.host.size());
+            port_ = in.port;
+            return true;
+        }
+        static_assert(sizeof(unknown_host) <= sizeof(host_), "bad buffer size");
+        std::memcpy(host_, unknown_host, sizeof(unknown_host)); // include null terminator
+        port_ = 0;
+        return false;
+    }
+
+    bool tor_address::store(epee::serialization::portable_storage& dest, epee::serialization::section* hparent) const
+    {
+        const tor_serialized out{std::string{host_}, port_};
+        return out.store(dest, hparent);
+    }
+
+    tor_address::tor_address(const tor_address& rhs) noexcept
+      : port_(rhs.port_)
+    {
+        std::memcpy(host_, rhs.host_, sizeof(host_));
+    }
+
+    tor_address& tor_address::operator=(const tor_address& rhs) noexcept
+    {
+        if (this != std::addressof(rhs))
+        {
+            port_ = rhs.port_;
+            std::memcpy(host_, rhs.host_, sizeof(host_));
+        }
+        return *this;
+    }
+
+    bool tor_address::is_unknown() const noexcept
+    {
+        static_assert(1 <= sizeof(host_), "host size too small");
+        return host_[0] == '<'; // character is not allowed otherwise
+    }
+
+    bool tor_address::equal(const tor_address& rhs) const noexcept
+    {
+        return port_ == rhs.port_ && is_same_host(rhs);
+    }
+
+    bool tor_address::less(const tor_address& rhs) const noexcept
+    {
+        return std::strcmp(host_str(), rhs.host_str()) < 0 || port() < rhs.port();
+    }
+
+    bool tor_address::is_same_host(const tor_address& rhs) const noexcept
+    {
+        //! \TODO v2 and v3 should be comparable - requires base32
+        return std::strcmp(host_str(), rhs.host_str()) == 0;
+    }
+
+    std::string tor_address::str() const
+    {
+        const std::size_t host_length = std::strlen(host_str());
+        const std::size_t port_length =
+            port_ == 0 ? 0 : std::numeric_limits<std::uint16_t>::digits10 + 2;
+
+        std::string out{};
+        out.reserve(host_length + port_length);
+        out.assign(host_str(), host_length);
+
+        if (port_ != 0)
+        {
+            out.push_back(':');
+            namespace karma = boost::spirit::karma;
+            karma::generate(std::back_inserter(out), karma::ushort_, port());
+        }
+        return out;
+    }
+}

src/net/tor_address.h

@@ -0,0 +1,140 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/utility/string_ref.hpp>
+#include <cstdint>
+#include <string>
+
+#include "common/expect.h"
+#include "net/enums.h"
+#include "net/error.h"
+
+namespace epee
+{
+namespace serialization
+{
+    class portable_storage;
+    struct section;
+}
+}
+
+namespace net
+{
+    //! Tor onion address; internal format not condensed/decoded.
+    class tor_address
+    {
+        std::uint16_t port_;
+        char host_[63]; // null-terminated
+
+        //! Keep in private, `host.size()` has no runtime check
+        tor_address(boost::string_ref host, std::uint16_t port) noexcept;
+
+    public:
+        //! \return Size of internal buffer for host.
+        static constexpr std::size_t buffer_size() noexcept { return sizeof(host_); }
+
+        //! \return `<unknown tor host>`.
+        static const char* unknown_str() noexcept;
+
+        //! An object with `port() == 0` and `host_str() == unknown_str()`.
+        tor_address() noexcept;
+
+        //! \return A default constructed `tor_address` object.
+        static tor_address unknown() noexcept { return tor_address{}; }
+
+        /*!
+            Parse `address` in onion v2 or v3 format with (i.e. x.onion:80)
+            with `default_port` being used iff port is not specified in
+            `address`.
+        */
+        static expect<tor_address> make(boost::string_ref address, std::uint16_t default_port = 0);
+
+        //! Load from epee p2p format, and \return false if not valid tor address
+        bool _load(epee::serialization::portable_storage& src, epee::serialization::section* hparent);
+
+        //! Store in epee p2p format
+        bool store(epee::serialization::portable_storage& dest, epee::serialization::section* hparent) const;
+
+        // Moves and  copies are currently identical
+
+        tor_address(const tor_address& rhs) noexcept;
+        ~tor_address() = default;
+        tor_address& operator=(const tor_address& rhs) noexcept;
+
+        //! \return True if default constructed or via `unknown()`.
+        bool is_unknown() const noexcept;
+
+        bool equal(const tor_address& rhs) const noexcept;
+        bool less(const tor_address& rhs) const noexcept;
+
+        //! \return True if onion addresses are identical.
+        bool is_same_host(const tor_address& rhs) const noexcept;
+
+        //! \return `x.onion` or `x.onion:z` if `port() != 0`.
+        std::string str() const;
+
+        //! \return Null-terminated `x.onion` value or `unknown_str()`.
+        const char* host_str() const noexcept { return host_; }
+
+        //! \return Port value or `0` if unspecified.
+        std::uint16_t port() const noexcept { return port_; }
+
+        static constexpr bool is_loopback() noexcept { return false; }
+        static constexpr bool is_local() noexcept { return false; }
+
+        static constexpr epee::net_utils::address_type get_type_id() noexcept
+        {
+            return epee::net_utils::address_type::tor;
+        }
+
+        static constexpr epee::net_utils::zone get_zone() noexcept
+        {
+            return epee::net_utils::zone::tor;
+        }
+
+        //! \return `!is_unknown()`.
+        bool is_blockable() const noexcept { return !is_unknown(); }
+    };
+
+    inline bool operator==(const tor_address& lhs, const tor_address& rhs) noexcept
+    {
+        return lhs.equal(rhs);
+    }
+
+    inline bool operator!=(const tor_address& lhs, const tor_address& rhs) noexcept
+    {
+        return !lhs.equal(rhs);
+    }
+
+    inline bool operator<(const tor_address& lhs, const tor_address& rhs) noexcept
+    {
+        return lhs.less(rhs);
+    }
+} // net

src/p2p/CMakeLists.txt

@@ -40,6 +40,7 @@ target_link_libraries(p2p
   PUBLIC
     version
     cryptonote_core
+    net
     ${UPNP_LIBRARIES}
     ${Boost_CHRONO_LIBRARY}
     ${Boost_PROGRAM_OPTIONS_LIBRARY}

src/p2p/net_node.cpp

@@ -28,9 +28,79 @@
 //
 // Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
 
+#include <boost/algorithm/string/find_iterator.hpp>
+#include <boost/algorithm/string/finder.hpp>
+#include <boost/chrono/duration.hpp>
+#include <boost/endian/conversion.hpp>
+#include <boost/optional/optional.hpp>
+#include <boost/thread/future.hpp>
+#include <boost/utility/string_ref.hpp>
+#include <chrono>
+#include <utility>
+
 #include "common/command_line.h"
 #include "cryptonote_core/cryptonote_core.h"
+#include "cryptonote_protocol/cryptonote_protocol_defs.h"
 #include "net_node.h"
+#include "net/net_utils_base.h"
+#include "net/socks.h"
+#include "net/parse.h"
+#include "net/tor_address.h"
+#include "p2p/p2p_protocol_defs.h"
+#include "string_tools.h"
+
+namespace
+{
+    constexpr const boost::chrono::milliseconds future_poll_interval{500};
+    constexpr const std::chrono::seconds tor_connect_timeout{P2P_DEFAULT_TOR_CONNECT_TIMEOUT};
+
+    std::int64_t get_max_connections(const boost::iterator_range<boost::string_ref::const_iterator> value) noexcept
+    {
+        // -1 is default, 0 is error
+        if (value.empty())
+            return -1;
+
+        std::uint32_t out = 0;
+        if (epee::string_tools::get_xtype_from_string(out, std::string{value.begin(), value.end()}))
+            return out;
+        return 0;
+    }
+
+    template<typename T>
+    epee::net_utils::network_address get_address(const boost::string_ref value)
+    {
+        expect<T> address = T::make(value);
+        if (!address)
+        {
+            MERROR(
+                "Failed to parse " << epee::net_utils::zone_to_string(T::get_zone()) << " address \"" << value << "\": " << address.error().message()
+            );
+            return {};
+        }
+        return {std::move(*address)};
+    }
+
+    bool start_socks(std::shared_ptr<net::socks::client> client, const boost::asio::ip::tcp::endpoint& proxy, const epee::net_utils::network_address& remote)
+    {
+        CHECK_AND_ASSERT_MES(client != nullptr, false, "Unexpected null client");
+
+        bool set = false;
+        switch (remote.get_type_id())
+        {
+        case net::tor_address::get_type_id():
+            set = client->set_connect_command(remote.as<net::tor_address>());
+            break;
+        default:
+            MERROR("Unsupported network address in socks_connect");
+            return false;
+        }
+
+        const bool sent =
+            set && net::socks::client::connect_and_send(std::move(client), proxy);
+        CHECK_AND_ASSERT_MES(sent, false, "Unexpected failure to init socks client");
+        return true;
+    }
+}
 
 namespace nodetool
 {
@@ -55,6 +125,8 @@ namespace nodetool
     const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_add_exclusive_node   = {"add-exclusive-node", "Specify list of peers to connect to only."
                                                                                                   " If this option is given the options add-priority-node and seed-node are ignored"};
     const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_seed_node   = {"seed-node", "Connect to a node to retrieve peer addresses, and disconnect"};
+    const command_line::arg_descriptor<std::vector<std::string> > arg_proxy = {"proxy", "<network-type>,<socks-ip:port>[,max_connections] i.e. \"tor,127.0.0.1:9050,100\""};
+    const command_line::arg_descriptor<std::vector<std::string> > arg_anonymous_inbound = {"anonymous-inbound", "<hidden-service-address>,<[bind-ip:]port>[,max_connections] i.e. \"x.onion,127.0.0.1:18083,100\""};
     const command_line::arg_descriptor<bool> arg_p2p_hide_my_port   =    {"hide-my-port", "Do not announce yourself as peerlist candidate", false, true};
 
     const command_line::arg_descriptor<bool>        arg_no_igd  = {"no-igd", "Disable UPnP port mapping"};
@@ -67,4 +139,196 @@ namespace nodetool
     const command_line::arg_descriptor<int64_t> arg_limit_rate = {"limit-rate", "set limit-rate [kB/s]", -1};
 
     const command_line::arg_descriptor<bool> arg_save_graph = {"save-graph", "Save data for dr monero", false};
+
+    boost::optional<std::vector<proxy>> get_proxies(boost::program_options::variables_map const& vm)
+    {
+	namespace ip = boost::asio::ip;
+
+        std::vector<proxy> proxies{};
+
+        const std::vector<std::string> args = command_line::get_arg(vm, arg_proxy);
+        proxies.reserve(args.size());
+
+        for (const boost::string_ref arg : args)
+        {
+            proxies.emplace_back();
+
+            auto next = boost::algorithm::make_split_iterator(arg, boost::algorithm::first_finder(","));
+            CHECK_AND_ASSERT_MES(!next.eof() && !next->empty(), boost::none, "No network type for --" << arg_proxy.name);
+            const boost::string_ref zone{next->begin(), next->size()};
+
+            ++next;
+            CHECK_AND_ASSERT_MES(!next.eof() && !next->empty(), boost::none, "No ipv4:port given for --" << arg_proxy.name);
+            const boost::string_ref proxy{next->begin(), next->size()};
+
+            ++next;
+            if (!next.eof())
+            {
+                proxies.back().max_connections = get_max_connections(*next);
+                if (proxies.back().max_connections == 0)
+                {
+                    MERROR("Invalid max connections given to --" << arg_proxy.name);
+                    return boost::none;
+                }
+            }
+
+            switch (epee::net_utils::zone_from_string(zone))
+            {
+            case epee::net_utils::zone::tor:
+                proxies.back().zone = epee::net_utils::zone::tor;
+                break;
+            default:
+                MERROR("Invalid network for --" << arg_proxy.name);
+                return boost::none;
+            }
+
+            std::uint32_t ip = 0;
+            std::uint16_t port = 0;
+            if (!epee::string_tools::parse_peer_from_string(ip, port, std::string{proxy}) || port == 0)
+            {
+                MERROR("Invalid ipv4:port given for --" << arg_proxy.name);
+                return boost::none;
+            }
+            proxies.back().address = ip::tcp::endpoint{ip::address_v4{boost::endian::native_to_big(ip)}, port};
+	}
+
+        return proxies;
+    }
+
+    boost::optional<std::vector<anonymous_inbound>> get_anonymous_inbounds(boost::program_options::variables_map const& vm)
+    {
+        std::vector<anonymous_inbound> inbounds{};
+
+        const std::vector<std::string> args = command_line::get_arg(vm, arg_anonymous_inbound);
+        inbounds.reserve(args.size());
+
+        for (const boost::string_ref arg : args)
+        {
+            inbounds.emplace_back();
+
+            auto next = boost::algorithm::make_split_iterator(arg, boost::algorithm::first_finder(","));
+            CHECK_AND_ASSERT_MES(!next.eof() && !next->empty(), boost::none, "No inbound address for --" << arg_anonymous_inbound.name);
+            const boost::string_ref address{next->begin(), next->size()};
+
+            ++next;
+            CHECK_AND_ASSERT_MES(!next.eof() && !next->empty(), boost::none, "No local ipv4:port given for --" << arg_anonymous_inbound.name);
+            const boost::string_ref bind{next->begin(), next->size()};
+
+            const std::size_t colon = bind.find_first_of(':');
+            CHECK_AND_ASSERT_MES(colon < bind.size(), boost::none, "No local port given for --" << arg_anonymous_inbound.name);
+
+            ++next;
+            if (!next.eof())
+            {
+                inbounds.back().max_connections = get_max_connections(*next);
+                if (inbounds.back().max_connections == 0)
+                {
+                    MERROR("Invalid max connections given to --" << arg_proxy.name);
+                    return boost::none;
+                }
+            }
+
+            expect<epee::net_utils::network_address> our_address = net::get_network_address(address, 0);
+            switch (our_address ? our_address->get_type_id() : epee::net_utils::address_type::invalid)
+            {
+            case net::tor_address::get_type_id():
+                inbounds.back().our_address = std::move(*our_address);
+                inbounds.back().default_remote = net::tor_address::unknown();
+                break;
+            default:
+                MERROR("Invalid inbound address (" << address << ") for --" << arg_anonymous_inbound.name << ": " << (our_address ? "invalid type" : our_address.error().message()));
+                return boost::none;
+            }
+
+            // get_address returns default constructed address on error
+            if (inbounds.back().our_address == epee::net_utils::network_address{})
+                return boost::none;
+
+            std::uint32_t ip = 0;
+            std::uint16_t port = 0;
+            if (!epee::string_tools::parse_peer_from_string(ip, port, std::string{bind}))
+            {
+                MERROR("Invalid ipv4:port given for --" << arg_anonymous_inbound.name);
+                return boost::none;
+            }
+            inbounds.back().local_ip = std::string{bind.substr(0, colon)};
+            inbounds.back().local_port = std::string{bind.substr(colon + 1)};
+        }
+
+        return inbounds;
+    }
+
+    bool is_filtered_command(const epee::net_utils::network_address& address, int command)
+    {
+        switch (command)
+        {
+        case nodetool::COMMAND_HANDSHAKE_T<cryptonote::CORE_SYNC_DATA>::ID:
+        case nodetool::COMMAND_TIMED_SYNC_T<cryptonote::CORE_SYNC_DATA>::ID:
+        case cryptonote::NOTIFY_NEW_TRANSACTIONS::ID:
+            return false;
+        default:
+            break;
+        }
+
+        if (address.get_zone() == epee::net_utils::zone::public_)
+            return false;
+
+        MWARNING("Filtered command (#" << command << ") to/from " << address.str());
+        return true;
+    }
+
+    boost::optional<boost::asio::ip::tcp::socket>
+    socks_connect_internal(const std::atomic<bool>& stop_signal, boost::asio::io_service& service, const boost::asio::ip::tcp::endpoint& proxy, const epee::net_utils::network_address& remote)
+    {
+        using socket_type = net::socks::client::stream_type::socket;
+        using client_result = std::pair<boost::system::error_code, socket_type>;
+
+        struct notify
+        {
+            boost::promise<client_result> socks_promise;
+
+            void operator()(boost::system::error_code error, socket_type&& sock)
+            {
+                socks_promise.set_value(std::make_pair(error, std::move(sock)));
+            }
+        };
+
+        boost::unique_future<client_result> socks_result{};
+        {
+            boost::promise<client_result> socks_promise{};
+            socks_result = socks_promise.get_future();
+
+            auto client = net::socks::make_connect_client(
+                boost::asio::ip::tcp::socket{service}, net::socks::version::v4a, notify{std::move(socks_promise)}
+             );
+            if (!start_socks(std::move(client), proxy, remote))
+                return boost::none;
+        }
+
+        const auto start = std::chrono::steady_clock::now();
+        while (socks_result.wait_for(future_poll_interval) == boost::future_status::timeout)
+        {
+            if (tor_connect_timeout < std::chrono::steady_clock::now() - start)
+            {
+                MERROR("Timeout on socks connect (" << proxy << " to " << remote.str() << ")");
+                return boost::none;
+            }
+
+            if (stop_signal)
+                return boost::none;
+        }
+
+        try
+        {
+            auto result = socks_result.get();
+            if (!result.first)
+                return {std::move(result.second)};
+
+            MERROR("Failed to make socks connection to " << remote.str() << " (via " << proxy << "): " << result.first.message());
+        }
+        catch (boost::broken_promise const&)
+        {}
+
+        return boost::none;
+    }
 }

src/p2p/net_node.h

@@ -30,12 +30,17 @@
 
 #pragma once
 #include <array>
+#include <atomic>
+#include <boost/asio/io_service.hpp>
+#include <boost/asio/ip/tcp.hpp>
 #include <boost/thread.hpp>
+#include <boost/optional/optional_fwd.hpp>
 #include <boost/program_options/options_description.hpp>
 #include <boost/program_options/variables_map.hpp>
-#include <boost/serialization/version.hpp>
 #include <boost/uuid/uuid.hpp>
-#include <boost/serialization/map.hpp>
+#include <functional>
+#include <utility>
+#include <vector>
 
 #include "cryptonote_config.h"
 #include "warnings.h"
@@ -47,6 +52,8 @@
 #include "net_peerlist.h"
 #include "math_helper.h"
 #include "net_node_common.h"
+#include "net/enums.h"
+#include "net/fwd.h"
 #include "common/command_line.h"
 
 PUSH_WARNINGS
@@ -54,6 +61,47 @@ DISABLE_VS_WARNINGS(4355)
 
 namespace nodetool
 {
+  struct proxy
+  {
+    proxy()
+      : max_connections(-1),
+        address(),
+        zone(epee::net_utils::zone::invalid)
+    {}
+
+    std::int64_t max_connections;
+    boost::asio::ip::tcp::endpoint address;
+    epee::net_utils::zone zone;
+  };
+
+  struct anonymous_inbound
+  {
+    anonymous_inbound()
+      : max_connections(-1),
+        local_ip(),
+        local_port(),
+        our_address(),
+        default_remote()
+    {}
+
+    std::int64_t max_connections;
+    std::string local_ip;
+    std::string local_port;
+    epee::net_utils::network_address our_address;
+    epee::net_utils::network_address default_remote;
+  };
+
+  boost::optional<std::vector<proxy>> get_proxies(const boost::program_options::variables_map& vm);
+  boost::optional<std::vector<anonymous_inbound>> get_anonymous_inbounds(const boost::program_options::variables_map& vm);
+
+  //! \return True if `commnd` is filtered (ignored/dropped) for `address`
+  bool is_filtered_command(epee::net_utils::network_address const& address, int command);
+
+  // hides boost::future and chrono stuff from mondo template file
+  boost::optional<boost::asio::ip::tcp::socket>
+  socks_connect_internal(const std::atomic<bool>& stop_signal, boost::asio::io_service& service, const boost::asio::ip::tcp::endpoint& proxy, const epee::net_utils::network_address& remote);
+
+
   template<class base_type>
   struct p2p_connection_context_t: base_type //t_payload_net_handler::connection_context //public net_utils::connection_context_base
   {
@@ -78,53 +126,124 @@ namespace nodetool
     typedef COMMAND_HANDSHAKE_T<typename t_payload_net_handler::payload_type> COMMAND_HANDSHAKE;
     typedef COMMAND_TIMED_SYNC_T<typename t_payload_net_handler::payload_type> COMMAND_TIMED_SYNC;
 
+    typedef epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<p2p_connection_context>> net_server;
+
+    struct network_zone;
+    using connect_func = boost::optional<p2p_connection_context>(network_zone&, epee::net_utils::network_address const&);
+
+    struct config
+    {
+      config()
+        : m_net_config(),
+          m_peer_id(crypto::rand<uint64_t>()),
+          m_support_flags(0)
+      {}
+
+      network_config m_net_config;
+      uint64_t m_peer_id;
+      uint32_t m_support_flags;
+    };
+
+    struct network_zone
+    {
+      network_zone()
+        : m_connect(nullptr),
+          m_net_server(epee::net_utils::e_connection_type_P2P),
+          m_bind_ip(),
+          m_port(),
+          m_our_address(),
+          m_peerlist(),
+          m_config{},
+          m_proxy_address(),
+          m_current_number_of_out_peers(0),
+          m_current_number_of_in_peers(0),
+          m_can_pingback(false)
+      {
+        set_config_defaults();
+      }
+
+      network_zone(boost::asio::io_service& public_service)
+        : m_connect(nullptr),
+          m_net_server(public_service, epee::net_utils::e_connection_type_P2P),
+          m_bind_ip(),
+          m_port(),
+          m_our_address(),
+          m_peerlist(),
+          m_config{},
+          m_proxy_address(),
+          m_current_number_of_out_peers(0),
+          m_current_number_of_in_peers(0),
+          m_can_pingback(false)
+      {
+        set_config_defaults();
+      }
+
+      connect_func* m_connect;
+      net_server m_net_server;
+      std::string m_bind_ip;
+      std::string m_port;
+      epee::net_utils::network_address m_our_address; // in anonymity networks
+      peerlist_manager m_peerlist;
+      config m_config;
+      boost::asio::ip::tcp::endpoint m_proxy_address;
+      std::atomic<unsigned int> m_current_number_of_out_peers;
+      std::atomic<unsigned int> m_current_number_of_in_peers;
+      bool m_can_pingback;
+
+    private:
+      void set_config_defaults() noexcept
+      {
+        // at this moment we have a hardcoded config
+        m_config.m_net_config.handshake_interval = P2P_DEFAULT_HANDSHAKE_INTERVAL;
+        m_config.m_net_config.packet_max_size = P2P_DEFAULT_PACKET_MAX_SIZE;
+        m_config.m_net_config.config_id = 0;
+        m_config.m_net_config.connection_timeout = P2P_DEFAULT_CONNECTION_TIMEOUT;
+        m_config.m_net_config.ping_connection_timeout = P2P_DEFAULT_PING_CONNECTION_TIMEOUT;
+        m_config.m_net_config.send_peerlist_sz = P2P_DEFAULT_PEERS_IN_HANDSHAKE;
+        m_config.m_support_flags = 0; // only set in public zone
+      }
+    };
+
   public:
     typedef t_payload_net_handler payload_net_handler;
 
     node_server(t_payload_net_handler& payload_handler)
       : m_payload_handler(payload_handler),
-    m_current_number_of_out_peers(0),
+        m_external_port(0),
-    m_current_number_of_in_peers(0),
         m_allow_local_ip(false),
         m_hide_my_port(false),
         m_no_igd(false),
         m_offline(false),
         m_save_graph(false),
         is_closing(false),
-    m_net_server( epee::net_utils::e_connection_type_P2P ) // this is a P2P connection of the main p2p node server, because this is class node_server<>
+        m_network_id()
-    {}
-    virtual ~node_server()
     {}
+    virtual ~node_server();
 
     static void init_options(boost::program_options::options_description& desc);
 
     bool run();
+    network_zone& add_zone(epee::net_utils::zone zone);
     bool init(const boost::program_options::variables_map& vm);
     bool deinit();
     bool send_stop_signal();
     uint32_t get_this_peer_port(){return m_listening_port;}
     t_payload_net_handler& get_payload_object();
 
-    template <class Archive, class t_version_type>
-    void serialize(Archive &a,  const t_version_type ver)
-    {
-      a & m_peerlist;
-      if (ver == 0)
-      {
-        // from v1, we do not store the peer id anymore
-        peerid_type peer_id = AUTO_VAL_INIT (peer_id);
-        a & peer_id;
-      }
-    }
     // debug functions
     bool log_peerlist();
     bool log_connections();
-    virtual uint64_t get_connections_count();
+
-    size_t get_outgoing_connections_count();
+    // These functions only return information for the "public" zone
-    size_t get_incoming_connections_count();
+    virtual uint64_t get_public_connections_count();
-    peerlist_manager& get_peerlist_manager(){return m_peerlist;}
+    size_t get_public_outgoing_connections_count();
-    void delete_out_connections(size_t count);
+    size_t get_public_white_peers_count();
-    void delete_in_connections(size_t count);
+    size_t get_public_gray_peers_count();
+    void get_public_peerlist(std::vector<peerlist_entry>& gray, std::vector<peerlist_entry>& white);
+    size_t get_zone_count() const { return m_network_zones.size(); }
+
+    void change_max_out_public_peers(size_t count);
+    void change_max_in_public_peers(size_t count);
     virtual bool block_host(const epee::net_utils::network_address &adress, time_t seconds = P2P_IP_BLOCKTIME);
     virtual bool unblock_host(const epee::net_utils::network_address &address);
     virtual std::map<std::string, time_t> get_blocked_hosts() { CRITICAL_REGION_LOCAL(m_blocked_hosts_lock); return m_blocked_hosts; }
@@ -150,6 +269,9 @@ namespace nodetool
     CHAIN_LEVIN_NOTIFY_MAP2(p2p_connection_context); //move levin_commands_handler interface notify(...) callbacks into nothing
 
     BEGIN_INVOKE_MAP2(node_server)
+      if (is_filtered_command(context.m_remote_address, command))
+        return LEVIN_ERROR_CONNECTION_HANDLER_NOT_DEFINED;
+
       HANDLE_INVOKE_T2(COMMAND_HANDSHAKE, &node_server::handle_handshake)
       HANDLE_INVOKE_T2(COMMAND_TIMED_SYNC, &node_server::handle_timed_sync)
       HANDLE_INVOKE_T2(COMMAND_PING, &node_server::handle_ping)
@@ -178,7 +300,7 @@ namespace nodetool
     bool make_default_peer_id();
     bool make_default_config();
     bool store_config();
-    bool check_trust(const proof_of_trust& tr);
+    bool check_trust(const proof_of_trust& tr, epee::net_utils::zone zone_type);
 
 
     //----------------- levin_commands_handler -------------------------------------------------------------
@@ -186,8 +308,7 @@ namespace nodetool
     virtual void on_connection_close(p2p_connection_context& context);
     virtual void callback(p2p_connection_context& context);
     //----------------- i_p2p_endpoint -------------------------------------------------------------
-    virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid> &connections);
+    virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections);
-    virtual bool relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context);
     virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context);
     virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context);
     virtual bool drop_connection(const epee::net_utils::connection_context_base& context);
@@ -204,7 +325,7 @@ namespace nodetool
       );
     bool idle_worker();
     bool handle_remote_peerlist(const std::vector<peerlist_entry>& peerlist, time_t local_time, const epee::net_utils::connection_context_base& context);
-    bool get_local_node_data(basic_node_data& node_data);
+    bool get_local_node_data(basic_node_data& node_data, const network_zone& zone);
     //bool get_local_handshake_data(handshake_data& hshd);
 
     bool merge_peerlist_with_local(const std::vector<peerlist_entry>& bs);
@@ -216,7 +337,7 @@ namespace nodetool
     bool do_peer_timed_sync(const epee::net_utils::connection_context_base& context, peerid_type peer_id);
 
     bool make_new_connection_from_anchor_peerlist(const std::vector<anchor_peerlist_entry>& anchor_peerlist);
-    bool make_new_connection_from_peerlist(bool use_white_list);
+    bool make_new_connection_from_peerlist(network_zone& zone, bool use_white_list);
     bool try_to_connect_and_handshake_with_new_peer(const epee::net_utils::network_address& na, bool just_take_peerlist = false, uint64_t last_seen_stamp = 0, PeerType peer_type = white, uint64_t first_seen_stamp = 0);
     size_t get_random_index_with_fixed_probability(size_t max_index);
     bool is_peer_used(const peerlist_entry& peer);
@@ -227,7 +348,7 @@ namespace nodetool
     template<class t_callback>
     bool try_ping(basic_node_data& node_data, p2p_connection_context& context, const t_callback &cb);
     bool try_get_support_flags(const p2p_connection_context& context, std::function<void(p2p_connection_context&, const uint32_t&)> f);
-    bool make_expected_connections_count(PeerType peer_type, size_t expected_connections);
+    bool make_expected_connections_count(network_zone& zone, PeerType peer_type, size_t expected_connections);
     void cache_connect_fail_info(const epee::net_utils::network_address& addr);
     bool is_addr_recently_failed(const epee::net_utils::network_address& addr);
     bool is_priority_node(const epee::net_utils::network_address& na);
@@ -240,14 +361,8 @@ namespace nodetool
     template <class Container>
     bool parse_peers_and_add_to_container(const boost::program_options::variables_map& vm, const command_line::arg_descriptor<std::vector<std::string> > & arg, Container& container);
 
-    bool set_max_out_peers(const boost::program_options::variables_map& vm, int64_t max);
+    bool set_max_out_peers(network_zone& zone, int64_t max);
-    bool get_max_out_peers() const { return m_config.m_net_config.max_out_connection_count; }
+    bool set_max_in_peers(network_zone& zone, int64_t max);
-    bool get_current_out_peers() const { return m_current_number_of_out_peers; }
-
-    bool set_max_in_peers(const boost::program_options::variables_map& vm, int64_t max);
-    bool get_max_in_peers() const { return m_config.m_net_config.max_in_connection_count; }
-    bool get_current_in_peers() const { return m_current_number_of_in_peers; }
-
     bool set_tos_flag(const boost::program_options::variables_map& vm, int limit);
 
     bool set_rate_up_limit(const boost::program_options::variables_map& vm, int64_t limit);
@@ -255,6 +370,11 @@ namespace nodetool
     bool set_rate_limit(const boost::program_options::variables_map& vm, int64_t limit);
 
     bool has_too_many_connections(const epee::net_utils::network_address &address);
+    uint64_t get_connections_count();
+    size_t get_incoming_connections_count();
+    size_t get_incoming_connections_count(network_zone&);
+    size_t get_outgoing_connections_count();
+    size_t get_outgoing_connections_count(network_zone&);
 
     bool check_connection_and_handshake_with_peer(const epee::net_utils::network_address& na, uint64_t last_seen_stamp);
     bool gray_peerlist_housekeeping();
@@ -272,25 +392,7 @@ namespace nodetool
     std::string print_connections_container();
 
 
-    typedef epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<p2p_connection_context> > net_server;
-
-    struct config
-    {
-      network_config m_net_config;
-      uint64_t m_peer_id;
-      uint32_t m_support_flags;
-
-      BEGIN_KV_SERIALIZE_MAP()
-        KV_SERIALIZE(m_net_config)
-        KV_SERIALIZE(m_peer_id)
-        KV_SERIALIZE(m_support_flags)
-      END_KV_SERIALIZE_MAP()
-    };
-
   public:
-    config m_config; // TODO was private, add getters?
-    std::atomic<unsigned int> m_current_number_of_out_peers;
-    std::atomic<unsigned int> m_current_number_of_in_peers;
 
     void set_save_graph(bool save_graph)
     {
@@ -304,7 +406,6 @@ namespace nodetool
     bool m_first_connection_maker_call;
     uint32_t m_listening_port;
     uint32_t m_external_port;
-    uint32_t m_ip_address;
     bool m_allow_local_ip;
     bool m_hide_my_port;
     bool m_no_igd;
@@ -316,7 +417,7 @@ namespace nodetool
     //connections_indexed_container m_connections;
 
     t_payload_net_handler& m_payload_handler;
-    peerlist_manager m_peerlist;
+    peerlist_storage m_peerlist_storage;
 
     epee::math_helper::once_a_time_seconds<P2P_DEFAULT_HANDSHAKE_INTERVAL> m_peer_handshake_idle_maker_interval;
     epee::math_helper::once_a_time_seconds<1> m_connections_maker_interval;
@@ -324,8 +425,6 @@ namespace nodetool
     epee::math_helper::once_a_time_seconds<60> m_gray_peerlist_housekeeping_interval;
     epee::math_helper::once_a_time_seconds<3600, false> m_incoming_connections_interval;
 
-    std::string m_bind_ip;
-    std::string m_port;
 #ifdef ALLOW_DEBUG_COMMANDS
     uint64_t m_last_stat_request_time;
 #endif
@@ -333,11 +432,22 @@ namespace nodetool
     std::vector<epee::net_utils::network_address> m_exclusive_peers;
     std::vector<epee::net_utils::network_address> m_seed_nodes;
     bool m_fallback_seed_nodes_added;
-    std::list<nodetool::peerlist_entry> m_command_line_peers;
+    std::vector<nodetool::peerlist_entry> m_command_line_peers;
     uint64_t m_peer_livetime;
     //keep connections to initiate some interactions
-    net_server m_net_server;
+
-    boost::uuids::uuid m_network_id;
+
+    static boost::optional<p2p_connection_context> public_connect(network_zone&, epee::net_utils::network_address const&);
+    static boost::optional<p2p_connection_context> socks_connect(network_zone&, epee::net_utils::network_address const&);
+
+
+    /* A `std::map` provides constant iterators and key/value pointers even with
+    inserts/erases to _other_ elements. This makes the configuration step easier
+    since references can safely be stored on the stack. Do not insert/erase
+    after configuration and before destruction, lock safety would need to be
+    added. `std::map::operator[]` WILL insert! */
+    std::map<epee::net_utils::zone, network_zone> m_network_zones;
+
 
     std::map<epee::net_utils::network_address, time_t> m_conn_fails_cache;
     epee::critical_section m_conn_fails_cache_lock;
@@ -351,6 +461,7 @@ namespace nodetool
     boost::mutex m_used_stripe_peers_mutex;
     std::array<std::list<epee::net_utils::network_address>, 1 << CRYPTONOTE_PRUNING_LOG_STRIPES> m_used_stripe_peers;
 
+    boost::uuids::uuid m_network_id;
     cryptonote::network_type m_nettype;
   };
 
@@ -364,6 +475,8 @@ namespace nodetool
     extern const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_add_priority_node;
     extern const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_add_exclusive_node;
     extern const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_seed_node;
+    extern const command_line::arg_descriptor<std::vector<std::string> > arg_proxy;
+    extern const command_line::arg_descriptor<std::vector<std::string> > arg_anonymous_inbound;
     extern const command_line::arg_descriptor<bool> arg_p2p_hide_my_port;
 
     extern const command_line::arg_descriptor<bool>        arg_no_igd;
@@ -380,3 +493,4 @@ namespace nodetool
 }
 
 POP_WARNINGS
+

src/p2p/net_node_common.h

@@ -31,6 +31,8 @@
 #pragma once
 
 #include <boost/uuid/uuid.hpp>
+#include <utility>
+#include <vector>
 #include "net/net_utils_base.h"
 #include "p2p_protocol_defs.h"
 
@@ -43,13 +45,13 @@ namespace nodetool
   template<class t_connection_context>
   struct i_p2p_endpoint
   {
-    virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid>& connections)=0;
+    virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections)=0;
-    virtual bool relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context)=0;
     virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context)=0;
     virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context)=0;
     virtual bool drop_connection(const epee::net_utils::connection_context_base& context)=0;
     virtual void request_callback(const epee::net_utils::connection_context_base& context)=0;
-    virtual uint64_t get_connections_count()=0;
+    virtual uint64_t get_public_connections_count()=0;
+    virtual size_t get_zone_count() const=0;
     virtual void for_each_connection(std::function<bool(t_connection_context&, peerid_type, uint32_t)> f)=0;
     virtual bool for_connection(const boost::uuids::uuid&, std::function<bool(t_connection_context&, peerid_type, uint32_t)> f)=0;
     virtual bool block_host(const epee::net_utils::network_address &address, time_t seconds = 0)=0;
@@ -64,11 +66,7 @@ namespace nodetool
   template<class t_connection_context>
   struct p2p_endpoint_stub: public i_p2p_endpoint<t_connection_context>
   {
-    virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid>& connections)
+    virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections)
-    {
-      return false;
-    }
-    virtual bool relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context)
     {
       return false;
     }
@@ -97,7 +95,12 @@ namespace nodetool
       return false;
     }
 
-    virtual uint64_t get_connections_count()    
+    virtual size_t get_zone_count() const
+    {
+      return 0;
+    }
+
+    virtual uint64_t get_public_connections_count()    
     {
       return false;
     }

src/p2p/net_peerlist.cpp

@@ -0,0 +1,295 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "net_peerlist.h"
+
+#include <algorithm>
+#include <functional>
+#include <fstream>
+#include <iterator>
+
+#include <boost/archive/binary_iarchive.hpp>
+#include <boost/archive/portable_binary_oarchive.hpp>
+#include <boost/archive/portable_binary_iarchive.hpp>
+#include <boost/filesystem/operations.hpp>
+#include <boost/range/join.hpp>
+#include <boost/serialization/version.hpp>
+
+#include "net_peerlist_boost_serialization.h"
+
+
+namespace nodetool
+{
+  namespace
+  {
+    constexpr unsigned CURRENT_PEERLIST_STORAGE_ARCHIVE_VER = 6;
+ 
+    struct by_zone
+    {
+      using zone = epee::net_utils::zone;
+
+      template<typename T>
+      bool operator()(const T& left, const zone right) const
+      {
+        return left.adr.get_zone() < right;
+      }
+
+      template<typename T>
+      bool operator()(const zone left, const T& right) const
+      {
+        return left < right.adr.get_zone();
+      }
+
+      template<typename T, typename U>
+      bool operator()(const T& left, const U& right) const
+      {
+        return left.adr.get_zone() < right.adr.get_zone();
+      }
+    };
+
+    template<typename Elem, typename Archive>
+    std::vector<Elem> load_peers(Archive& a, unsigned ver)
+    {
+      // at v6, we drop existing peerlists, because annoying change
+      if (ver < 6)
+        return {};
+
+      uint64_t size = 0;
+      a & size;
+      
+      Elem ple{};
+
+      std::vector<Elem> elems{};
+      elems.reserve(size);
+      while (size--)
+      {
+        a & ple;
+        elems.push_back(std::move(ple));
+      }
+
+      return elems;
+    }
+
+    template<typename Archive, typename Range>
+    void save_peers(Archive& a, const Range& elems)
+    {
+      const uint64_t size = elems.size();
+      a & size;
+      for (const auto& elem : elems)
+        a & elem;
+    }
+ 
+    template<typename T>
+    std::vector<T> do_take_zone(std::vector<T>& src, epee::net_utils::zone zone)
+    {
+      const auto start = std::lower_bound(src.begin(), src.end(), zone, by_zone{});
+      const auto end = std::upper_bound(start, src.end(), zone, by_zone{});
+
+      std::vector<T> out{};
+      out.assign(std::make_move_iterator(start), std::make_move_iterator(end));
+      src.erase(start, end);
+      return out;
+    }
+
+    template<typename Container, typename T>
+    void add_peers(Container& dest, std::vector<T>&& src)
+    {
+      dest.insert(std::make_move_iterator(src.begin()), std::make_move_iterator(src.end()));
+    }
+
+    template<typename Container, typename Range>
+    void copy_peers(Container& dest, const Range& src)
+    {
+      std::copy(src.begin(), src.end(), std::back_inserter(dest));
+    }
+  } // anonymous
+
+  struct peerlist_join
+  {
+    const peerlist_types& ours;
+    const peerlist_types& other;
+  };
+
+  template<typename Archive>
+  void serialize(Archive& a, peerlist_types& elem, unsigned ver)
+  {
+    elem.white = load_peers<peerlist_entry>(a, ver);
+    elem.gray = load_peers<peerlist_entry>(a, ver);
+    elem.anchor = load_peers<anchor_peerlist_entry>(a, ver);
+
+    if (ver == 0)
+    {
+      // from v1, we do not store the peer id anymore
+      peerid_type peer_id{};
+      a & peer_id;
+    }
+  }
+ 
+  template<typename Archive>
+  void serialize(Archive& a, peerlist_join elem, unsigned ver)
+  {
+    save_peers(a, boost::range::join(elem.ours.white, elem.other.white));
+    save_peers(a, boost::range::join(elem.ours.gray, elem.other.gray));
+    save_peers(a, boost::range::join(elem.ours.anchor, elem.other.anchor));
+  }
+
+  boost::optional<peerlist_storage> peerlist_storage::open(std::istream& src, const bool new_format)
+  {
+    try
+    {
+      peerlist_storage out{};
+      if (new_format)
+      {
+        boost::archive::portable_binary_iarchive a{src};
+        a >> out.m_types;
+      }
+      else
+      {
+        boost::archive::binary_iarchive a{src};
+        a >> out.m_types;
+      }
+
+      if (src.good())
+      {
+        std::sort(out.m_types.white.begin(), out.m_types.white.end(), by_zone{});
+        std::sort(out.m_types.gray.begin(), out.m_types.gray.end(), by_zone{});
+        std::sort(out.m_types.anchor.begin(), out.m_types.anchor.end(), by_zone{});
+        return {std::move(out)};
+      }
+    }
+    catch (const std::exception& e)
+    {}
+
+    return boost::none;
+  }
+
+  boost::optional<peerlist_storage> peerlist_storage::open(const std::string& path)
+  {
+    std::ifstream src_file{};
+    src_file.open( path , std::ios_base::binary | std::ios_base::in);
+    if(src_file.fail())
+      return boost::none;
+
+    boost::optional<peerlist_storage> out = open(src_file, true);
+    if (!out)
+    {
+      // if failed, try reading in unportable mode
+      boost::filesystem::copy_file(path, path + ".unportable", boost::filesystem::copy_option::overwrite_if_exists);
+      src_file.close();
+      src_file.open( path , std::ios_base::binary | std::ios_base::in);
+      if(src_file.fail())
+        return boost::none;
+
+      out = open(src_file, false);
+      if (!out)
+      {
+        // This is different from the `return boost::none` cases above. Those
+        // cases could fail due to bad file permissions, so a shutdown is
+        // likely more appropriate.
+        MWARNING("Failed to load p2p config file, falling back to default config");
+        out.emplace();
+      }
+    }
+
+    return out;
+  }
+
+  peerlist_storage::~peerlist_storage() noexcept
+  {}
+
+  bool peerlist_storage::store(std::ostream& dest, const peerlist_types& other) const
+  {
+    try
+    {
+      boost::archive::portable_binary_oarchive a{dest};
+      const peerlist_join pj{std::cref(m_types), std::cref(other)};
+      a << pj;
+      return dest.good();
+    }
+    catch (const boost::archive::archive_exception& e)
+    {}
+
+    return false;
+  }
+
+  bool peerlist_storage::store(const std::string& path, const peerlist_types& other) const
+  {
+    std::ofstream dest_file{};
+    dest_file.open( path , std::ios_base::binary | std::ios_base::out| std::ios::trunc);
+    if(dest_file.fail())
+      return false;
+
+    return store(dest_file, other);
+  }
+
+  peerlist_types peerlist_storage::take_zone(epee::net_utils::zone zone)
+  {
+    peerlist_types out{};
+    out.white = do_take_zone(m_types.white, zone);
+    out.gray = do_take_zone(m_types.gray, zone);
+    out.anchor = do_take_zone(m_types.anchor, zone);
+    return out;
+  }
+
+  bool peerlist_manager::init(peerlist_types&& peers, bool allow_local_ip)
+  {
+    CRITICAL_REGION_LOCAL(m_peerlist_lock);
+
+    if (!m_peers_white.empty() || !m_peers_gray.empty() || !m_peers_anchor.empty())
+      return false;
+
+    add_peers(m_peers_white.get<by_addr>(), std::move(peers.white));
+    add_peers(m_peers_gray.get<by_addr>(), std::move(peers.gray));
+    add_peers(m_peers_anchor.get<by_addr>(), std::move(peers.anchor));
+    m_allow_local_ip = allow_local_ip;
+    return true;
+  }
+
+  void peerlist_manager::get_peerlist(std::vector<peerlist_entry>& pl_gray, std::vector<peerlist_entry>& pl_white)
+  {
+    CRITICAL_REGION_LOCAL(m_peerlist_lock);
+    copy_peers(pl_gray, m_peers_gray.get<by_addr>());
+    copy_peers(pl_white, m_peers_white.get<by_addr>());
+  }
+
+  void peerlist_manager::get_peerlist(peerlist_types& peers)
+  { 
+    CRITICAL_REGION_LOCAL(m_peerlist_lock);
+    peers.white.reserve(peers.white.size() + m_peers_white.size());
+    peers.gray.reserve(peers.gray.size() + m_peers_gray.size());
+    peers.anchor.reserve(peers.anchor.size() + m_peers_anchor.size());
+
+    copy_peers(peers.white, m_peers_white.get<by_addr>());
+    copy_peers(peers.gray, m_peers_gray.get<by_addr>());
+    copy_peers(peers.anchor, m_peers_anchor.get<by_addr>());
+  }
+}
+
+BOOST_CLASS_VERSION(nodetool::peerlist_types, nodetool::CURRENT_PEERLIST_STORAGE_ARCHIVE_VER);
+BOOST_CLASS_VERSION(nodetool::peerlist_join, nodetool::CURRENT_PEERLIST_STORAGE_ARCHIVE_VER);
+

src/p2p/net_peerlist.h

@@ -30,33 +30,67 @@
 
 #pragma once
 
+#include <iosfwd>
 #include <list>
-#include <set>
+#include <string>
-#include <map>
+#include <vector>
-#include <boost/archive/binary_iarchive.hpp>
-#include <boost/archive/portable_binary_oarchive.hpp>
-#include <boost/archive/portable_binary_iarchive.hpp>
-#include <boost/serialization/version.hpp>
 
 #include <boost/multi_index_container.hpp>
 #include <boost/multi_index/ordered_index.hpp>
 #include <boost/multi_index/identity.hpp>
 #include <boost/multi_index/member.hpp>
+#include <boost/optional/optional.hpp>
 #include <boost/range/adaptor/reversed.hpp>
 
 
-#include "syncobj.h"
+#include "cryptonote_config.h"
+#include "net/enums.h"
 #include "net/local_ip.h"
 #include "p2p_protocol_defs.h"
-#include "cryptonote_config.h"
+#include "syncobj.h"
-#include "net_peerlist_boost_serialization.h"
-
-
-#define CURRENT_PEERLIST_STORAGE_ARCHIVE_VER    6
 
 namespace nodetool
 {
+  struct peerlist_types
+  {
+    std::vector<peerlist_entry> white;
+    std::vector<peerlist_entry> gray;
+    std::vector<anchor_peerlist_entry> anchor;
+  };
 
+  class peerlist_storage
+  {
+  public:
+    peerlist_storage()
+      : m_types{}
+    {}
+
+    //! \return Peers stored in stream `src` in `new_format` (portable archive or older non-portable).
+    static boost::optional<peerlist_storage> open(std::istream& src, const bool new_format);
+
+    //! \return Peers stored in file at `path`
+    static boost::optional<peerlist_storage> open(const std::string& path);
+
+    peerlist_storage(peerlist_storage&&) = default;
+    peerlist_storage(const peerlist_storage&) = delete;
+
+    ~peerlist_storage() noexcept;
+
+    peerlist_storage& operator=(peerlist_storage&&) = default;
+    peerlist_storage& operator=(const peerlist_storage&) = delete;
+
+    //! Save peers from `this` and `other` in stream `dest`.
+    bool store(std::ostream& dest, const peerlist_types& other) const;
+
+    //! Save peers from `this` and `other` in one file at `path`.
+    bool store(const std::string& path, const peerlist_types& other) const;
+
+    //! \return Peers in `zone` and from remove from `this`.
+    peerlist_types take_zone(epee::net_utils::zone zone);
+
+  private:
+    peerlist_types m_types;
+  };
 
   /************************************************************************/
   /*                                                                      */
@@ -64,13 +98,13 @@ namespace nodetool
   class peerlist_manager
   {
   public: 
-    bool init(bool allow_local_ip);
+    bool init(peerlist_types&& peers, bool allow_local_ip);
-    bool deinit();
     size_t get_white_peers_count(){CRITICAL_REGION_LOCAL(m_peerlist_lock); return m_peers_white.size();}
     size_t get_gray_peers_count(){CRITICAL_REGION_LOCAL(m_peerlist_lock); return m_peers_gray.size();}
     bool merge_peerlist(const std::vector<peerlist_entry>& outer_bs);
     bool get_peerlist_head(std::vector<peerlist_entry>& bs_head, uint32_t depth = P2P_DEFAULT_PEERS_IN_HANDSHAKE);
-    bool get_peerlist_full(std::vector<peerlist_entry>& pl_gray, std::vector<peerlist_entry>& pl_white);
+    void get_peerlist(std::vector<peerlist_entry>& pl_gray, std::vector<peerlist_entry>& pl_white);
+    void get_peerlist(peerlist_types& peers);
     bool get_white_peer_by_index(peerlist_entry& p, size_t i);
     bool get_gray_peer_by_index(peerlist_entry& p, size_t i);
     template<typename F> bool foreach(bool white, const F &f);
@@ -135,18 +169,6 @@ namespace nodetool
       > 
     > peers_indexed;
 
-    typedef boost::multi_index_container<
-      peerlist_entry,
-      boost::multi_index::indexed_by<
-      // access by peerlist_entry::id<
-      boost::multi_index::ordered_unique<boost::multi_index::tag<by_id>, boost::multi_index::member<peerlist_entry,uint64_t,&peerlist_entry::id> >,
-      // access by peerlist_entry::net_adress
-      boost::multi_index::ordered_unique<boost::multi_index::tag<by_addr>, boost::multi_index::member<peerlist_entry,epee::net_utils::network_address,&peerlist_entry::adr> >,
-      // sort by peerlist_entry::last_seen<
-      boost::multi_index::ordered_non_unique<boost::multi_index::tag<by_time>, boost::multi_index::member<peerlist_entry,int64_t,&peerlist_entry::last_seen> >
-      > 
-    > peers_indexed_old;
-
     typedef boost::multi_index_container<
       anchor_peerlist_entry,
       boost::multi_index::indexed_by<
@@ -156,56 +178,8 @@ namespace nodetool
       boost::multi_index::ordered_non_unique<boost::multi_index::tag<by_time>, boost::multi_index::member<anchor_peerlist_entry,int64_t,&anchor_peerlist_entry::first_seen> >
       >
     > anchor_peers_indexed;
-  public:    
-    
-    template <class Archive, class List, class Element, class t_version_type>
-    void serialize_peers(Archive &a, List &list, Element ple, const t_version_type ver)
-    {
-      if (typename Archive::is_saving())
-      {
-        uint64_t size = list.size();
-        a & size;
-        for (auto p: list)
-        {
-          a & p;
-        }
-      }
-      else
-      {
-        uint64_t size;
-        a & size;
-        list.clear();
-        while (size--)
-        {
-          a & ple;
-          list.insert(ple);
-        }
-      }
-    }
-
-    template <class Archive, class t_version_type>
-    void serialize(Archive &a,  const t_version_type ver)
-    {
-      // at v6, we drop existing peerlists, because annoying change
-      if (ver < 6)
-        return;
-
-      CRITICAL_REGION_LOCAL(m_peerlist_lock);
-
-#if 0
-      // trouble loading more than one peer, can't find why
-      a & m_peers_white;
-      a & m_peers_gray;
-      a & m_peers_anchor;
-#else
-      serialize_peers(a, m_peers_white, peerlist_entry(), ver);
-      serialize_peers(a, m_peers_gray, peerlist_entry(), ver);
-      serialize_peers(a, m_peers_anchor, anchor_peerlist_entry(), ver);
-#endif
-    }
 
   private: 
-    bool peers_indexed_from_old(const peers_indexed_old& pio, peers_indexed& pi);
     void trim_white_peerlist();
     void trim_gray_peerlist();
 
@@ -220,34 +194,6 @@ namespace nodetool
     anchor_peers_indexed m_peers_anchor;
   };
   //--------------------------------------------------------------------------------------------------
-  inline
-  bool peerlist_manager::init(bool allow_local_ip)
-  {
-    m_allow_local_ip = allow_local_ip;
-    return true;
-  } 
-  //--------------------------------------------------------------------------------------------------
-  inline
-    bool peerlist_manager::deinit()
-  {
-    return true;
-  }
-  //--------------------------------------------------------------------------------------------------
-  inline 
-  bool peerlist_manager::peers_indexed_from_old(const peers_indexed_old& pio, peers_indexed& pi)
-  {
-    for(auto x: pio)
-    {
-      auto by_addr_it = pi.get<by_addr>().find(x.adr);
-      if(by_addr_it == pi.get<by_addr>().end())
-      {
-        pi.insert(x);
-      }
-    }
-
-    return true;
-  }
-  //--------------------------------------------------------------------------------------------------
   inline void peerlist_manager::trim_gray_peerlist()
   {
     while(m_peers_gray.size() > P2P_LOCAL_GRAY_PEERLIST_LIMIT)
@@ -337,27 +283,6 @@ namespace nodetool
     return true;
   }
   //--------------------------------------------------------------------------------------------------
-  inline
-  bool peerlist_manager::get_peerlist_full(std::vector<peerlist_entry>& pl_gray, std::vector<peerlist_entry>& pl_white)
-  {    
-    CRITICAL_REGION_LOCAL(m_peerlist_lock);
-    peers_indexed::index<by_time>::type& by_time_index_gr=m_peers_gray.get<by_time>();
-    pl_gray.resize(pl_gray.size() + by_time_index_gr.size());
-    for(const peers_indexed::value_type& vl: boost::adaptors::reverse(by_time_index_gr))
-    {
-      pl_gray.push_back(vl);      
-    }
-
-    peers_indexed::index<by_time>::type& by_time_index_wt=m_peers_white.get<by_time>();
-    pl_white.resize(pl_white.size() + by_time_index_wt.size());
-    for(const peers_indexed::value_type& vl: boost::adaptors::reverse(by_time_index_wt))
-    {
-      pl_white.push_back(vl);      
-    }
-
-    return true;
-  }
-  //--------------------------------------------------------------------------------------------------
   template<typename F> inline
   bool peerlist_manager::foreach(bool white, const F &f)
   {
@@ -559,4 +484,3 @@ namespace nodetool
   //--------------------------------------------------------------------------------------------------
 }
 
-BOOST_CLASS_VERSION(nodetool::peerlist_manager, CURRENT_PEERLIST_STORAGE_ARCHIVE_VER)

src/p2p/net_peerlist_boost_serialization.h

@@ -30,7 +30,11 @@
 
 #pragma once
 
+#include <cstring>
+
+#include "common/expect.h"
 #include "net/net_utils_base.h"
+#include "net/tor_address.h"
 #include "p2p/p2p_protocol_defs.h"
 
 #ifdef CRYPTONOTE_PRUNING_DEBUG_SPOOF_SEED
@@ -42,24 +46,37 @@ namespace boost
   namespace serialization
   {
     template <class T, class Archive>
-    inline void do_serialize(Archive &a, epee::net_utils::network_address& na, T local)
+    inline void do_serialize(boost::mpl::false_, Archive &a, epee::net_utils::network_address& na)
     {
-      if (typename Archive::is_saving()) local = na.as<T>();
+      T addr{};
-      a & local;
+      a & addr;
-      if (!typename Archive::is_saving()) na = local;
+      na = std::move(addr);
     }
+
+    template <class T, class Archive>
+    inline void do_serialize(boost::mpl::true_, Archive &a, const epee::net_utils::network_address& na)
+    {
+      a & na.as<T>();
+    }
+
     template <class Archive, class ver_type>
     inline void serialize(Archive &a, epee::net_utils::network_address& na, const ver_type ver)
     {
+      static constexpr const typename Archive::is_saving is_saving{};
+
       uint8_t type;
-      if (typename Archive::is_saving())
+      if (is_saving)
-        type = na.get_type_id();
+        type = uint8_t(na.get_type_id());
       a & type;
-      switch (type)
+      switch (epee::net_utils::address_type(type))
       {
-        case epee::net_utils::ipv4_network_address::ID:
+        case epee::net_utils::ipv4_network_address::get_type_id():
-	  do_serialize(a, na, epee::net_utils::ipv4_network_address{0, 0});
+          do_serialize<epee::net_utils::ipv4_network_address>(is_saving, a, na);
           break;
+        case net::tor_address::get_type_id():
+          do_serialize<net::tor_address>(is_saving, a, na);
+          break;
+        case epee::net_utils::address_type::invalid:
         default:
           throw std::runtime_error("Unsupported network address type");
       }
@@ -75,6 +92,47 @@ namespace boost
         na = epee::net_utils::ipv4_network_address{ip, port};
     }
 
+    template <class Archive, class ver_type>
+    inline void save(Archive& a, const net::tor_address& na, const ver_type)
+    {
+      const size_t length = std::strlen(na.host_str());
+      if (length > 255)
+        MONERO_THROW(net::error::invalid_tor_address, "Tor address too long");
+
+      const uint16_t port{na.port()};
+      const uint8_t len = length;
+      a & port;
+      a & len;
+      a.save_binary(na.host_str(), length);
+    }
+
+    template <class Archive, class ver_type>
+    inline void load(Archive& a, net::tor_address& na, const ver_type)
+    {
+      uint16_t port = 0;
+      uint8_t length = 0;
+      a & port;
+      a & length;
+
+      if (length > net::tor_address::buffer_size())
+        MONERO_THROW(net::error::invalid_tor_address, "Tor address too long");
+
+      char host[net::tor_address::buffer_size()] = {0};
+      a.load_binary(host, length);
+      host[sizeof(host) - 1] = 0;
+
+      if (std::strcmp(host, net::tor_address::unknown_str()) == 0)
+        na = net::tor_address::unknown();
+      else
+        na = MONERO_UNWRAP(net::tor_address::make(host, port));
+    }
+
+    template <class Archive, class ver_type>
+    inline void serialize(Archive &a, net::tor_address& na, const ver_type ver)
+    {
+      boost::serialization::split_free(a, na, ver);
+    }
+
     template <class Archive, class ver_type>
     inline void serialize(Archive &a,  nodetool::peerlist_entry& pl, const ver_type ver)
     {

src/p2p/p2p_protocol_defs.h

@@ -34,6 +34,7 @@
 #include <boost/serialization/version.hpp>
 #include "serialization/keyvalue_serialization.h"
 #include "net/net_utils_base.h"
+#include "net/tor_address.h" // needed for serialization
 #include "misc_language.h"
 #include "string_tools.h"
 #include "time_helper.h"
@@ -204,7 +205,7 @@ namespace nodetool
           std::vector<peerlist_entry_base<network_address_old>> local_peerlist;
           for (const auto &p: this_ref.local_peerlist_new)
           {
-            if (p.adr.get_type_id() == epee::net_utils::ipv4_network_address::ID)
+            if (p.adr.get_type_id() == epee::net_utils::ipv4_network_address::get_type_id())
             {
               const epee::net_utils::network_address  &na = p.adr;
               const epee::net_utils::ipv4_network_address &ipv4 = na.as<const epee::net_utils::ipv4_network_address>();
@@ -263,7 +264,7 @@ namespace nodetool
           std::vector<peerlist_entry_base<network_address_old>> local_peerlist;
           for (const auto &p: this_ref.local_peerlist_new)
           {
-            if (p.adr.get_type_id() == epee::net_utils::ipv4_network_address::ID)
+            if (p.adr.get_type_id() == epee::net_utils::ipv4_network_address::get_type_id())
             {
               const epee::net_utils::network_address  &na = p.adr;
               const epee::net_utils::ipv4_network_address &ipv4 = na.as<const epee::net_utils::ipv4_network_address>();

src/rpc/CMakeLists.txt

@@ -112,6 +112,7 @@ target_link_libraries(rpc
     common
     cryptonote_core
     cryptonote_protocol
+    net
     version
     ${Boost_REGEX_LIBRARY}
     ${Boost_THREAD_LIBRARY}

src/rpc/core_rpc_server.cpp

@@ -42,6 +42,7 @@ using namespace epee;
 #include "cryptonote_basic/account.h"
 #include "cryptonote_basic/cryptonote_basic_impl.h"
 #include "misc_language.h"
+#include "net/parse.h"
 #include "storages/http_abstract_invoke.h"
 #include "crypto/hash.h"
 #include "rpc/rpc_args.h"
@@ -185,12 +186,12 @@ namespace cryptonote
     res.tx_count = m_core.get_blockchain_storage().get_total_transactions() - res.height; //without coinbase
     res.tx_pool_size = m_core.get_pool_transactions_count();
     res.alt_blocks_count = restricted ? 0 : m_core.get_blockchain_storage().get_alternative_blocks_count();
-    uint64_t total_conn = restricted ? 0 : m_p2p.get_connections_count();
+    uint64_t total_conn = restricted ? 0 : m_p2p.get_public_connections_count();
-    res.outgoing_connections_count = restricted ? 0 : m_p2p.get_outgoing_connections_count();
+    res.outgoing_connections_count = restricted ? 0 : m_p2p.get_public_outgoing_connections_count();
     res.incoming_connections_count = restricted ? 0 : (total_conn - res.outgoing_connections_count);
     res.rpc_connections_count = restricted ? 0 : get_connections_count();
-    res.white_peerlist_size = restricted ? 0 : m_p2p.get_peerlist_manager().get_white_peers_count();
+    res.white_peerlist_size = restricted ? 0 : m_p2p.get_public_white_peers_count();
-    res.grey_peerlist_size = restricted ? 0 : m_p2p.get_peerlist_manager().get_gray_peers_count();
+    res.grey_peerlist_size = restricted ? 0 : m_p2p.get_public_gray_peers_count();
 
     cryptonote::network_type net_type = nettype();
     res.mainnet = net_type == MAINNET;
@@ -902,12 +903,12 @@ namespace cryptonote
     PERF_TIMER(on_get_peer_list);
     std::vector<nodetool::peerlist_entry> white_list;
     std::vector<nodetool::peerlist_entry> gray_list;
-    m_p2p.get_peerlist_manager().get_peerlist_full(gray_list, white_list);
+    m_p2p.get_public_peerlist(gray_list, white_list);
 
     res.white_list.reserve(white_list.size());
     for (auto & entry : white_list)
     {
-      if (entry.adr.get_type_id() == epee::net_utils::ipv4_network_address::ID)
+      if (entry.adr.get_type_id() == epee::net_utils::ipv4_network_address::get_type_id())
         res.white_list.emplace_back(entry.id, entry.adr.as<epee::net_utils::ipv4_network_address>().ip(),
             entry.adr.as<epee::net_utils::ipv4_network_address>().port(), entry.last_seen, entry.pruning_seed);
       else
@@ -917,7 +918,7 @@ namespace cryptonote
     res.gray_list.reserve(gray_list.size());
     for (auto & entry : gray_list)
     {
-      if (entry.adr.get_type_id() == epee::net_utils::ipv4_network_address::ID)
+      if (entry.adr.get_type_id() == epee::net_utils::ipv4_network_address::get_type_id())
         res.gray_list.emplace_back(entry.id, entry.adr.as<epee::net_utils::ipv4_network_address>().ip(),
             entry.adr.as<epee::net_utils::ipv4_network_address>().port(), entry.last_seen, entry.pruning_seed);
       else
@@ -1646,12 +1647,12 @@ namespace cryptonote
     res.tx_count = m_core.get_blockchain_storage().get_total_transactions() - res.height; //without coinbase
     res.tx_pool_size = m_core.get_pool_transactions_count();
     res.alt_blocks_count = restricted ? 0 : m_core.get_blockchain_storage().get_alternative_blocks_count();
-    uint64_t total_conn = restricted ? 0 : m_p2p.get_connections_count();
+    uint64_t total_conn = restricted ? 0 : m_p2p.get_public_connections_count();
-    res.outgoing_connections_count = restricted ? 0 : m_p2p.get_outgoing_connections_count();
+    res.outgoing_connections_count = restricted ? 0 : m_p2p.get_public_outgoing_connections_count();
     res.incoming_connections_count = restricted ? 0 : (total_conn - res.outgoing_connections_count);
     res.rpc_connections_count = restricted ? 0 : get_connections_count();
-    res.white_peerlist_size = restricted ? 0 : m_p2p.get_peerlist_manager().get_white_peers_count();
+    res.white_peerlist_size = restricted ? 0 : m_p2p.get_public_white_peers_count();
-    res.grey_peerlist_size = restricted ? 0 : m_p2p.get_peerlist_manager().get_gray_peers_count();
+    res.grey_peerlist_size = restricted ? 0 : m_p2p.get_public_gray_peers_count();
 
     cryptonote::network_type net_type = nettype();
     res.mainnet = net_type == MAINNET;
@@ -1730,12 +1731,14 @@ namespace cryptonote
       epee::net_utils::network_address na;
       if (!i->host.empty())
       {
-        if (!epee::net_utils::create_network_address(na, i->host))
+        auto na_parsed = net::get_network_address(i->host, 0);
+        if (!na_parsed)
         {
           error_resp.code = CORE_RPC_ERROR_CODE_WRONG_PARAM;
           error_resp.message = "Unsupported host type";
           return false;
         }
+        na = std::move(*na_parsed);
       }
       else
       {
@@ -1958,11 +1961,7 @@ namespace cryptonote
   bool core_rpc_server::on_out_peers(const COMMAND_RPC_OUT_PEERS::request& req, COMMAND_RPC_OUT_PEERS::response& res, const connection_context *ctx)
   {
     PERF_TIMER(on_out_peers);
-    size_t n_connections = m_p2p.get_outgoing_connections_count();
+    m_p2p.change_max_out_public_peers(req.out_peers);
-    size_t n_delete = (n_connections > req.out_peers) ? n_connections - req.out_peers : 0;
-    m_p2p.m_config.m_net_config.max_out_connection_count = req.out_peers;
-    if (n_delete)
-      m_p2p.delete_out_connections(n_delete);
     res.status = CORE_RPC_STATUS_OK;
     return true;
   }
@@ -1970,11 +1969,7 @@ namespace cryptonote
   bool core_rpc_server::on_in_peers(const COMMAND_RPC_IN_PEERS::request& req, COMMAND_RPC_IN_PEERS::response& res, const connection_context *ctx)
   {
     PERF_TIMER(on_in_peers);
-    size_t n_connections = m_p2p.get_incoming_connections_count();
+    m_p2p.change_max_in_public_peers(req.in_peers);
-    size_t n_delete = (n_connections > req.in_peers) ? n_connections - req.in_peers : 0;
-    m_p2p.m_config.m_net_config.max_in_connection_count = req.in_peers;
-    if (n_delete)
-      m_p2p.delete_in_connections(n_delete);
     res.status = CORE_RPC_STATUS_OK;
     return true;
   }

src/rpc/daemon_handler.cpp

@@ -423,13 +423,13 @@ namespace rpc
 
     res.info.alt_blocks_count = chain.get_alternative_blocks_count();
 
-    uint64_t total_conn = m_p2p.get_connections_count();
+    uint64_t total_conn = m_p2p.get_public_connections_count();
-    res.info.outgoing_connections_count = m_p2p.get_outgoing_connections_count();
+    res.info.outgoing_connections_count = m_p2p.get_public_outgoing_connections_count();
     res.info.incoming_connections_count = total_conn - res.info.outgoing_connections_count;
 
-    res.info.white_peerlist_size = m_p2p.get_peerlist_manager().get_white_peers_count();
+    res.info.white_peerlist_size = m_p2p.get_public_white_peers_count();
 
-    res.info.grey_peerlist_size = m_p2p.get_peerlist_manager().get_gray_peers_count();
+    res.info.grey_peerlist_size = m_p2p.get_public_gray_peers_count();
 
     res.info.mainnet = m_core.get_nettype() == MAINNET;
     res.info.testnet = m_core.get_nettype() == TESTNET;

tests/unit_tests/CMakeLists.txt

@@ -62,6 +62,7 @@ set(unit_tests_sources
   mul_div.cpp
   multiexp.cpp
   multisig.cpp
+  net.cpp
   notify.cpp
   output_distribution.cpp
   parse_amount.cpp
@@ -100,6 +101,7 @@ target_link_libraries(unit_tests
     cryptonote_core
     blockchain_db
     rpc
+    net
     serialization
     wallet
     p2p

tests/unit_tests/epee_utils.cpp

@@ -545,6 +545,8 @@ TEST(StringTools, GetIpInt32)
 
 TEST(NetUtils, IPv4NetworkAddress)
 {
+  static_assert(epee::net_utils::ipv4_network_address::get_type_id() == epee::net_utils::address_type::ipv4, "bad ipv4 type id");
+
   const auto ip1 = boost::endian::native_to_big(0x330012FFu);
   const auto ip_loopback = boost::endian::native_to_big(0x7F000001u);
   const auto ip_local = boost::endian::native_to_big(0x0A000000u);
@@ -555,7 +557,7 @@ TEST(NetUtils, IPv4NetworkAddress)
   EXPECT_STREQ("51.0.18.255", address1.host_str().c_str());
   EXPECT_FALSE(address1.is_loopback());
   EXPECT_FALSE(address1.is_local());
-  EXPECT_EQ(epee::net_utils::ipv4_network_address::ID, address1.get_type_id());
+  EXPECT_EQ(epee::net_utils::ipv4_network_address::get_type_id(), address1.get_type_id());
   EXPECT_EQ(ip1, address1.ip());
   EXPECT_EQ(65535, address1.port());
   EXPECT_TRUE(epee::net_utils::ipv4_network_address{std::move(address1)} == address1);
@@ -568,7 +570,7 @@ TEST(NetUtils, IPv4NetworkAddress)
   EXPECT_STREQ("127.0.0.1", loopback.host_str().c_str());
   EXPECT_TRUE(loopback.is_loopback());
   EXPECT_FALSE(loopback.is_local());
-  EXPECT_EQ(epee::net_utils::ipv4_network_address::ID, address1.get_type_id());
+  EXPECT_EQ(epee::net_utils::ipv4_network_address::get_type_id(), address1.get_type_id());
   EXPECT_EQ(ip_loopback, loopback.ip());
   EXPECT_EQ(0, loopback.port());
 
@@ -624,7 +626,9 @@ TEST(NetUtils, NetworkAddress)
     constexpr static bool is_local() noexcept { return false; }
     static std::string str() { return {}; }
     static std::string host_str() { return {}; }
-    constexpr static uint8_t get_type_id() noexcept { return uint8_t(-1); }
+    constexpr static epee::net_utils::address_type get_type_id() noexcept { return epee::net_utils::address_type(-1); }
+    constexpr static epee::net_utils::zone get_zone() noexcept { return epee::net_utils::zone::invalid; }
+    constexpr static bool is_blockable() noexcept { return false; }
   };
 
   const epee::net_utils::network_address empty;
@@ -634,7 +638,9 @@ TEST(NetUtils, NetworkAddress)
   EXPECT_STREQ("<none>", empty.host_str().c_str());
   EXPECT_FALSE(empty.is_loopback());
   EXPECT_FALSE(empty.is_local());
-  EXPECT_EQ(0, empty.get_type_id());
+  EXPECT_EQ(epee::net_utils::address_type::invalid, empty.get_type_id());
+  EXPECT_EQ(epee::net_utils::zone::invalid, empty.get_zone());
+  EXPECT_FALSE(empty.is_blockable());
   EXPECT_THROW(empty.as<custom_address>(), std::bad_cast);
 
   epee::net_utils::network_address address1{
@@ -650,7 +656,9 @@ TEST(NetUtils, NetworkAddress)
   EXPECT_STREQ("51.0.18.255", address1.host_str().c_str());
   EXPECT_FALSE(address1.is_loopback());
   EXPECT_FALSE(address1.is_local());
-  EXPECT_EQ(epee::net_utils::ipv4_network_address::ID, address1.get_type_id());
+  EXPECT_EQ(epee::net_utils::ipv4_network_address::get_type_id(), address1.get_type_id());
+  EXPECT_EQ(epee::net_utils::zone::public_, address1.get_zone());
+  EXPECT_TRUE(address1.is_blockable());
   EXPECT_NO_THROW(address1.as<epee::net_utils::ipv4_network_address>());
   EXPECT_THROW(address1.as<custom_address>(), std::bad_cast);
 
@@ -667,7 +675,9 @@ TEST(NetUtils, NetworkAddress)
   EXPECT_STREQ("127.0.0.1", loopback.host_str().c_str());
   EXPECT_TRUE(loopback.is_loopback());
   EXPECT_FALSE(loopback.is_local());
-  EXPECT_EQ(epee::net_utils::ipv4_network_address::ID, address1.get_type_id());
+  EXPECT_EQ(epee::net_utils::ipv4_network_address::get_type_id(), address1.get_type_id());
+  EXPECT_EQ(epee::net_utils::zone::public_, address1.get_zone());
+  EXPECT_EQ(epee::net_utils::ipv4_network_address::get_type_id(), address1.get_type_id());
 
   const epee::net_utils::network_address local{
     epee::net_utils::ipv4_network_address{ip_local, 8080}

tests/unit_tests/net.cpp

@@ -0,0 +1,745 @@
+// Copyright (c) 2018, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <atomic>
+#include <boost/archive/portable_binary_oarchive.hpp>
+#include <boost/archive/portable_binary_iarchive.hpp>
+#include <boost/asio/buffer.hpp>
+#include <boost/asio/io_service.hpp>
+#include <boost/asio/ip/tcp.hpp>
+#include <boost/asio/read.hpp>
+#include <boost/asio/write.hpp>
+#include <boost/endian/conversion.hpp>
+#include <boost/system/error_code.hpp>
+#include <boost/thread/thread.hpp>
+#include <cstring>
+#include <functional>
+#include <gtest/gtest.h>
+#include <memory>
+
+#include "net/error.h"
+#include "net/net_utils_base.h"
+#include "net/socks.h"
+#include "net/parse.h"
+#include "net/tor_address.h"
+#include "p2p/net_peerlist_boost_serialization.h"
+#include "serialization/keyvalue_serialization.h"
+#include "storages/portable_storage.h"
+
+namespace
+{
+    static constexpr const char v2_onion[] =
+        "xmrto2bturnore26.onion";
+    static constexpr const char v3_onion[] =
+        "vww6ybal4bd7szmgncyruucpgfkqahzddi37ktceo3ah7ngmcopnpyyd.onion";
+}
+
+TEST(tor_address, constants)
+{
+    static_assert(!net::tor_address::is_local(), "bad is_local() response");
+    static_assert(!net::tor_address::is_loopback(), "bad is_loopback() response");
+    static_assert(net::tor_address::get_type_id() == epee::net_utils::address_type::tor, "bad get_type_id() response");
+
+    EXPECT_FALSE(net::tor_address::is_local());
+    EXPECT_FALSE(net::tor_address::is_loopback());
+    EXPECT_EQ(epee::net_utils::address_type::tor, net::tor_address::get_type_id());
+    EXPECT_EQ(epee::net_utils::address_type::tor, net::tor_address::get_type_id());
+}
+
+TEST(tor_address, invalid)
+{
+    EXPECT_TRUE(net::tor_address::make("").has_error());
+    EXPECT_TRUE(net::tor_address::make(":").has_error());
+    EXPECT_TRUE(net::tor_address::make(".onion").has_error());
+    EXPECT_TRUE(net::tor_address::make(".onion:").has_error());
+    EXPECT_TRUE(net::tor_address::make(v2_onion + 1).has_error());
+    EXPECT_TRUE(net::tor_address::make(v3_onion + 1).has_error());
+    EXPECT_TRUE(net::tor_address::make(boost::string_ref{v2_onion, sizeof(v2_onion) - 2}).has_error());
+    EXPECT_TRUE(net::tor_address::make(boost::string_ref{v3_onion, sizeof(v3_onion) - 2}).has_error());
+    EXPECT_TRUE(net::tor_address::make(std::string{v2_onion} + ":-").has_error());
+    EXPECT_TRUE(net::tor_address::make(std::string{v2_onion} + ":900a").has_error());
+    EXPECT_TRUE(net::tor_address::make(std::string{v3_onion} + ":65536").has_error());
+    EXPECT_TRUE(net::tor_address::make(std::string{v3_onion} + ":-1").has_error());
+
+    std::string onion{v3_onion};
+    onion.at(10) = 1;
+    EXPECT_TRUE(net::tor_address::make(onion).has_error());
+}
+
+TEST(tor_address, unblockable_types)
+{
+    net::tor_address tor{};
+
+    ASSERT_NE(nullptr, tor.host_str());
+    EXPECT_STREQ("<unknown tor host>", tor.host_str());
+    EXPECT_STREQ("<unknown tor host>", tor.str().c_str());
+    EXPECT_EQ(0u, tor.port());
+    EXPECT_TRUE(tor.is_unknown());
+    EXPECT_FALSE(tor.is_local());
+    EXPECT_FALSE(tor.is_loopback());
+    EXPECT_EQ(epee::net_utils::address_type::tor, tor.get_type_id());
+    EXPECT_EQ(epee::net_utils::zone::tor, tor.get_zone());
+
+    tor = net::tor_address::unknown();
+    ASSERT_NE(nullptr, tor.host_str());
+    EXPECT_STREQ("<unknown tor host>", tor.host_str());
+    EXPECT_STREQ("<unknown tor host>", tor.str().c_str());
+    EXPECT_EQ(0u, tor.port());
+    EXPECT_TRUE(tor.is_unknown());
+    EXPECT_FALSE(tor.is_local());
+    EXPECT_FALSE(tor.is_loopback());
+    EXPECT_EQ(epee::net_utils::address_type::tor, tor.get_type_id());
+    EXPECT_EQ(epee::net_utils::zone::tor, tor.get_zone());
+
+    EXPECT_EQ(net::tor_address{}, net::tor_address::unknown());
+}
+
+TEST(tor_address, valid)
+{
+    const auto address1 = net::tor_address::make(v3_onion);
+
+    ASSERT_TRUE(address1.has_value());
+    EXPECT_EQ(0u, address1->port());
+    EXPECT_STREQ(v3_onion, address1->host_str());
+    EXPECT_STREQ(v3_onion, address1->str().c_str());
+    EXPECT_TRUE(address1->is_blockable());
+
+    net::tor_address address2{*address1};
+
+    EXPECT_EQ(0u, address2.port());
+    EXPECT_STREQ(v3_onion, address2.host_str());
+    EXPECT_STREQ(v3_onion, address2.str().c_str());
+    EXPECT_TRUE(address2.is_blockable());
+    EXPECT_TRUE(address2.equal(*address1));
+    EXPECT_TRUE(address1->equal(address2));
+    EXPECT_TRUE(address2 == *address1);
+    EXPECT_TRUE(*address1 == address2);
+    EXPECT_FALSE(address2 != *address1);
+    EXPECT_FALSE(*address1 != address2);
+    EXPECT_TRUE(address2.is_same_host(*address1));
+    EXPECT_TRUE(address1->is_same_host(address2));
+    EXPECT_FALSE(address2.less(*address1));
+    EXPECT_FALSE(address1->less(address2));
+
+    address2 = MONERO_UNWRAP(net::tor_address::make(std::string{v2_onion} + ":6545"));
+
+    EXPECT_EQ(6545, address2.port());
+    EXPECT_STREQ(v2_onion, address2.host_str());
+    EXPECT_EQ(std::string{v2_onion} + ":6545", address2.str().c_str());
+    EXPECT_TRUE(address2.is_blockable());
+    EXPECT_FALSE(address2.equal(*address1));
+    EXPECT_FALSE(address1->equal(address2));
+    EXPECT_FALSE(address2 == *address1);
+    EXPECT_FALSE(*address1 == address2);
+    EXPECT_TRUE(address2 != *address1);
+    EXPECT_TRUE(*address1 != address2);
+    EXPECT_FALSE(address2.is_same_host(*address1));
+    EXPECT_FALSE(address1->is_same_host(address2));
+    EXPECT_FALSE(address2.less(*address1));
+    EXPECT_TRUE(address1->less(address2));
+
+    address2 = MONERO_UNWRAP(net::tor_address::make(std::string{v3_onion} + ":", 65535));
+
+    EXPECT_EQ(65535, address2.port());
+    EXPECT_STREQ(v3_onion, address2.host_str());
+    EXPECT_EQ(std::string{v3_onion} + ":65535", address2.str().c_str());
+    EXPECT_TRUE(address2.is_blockable());
+    EXPECT_FALSE(address2.equal(*address1));
+    EXPECT_FALSE(address1->equal(address2));
+    EXPECT_FALSE(address2 == *address1);
+    EXPECT_FALSE(*address1 == address2);
+    EXPECT_TRUE(address2 != *address1);
+    EXPECT_TRUE(*address1 != address2);
+    EXPECT_TRUE(address2.is_same_host(*address1));
+    EXPECT_TRUE(address1->is_same_host(address2));
+    EXPECT_FALSE(address2.less(*address1));
+    EXPECT_TRUE(address1->less(address2));
+}
+
+TEST(tor_address, generic_network_address)
+{
+    const epee::net_utils::network_address tor1{MONERO_UNWRAP(net::tor_address::make(v3_onion, 8080))};
+    const epee::net_utils::network_address tor2{MONERO_UNWRAP(net::tor_address::make(v3_onion, 8080))};
+    const epee::net_utils::network_address ip{epee::net_utils::ipv4_network_address{100, 200}};
+
+    EXPECT_EQ(tor1, tor2);
+    EXPECT_NE(ip, tor1);
+    EXPECT_LT(ip, tor1);
+
+    EXPECT_STREQ(v3_onion, tor1.host_str().c_str());
+    EXPECT_EQ(std::string{v3_onion} + ":8080", tor1.str());
+    EXPECT_EQ(epee::net_utils::address_type::tor, tor1.get_type_id());
+    EXPECT_EQ(epee::net_utils::address_type::tor, tor2.get_type_id());
+    EXPECT_EQ(epee::net_utils::address_type::ipv4, ip.get_type_id());
+    EXPECT_EQ(epee::net_utils::zone::tor, tor1.get_zone());
+    EXPECT_EQ(epee::net_utils::zone::tor, tor2.get_zone());
+    EXPECT_EQ(epee::net_utils::zone::public_, ip.get_zone());
+    EXPECT_TRUE(tor1.is_blockable());
+    EXPECT_TRUE(tor2.is_blockable());
+    EXPECT_TRUE(ip.is_blockable());
+}
+
+namespace
+{
+    struct test_command
+    {
+        net::tor_address tor;
+
+        BEGIN_KV_SERIALIZE_MAP()
+            KV_SERIALIZE(tor);
+        END_KV_SERIALIZE_MAP()
+    };
+}
+
+TEST(tor_address, epee_serializev_v2)
+{
+    std::string buffer{};
+    {
+        test_command command{MONERO_UNWRAP(net::tor_address::make(v2_onion, 10))};
+        EXPECT_FALSE(command.tor.is_unknown());
+        EXPECT_NE(net::tor_address{}, command.tor);
+        EXPECT_STREQ(v2_onion, command.tor.host_str());
+        EXPECT_EQ(10u, command.tor.port());
+
+        epee::serialization::portable_storage stg{};
+        EXPECT_TRUE(command.store(stg));
+        EXPECT_TRUE(stg.store_to_binary(buffer));
+    }
+
+    test_command command{};
+    {
+        EXPECT_TRUE(command.tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, command.tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), command.tor.host_str());
+        EXPECT_EQ(0u, command.tor.port());
+
+        epee::serialization::portable_storage stg{};
+        EXPECT_TRUE(stg.load_from_binary(buffer));
+        EXPECT_TRUE(command.load(stg));
+    }
+    EXPECT_FALSE(command.tor.is_unknown());
+    EXPECT_NE(net::tor_address{}, command.tor);
+    EXPECT_STREQ(v2_onion, command.tor.host_str());
+    EXPECT_EQ(10u, command.tor.port());
+
+    // make sure that exceeding max buffer doesn't destroy tor_address::_load
+    {
+        epee::serialization::portable_storage stg{};
+        stg.load_from_binary(buffer);
+
+        std::string host{};
+        ASSERT_TRUE(stg.get_value("host", host, stg.open_section("tor", nullptr, false)));
+        EXPECT_EQ(std::strlen(v2_onion), host.size());
+
+        host.push_back('k');
+        EXPECT_TRUE(stg.set_value("host", host, stg.open_section("tor", nullptr, false)));
+        EXPECT_TRUE(command.load(stg)); // poor error reporting from `KV_SERIALIZE`
+    }
+
+    EXPECT_TRUE(command.tor.is_unknown());
+    EXPECT_EQ(net::tor_address{}, command.tor);
+    EXPECT_STREQ(net::tor_address::unknown_str(), command.tor.host_str());
+    EXPECT_EQ(0u, command.tor.port());
+}
+
+TEST(tor_address, epee_serializev_v3)
+{
+    std::string buffer{};
+    {
+        test_command command{MONERO_UNWRAP(net::tor_address::make(v3_onion, 10))};
+        EXPECT_FALSE(command.tor.is_unknown());
+        EXPECT_NE(net::tor_address{}, command.tor);
+        EXPECT_STREQ(v3_onion, command.tor.host_str());
+        EXPECT_EQ(10u, command.tor.port());
+
+        epee::serialization::portable_storage stg{};
+        EXPECT_TRUE(command.store(stg));
+        EXPECT_TRUE(stg.store_to_binary(buffer));
+    }
+
+    test_command command{};
+    {
+        EXPECT_TRUE(command.tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, command.tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), command.tor.host_str());
+        EXPECT_EQ(0u, command.tor.port());
+
+        epee::serialization::portable_storage stg{};
+        EXPECT_TRUE(stg.load_from_binary(buffer));
+        EXPECT_TRUE(command.load(stg));
+    }
+    EXPECT_FALSE(command.tor.is_unknown());
+    EXPECT_NE(net::tor_address{}, command.tor);
+    EXPECT_STREQ(v3_onion, command.tor.host_str());
+    EXPECT_EQ(10u, command.tor.port());
+
+    // make sure that exceeding max buffer doesn't destroy tor_address::_load
+    {
+        epee::serialization::portable_storage stg{};
+        stg.load_from_binary(buffer);
+
+        std::string host{};
+        ASSERT_TRUE(stg.get_value("host", host, stg.open_section("tor", nullptr, false)));
+        EXPECT_EQ(std::strlen(v3_onion), host.size());
+
+        host.push_back('k');
+        EXPECT_TRUE(stg.set_value("host", host, stg.open_section("tor", nullptr, false)));
+        EXPECT_TRUE(command.load(stg)); // poor error reporting from `KV_SERIALIZE`
+    }
+
+    EXPECT_TRUE(command.tor.is_unknown());
+    EXPECT_EQ(net::tor_address{}, command.tor);
+    EXPECT_STRNE(v3_onion, command.tor.host_str());
+    EXPECT_EQ(0u, command.tor.port());
+}
+
+TEST(tor_address, epee_serialize_unknown)
+{
+    std::string buffer{};
+    {
+        test_command command{net::tor_address::unknown()};
+        EXPECT_TRUE(command.tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, command.tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), command.tor.host_str());
+        EXPECT_EQ(0u, command.tor.port());
+
+        epee::serialization::portable_storage stg{};
+        EXPECT_TRUE(command.store(stg));
+        EXPECT_TRUE(stg.store_to_binary(buffer));
+    }
+
+    test_command command{};
+    {
+        EXPECT_TRUE(command.tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, command.tor);
+        EXPECT_STRNE(v3_onion, command.tor.host_str());
+        EXPECT_EQ(0u, command.tor.port());
+
+        epee::serialization::portable_storage stg{};
+        EXPECT_TRUE(stg.load_from_binary(buffer));
+        EXPECT_TRUE(command.load(stg));
+    }
+    EXPECT_TRUE(command.tor.is_unknown());
+    EXPECT_EQ(net::tor_address{}, command.tor);
+    EXPECT_STREQ(net::tor_address::unknown_str(), command.tor.host_str());
+    EXPECT_EQ(0u, command.tor.port());
+
+    // make sure that exceeding max buffer doesn't destroy tor_address::_load
+    {
+        epee::serialization::portable_storage stg{};
+        stg.load_from_binary(buffer);
+
+        std::string host{};
+        ASSERT_TRUE(stg.get_value("host", host, stg.open_section("tor", nullptr, false)));
+        EXPECT_EQ(std::strlen(net::tor_address::unknown_str()), host.size());
+
+        host.push_back('k');
+        EXPECT_TRUE(stg.set_value("host", host, stg.open_section("tor", nullptr, false)));
+        EXPECT_TRUE(command.load(stg)); // poor error reporting from `KV_SERIALIZE`
+    }
+
+    EXPECT_TRUE(command.tor.is_unknown());
+    EXPECT_EQ(net::tor_address{}, command.tor);
+    EXPECT_STRNE(v3_onion, command.tor.host_str());
+    EXPECT_EQ(0u, command.tor.port());
+}
+
+TEST(tor_address, boost_serialize_v2)
+{
+    std::string buffer{};
+    {
+        const net::tor_address tor = MONERO_UNWRAP(net::tor_address::make(v2_onion, 10));
+        EXPECT_FALSE(tor.is_unknown());
+        EXPECT_NE(net::tor_address{}, tor);
+        EXPECT_STREQ(v2_onion, tor.host_str());
+        EXPECT_EQ(10u, tor.port());
+
+        std::ostringstream stream{};
+        {
+            boost::archive::portable_binary_oarchive archive{stream};
+            archive << tor;
+        }
+        buffer = stream.str();
+    }
+
+    net::tor_address tor{};
+    {
+        EXPECT_TRUE(tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), tor.host_str());
+        EXPECT_EQ(0u, tor.port());
+
+        std::istringstream stream{buffer};
+        boost::archive::portable_binary_iarchive archive{stream};
+        archive >> tor;
+    }
+    EXPECT_FALSE(tor.is_unknown());
+    EXPECT_NE(net::tor_address{}, tor);
+    EXPECT_STREQ(v2_onion, tor.host_str());
+    EXPECT_EQ(10u, tor.port());
+}
+
+TEST(tor_address, boost_serialize_v3)
+{
+    std::string buffer{};
+    {
+        const net::tor_address tor = MONERO_UNWRAP(net::tor_address::make(v3_onion, 10));
+        EXPECT_FALSE(tor.is_unknown());
+        EXPECT_NE(net::tor_address{}, tor);
+        EXPECT_STREQ(v3_onion, tor.host_str());
+        EXPECT_EQ(10u, tor.port());
+
+        std::ostringstream stream{};
+        {
+            boost::archive::portable_binary_oarchive archive{stream};
+            archive << tor;
+        }
+        buffer = stream.str();
+    }
+
+    net::tor_address tor{};
+    {
+        EXPECT_TRUE(tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), tor.host_str());
+        EXPECT_EQ(0u, tor.port());
+
+        std::istringstream stream{buffer};
+        boost::archive::portable_binary_iarchive archive{stream};
+        archive >> tor;
+    }
+    EXPECT_FALSE(tor.is_unknown());
+    EXPECT_NE(net::tor_address{}, tor);
+    EXPECT_STREQ(v3_onion, tor.host_str());
+    EXPECT_EQ(10u, tor.port());
+}
+
+TEST(tor_address, boost_serialize_unknown)
+{
+    std::string buffer{};
+    {
+        const net::tor_address tor{};
+        EXPECT_TRUE(tor.is_unknown());
+        EXPECT_EQ(net::tor_address::unknown(), tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), tor.host_str());
+        EXPECT_EQ(0u, tor.port());
+
+        std::ostringstream stream{};
+        {
+            boost::archive::portable_binary_oarchive archive{stream};
+            archive << tor;
+        }
+        buffer = stream.str();
+    }
+
+    net::tor_address tor{};
+    {
+        EXPECT_TRUE(tor.is_unknown());
+        EXPECT_EQ(net::tor_address{}, tor);
+        EXPECT_STREQ(net::tor_address::unknown_str(), tor.host_str());
+        EXPECT_EQ(0u, tor.port());
+
+        std::istringstream stream{buffer};
+        boost::archive::portable_binary_iarchive archive{stream};
+        archive >> tor;
+    }
+    EXPECT_TRUE(tor.is_unknown());
+    EXPECT_EQ(net::tor_address::unknown(), tor);
+    EXPECT_STREQ(net::tor_address::unknown_str(), tor.host_str());
+    EXPECT_EQ(0u, tor.port());
+}
+
+TEST(get_network_address, onion)
+{
+    expect<epee::net_utils::network_address> address =
+        net::get_network_address("onion", 0);
+    EXPECT_EQ(net::error::unsupported_address, address);
+
+    address = net::get_network_address(".onion", 0);
+    EXPECT_EQ(net::error::invalid_tor_address, address);
+
+    address = net::get_network_address(v3_onion, 1000);
+    ASSERT_TRUE(bool(address));
+    EXPECT_EQ(epee::net_utils::address_type::tor, address->get_type_id());
+    EXPECT_STREQ(v3_onion, address->host_str().c_str());
+    EXPECT_EQ(std::string{v3_onion} + ":1000", address->str());
+
+    address = net::get_network_address(std::string{v3_onion} + ":2000", 1000);
+    ASSERT_TRUE(bool(address));
+    EXPECT_EQ(epee::net_utils::address_type::tor, address->get_type_id());
+    EXPECT_STREQ(v3_onion, address->host_str().c_str());
+    EXPECT_EQ(std::string{v3_onion} + ":2000", address->str());
+
+    address = net::get_network_address(std::string{v3_onion} + ":65536", 1000);
+    EXPECT_EQ(net::error::invalid_port, address);
+}
+
+
+TEST(get_network_address, ipv4)
+{
+    expect<epee::net_utils::network_address> address =
+        net::get_network_address("0.0.0.", 0);
+    EXPECT_EQ(net::error::unsupported_address, address);
+
+    address = net::get_network_address("0.0.0.257", 0);
+    EXPECT_EQ(net::error::unsupported_address, address);
+
+    address = net::get_network_address("0.0.0.254", 1000);
+    ASSERT_TRUE(bool(address));
+    EXPECT_EQ(epee::net_utils::address_type::ipv4, address->get_type_id());
+    EXPECT_STREQ("0.0.0.254", address->host_str().c_str());
+    EXPECT_STREQ("0.0.0.254:1000", address->str().c_str());
+
+    address = net::get_network_address("23.0.0.254:2000", 1000);
+    ASSERT_TRUE(bool(address));
+    EXPECT_EQ(epee::net_utils::address_type::ipv4, address->get_type_id());
+    EXPECT_STREQ("23.0.0.254", address->host_str().c_str());
+    EXPECT_STREQ("23.0.0.254:2000", address->str().c_str());
+}
+
+namespace
+{
+    using stream_type = boost::asio::ip::tcp;
+
+    struct io_thread
+    {
+        boost::asio::io_service io_service;
+        boost::asio::io_service::work work;
+        stream_type::socket server;
+        stream_type::acceptor acceptor;
+        boost::thread io;
+        std::atomic<bool> connected;
+
+        io_thread()
+          : io_service(),
+            work(io_service),
+            server(io_service),
+            acceptor(io_service),
+            io([this] () { try { this->io_service.run(); } catch (const std::exception& e) { MERROR(e.what()); }}),
+            connected(false)
+        {
+            acceptor.open(boost::asio::ip::tcp::v4());
+            acceptor.bind(stream_type::endpoint{boost::asio::ip::tcp::v4(), 0});
+            acceptor.listen();
+            acceptor.async_accept(server, [this] (boost::system::error_code error) {
+                this->connected = true;
+                if (error)
+                    throw boost::system::system_error{error};
+            });
+        }
+
+        ~io_thread() noexcept
+        {
+            io_service.stop();
+            if (io.joinable())
+                io.join();
+        }
+    };
+
+    struct checked_client
+    {
+        std::atomic<bool>* called_;
+        bool expected_;
+
+        void operator()(boost::system::error_code error, net::socks::client::stream_type::socket&&) const
+        {
+            EXPECT_EQ(expected_, bool(error)) << "Socks server: " << error.message();
+            ASSERT_TRUE(called_ != nullptr);
+            (*called_) = true;
+        }
+    };
+}
+
+TEST(socks_client, unsupported_command)
+{
+    boost::asio::io_service io_service{};
+    stream_type::socket client{io_service};
+
+    auto test_client = net::socks::make_connect_client(
+        std::move(client), net::socks::version::v4, std::bind( [] {} )
+    );
+    ASSERT_TRUE(bool(test_client));
+    EXPECT_TRUE(test_client->buffer().empty());
+
+    EXPECT_FALSE(test_client->set_connect_command("example.com", 8080));
+    EXPECT_TRUE(test_client->buffer().empty());
+
+    EXPECT_FALSE(test_client->set_resolve_command("example.com"));
+    EXPECT_TRUE(test_client->buffer().empty());
+}
+
+TEST(socks_client, no_command)
+{
+    boost::asio::io_service io_service{};
+    stream_type::socket client{io_service};
+
+    auto test_client = net::socks::make_connect_client(
+        std::move(client), net::socks::version::v4a, std::bind( [] {} )
+    );
+    ASSERT_TRUE(bool(test_client));
+    EXPECT_FALSE(net::socks::client::send(std::move(test_client)));
+}
+
+TEST(socks_client, connect_command)
+{
+    io_thread io{};
+    stream_type::socket client{io.io_service};
+
+    std::atomic<bool> called{false};
+    auto test_client = net::socks::make_connect_client(
+        std::move(client), net::socks::version::v4a, checked_client{std::addressof(called), false}
+    );
+    ASSERT_TRUE(bool(test_client));
+
+    ASSERT_TRUE(test_client->set_connect_command("example.com", 8080));
+    EXPECT_FALSE(test_client->buffer().empty());
+    ASSERT_TRUE(net::socks::client::connect_and_send(std::move(test_client), io.acceptor.local_endpoint()));
+    while (!io.connected);
+
+    const std::uint8_t expected_bytes[] = {
+        4, 1, 0x1f, 0x90, 0x00, 0x00, 0x00, 0x01, 0x00,
+        'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'c', 'o', 'm', 0x00
+    };
+
+    std::uint8_t actual_bytes[sizeof(expected_bytes)];
+    boost::asio::read(io.server, boost::asio::buffer(actual_bytes));
+    EXPECT_TRUE(std::memcmp(expected_bytes, actual_bytes, sizeof(actual_bytes)) == 0);
+
+    const std::uint8_t reply_bytes[] = {0, 90, 0, 0, 0, 0, 0, 0};
+    boost::asio::write(io.server, boost::asio::buffer(reply_bytes));
+
+    // yikes!
+    while (!called);
+}
+
+TEST(socks_client, connect_command_failed)
+{
+    io_thread io{};
+    stream_type::socket client{io.io_service};
+
+    std::atomic<bool> called{false};
+    auto test_client = net::socks::make_connect_client(
+        std::move(client), net::socks::version::v4, checked_client{std::addressof(called), true}
+    );
+    ASSERT_TRUE(bool(test_client));
+
+    ASSERT_TRUE(
+        test_client->set_connect_command(
+            epee::net_utils::ipv4_network_address{boost::endian::native_to_big(std::uint32_t(5000)), 3000}
+        )
+    );
+    EXPECT_FALSE(test_client->buffer().empty());
+    ASSERT_TRUE(net::socks::client::connect_and_send(std::move(test_client), io.acceptor.local_endpoint()));
+    while (!io.connected);
+
+    const std::uint8_t expected_bytes[] = {
+        4, 1, 0x0b, 0xb8, 0x00, 0x00, 0x13, 0x88, 0x00
+    };
+
+    std::uint8_t actual_bytes[sizeof(expected_bytes)];
+    boost::asio::read(io.server, boost::asio::buffer(actual_bytes));
+    EXPECT_TRUE(std::memcmp(expected_bytes, actual_bytes, sizeof(actual_bytes)) == 0);
+
+    const std::uint8_t reply_bytes[] = {0, 91, 0, 0, 0, 0, 0, 0};
+    boost::asio::write(io.server, boost::asio::buffer(reply_bytes));
+
+    // yikes!
+    while (!called);
+}
+
+TEST(socks_client, resolve_command)
+{
+    static std::uint8_t reply_bytes[] = {0, 90, 0, 0, 0xff, 0, 0xad, 0};
+
+    struct resolve_client : net::socks::client
+    {
+        std::atomic<unsigned> called_;
+        bool expected_;
+
+        resolve_client(stream_type::socket&& proxy)
+          : net::socks::client(std::move(proxy), net::socks::version::v4a_tor)
+          , called_(0)
+          , expected_(false)
+        {};
+
+        virtual void done(boost::system::error_code error, std::shared_ptr<client> self) override
+        {
+            EXPECT_EQ(this, self.get());
+            EXPECT_EQ(expected_, bool(error)) << "Resolve failure: " << error.message();
+
+            if (!error)
+            {
+                ASSERT_EQ(sizeof(reply_bytes), buffer().size());
+                EXPECT_EQ(0u, std::memcmp(buffer().data(), reply_bytes, sizeof(reply_bytes)));
+            }
+
+            ++called_;
+        }
+    };
+
+    io_thread io{};
+    stream_type::socket client{io.io_service};
+
+    auto test_client = std::make_shared<resolve_client>(std::move(client));
+    ASSERT_TRUE(bool(test_client));
+
+    ASSERT_TRUE(test_client->set_resolve_command("example.com"));
+    EXPECT_FALSE(test_client->buffer().empty());
+    ASSERT_TRUE(net::socks::client::connect_and_send(test_client, io.acceptor.local_endpoint()));
+    while (!io.connected);
+
+    const std::uint8_t expected_bytes[] = {
+        4, 0xf0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
+        'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'c', 'o', 'm', 0x00
+    };
+
+    std::uint8_t actual_bytes[sizeof(expected_bytes)];
+    boost::asio::read(io.server, boost::asio::buffer(actual_bytes));
+    EXPECT_TRUE(std::memcmp(expected_bytes, actual_bytes, sizeof(actual_bytes)) == 0);
+
+    boost::asio::write(io.server, boost::asio::buffer(reply_bytes));
+
+    // yikes!
+    while (test_client->called_ == 0);
+
+    test_client->expected_ = true;
+    ASSERT_TRUE(test_client->set_resolve_command("example.com"));
+    EXPECT_FALSE(test_client->buffer().empty());
+    ASSERT_TRUE(net::socks::client::send(test_client));
+
+    boost::asio::read(io.server, boost::asio::buffer(actual_bytes));
+    EXPECT_TRUE(std::memcmp(expected_bytes, actual_bytes, sizeof(actual_bytes)) == 0);
+
+    reply_bytes[1] = 91;
+    boost::asio::write(io.server, boost::asio::buffer(reply_bytes));
+
+    // yikes!
+    while (test_client->called_ == 1);
+}
+
+

tests/unit_tests/test_peerlist.cpp

@@ -37,7 +37,7 @@
 TEST(peer_list, peer_list_general)
 {
   nodetool::peerlist_manager plm;
-  plm.init(false);
+  plm.init(nodetool::peerlist_types{}, false);
 #define MAKE_IPV4_ADDRESS(a,b,c,d,e) epee::net_utils::ipv4_network_address{MAKE_IP(a,b,c,d),e}
 #define ADD_GRAY_NODE(addr_, id_, last_seen_) {  nodetool::peerlist_entry ple; ple.last_seen=last_seen_;ple.adr = addr_; ple.id = id_;plm.append_with_peer_gray(ple);}  
 #define ADD_WHITE_NODE(addr_, id_, last_seen_) {  nodetool::peerlist_entry ple;ple.last_seen=last_seen_; ple.adr = addr_; ple.id = id_;plm.append_with_peer_white(ple);}  
@@ -77,9 +77,180 @@ TEST(peer_list, merge_peer_lists)
   //([^ \t]*)\t([^ \t]*):([^ \t]*) \tlast_seen: d(\d+)\.h(\d+)\.m(\d+)\.s(\d+)\n
   //ADD_NODE_TO_PL("\2", \3, 0x\1, (1353346618 -(\4*60*60*24+\5*60*60+\6*60+\7 )));\n
   nodetool::peerlist_manager plm;
-  plm.init(false);
+  plm.init(nodetool::peerlist_types{}, false);
   std::vector<nodetool::peerlist_entry> outer_bs;
 #define ADD_NODE_TO_PL(ip_, port_, id_, timestamp_) {  nodetool::peerlist_entry ple; epee::string_tools::get_ip_int32_from_string(ple.adr.ip, ip_); ple.last_seen = timestamp_; ple.adr.port = port_; ple.id = id_;outer_bs.push_back(ple);}  
-
+}
-
+
+namespace
+{
+  bool check_empty(nodetool::peerlist_storage& peers, std::initializer_list<epee::net_utils::zone> zones)
+  {
+    bool pass = false;
+    for (const epee::net_utils::zone zone : zones)
+    {
+      const nodetool::peerlist_types types{peers.take_zone(zone)};
+      EXPECT_TRUE(types.white.empty());
+      EXPECT_TRUE(types.gray.empty());
+      EXPECT_TRUE(types.anchor.empty());
+      pass = (types.white.empty() && types.gray.empty() && types.anchor.empty());
+    }
+    return pass;
+  }
+}
+
+TEST(peerlist_storage, store)
+{
+
+  using address_type = epee::net_utils::address_type;
+  using zone = epee::net_utils::zone;
+
+  nodetool::peerlist_storage peers{};
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::tor, zone::i2p}));
+
+  std::string buffer{};
+  {
+    nodetool::peerlist_types types{};
+    types.white.push_back({epee::net_utils::ipv4_network_address{1000, 10}, 44, 55});
+    types.white.push_back({net::tor_address::unknown(), 64, 75});
+    types.gray.push_back({net::tor_address::unknown(), 99, 88});
+    types.gray.push_back({epee::net_utils::ipv4_network_address{2000, 20}, 84, 45});
+    types.anchor.push_back({epee::net_utils::ipv4_network_address{999, 654}, 444, 555});
+    types.anchor.push_back({net::tor_address::unknown(), 14, 33});
+    types.anchor.push_back({net::tor_address::unknown(), 24, 22});
+
+    std::ostringstream stream{};
+    EXPECT_TRUE(peers.store(stream, types));
+    buffer = stream.str();
+  }
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::tor, zone::i2p}));
+  {
+    std::istringstream stream{buffer};
+    boost::optional<nodetool::peerlist_storage> read_peers =
+      nodetool::peerlist_storage::open(stream, true);
+    ASSERT_TRUE(bool(read_peers));
+    peers = std::move(*read_peers);
+  }
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::i2p}));
+
+  nodetool::peerlist_types types = peers.take_zone(zone::public_);
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::i2p}));
+
+  ASSERT_EQ(1u, types.white.size());
+  ASSERT_EQ(address_type::ipv4, types.white[0].adr.get_type_id());
+  EXPECT_EQ(1000u, types.white[0].adr.template as<epee::net_utils::ipv4_network_address>().ip());
+  EXPECT_EQ(10u, types.white[0].adr.template as<epee::net_utils::ipv4_network_address>().port());
+  EXPECT_EQ(44u, types.white[0].id);
+  EXPECT_EQ(55u, types.white[0].last_seen);
+
+  ASSERT_EQ(1u, types.gray.size());
+  ASSERT_EQ(address_type::ipv4, types.gray[0].adr.get_type_id());
+  EXPECT_EQ(2000u, types.gray[0].adr.template as<epee::net_utils::ipv4_network_address>().ip());
+  EXPECT_EQ(20u, types.gray[0].adr.template as<epee::net_utils::ipv4_network_address>().port());
+  EXPECT_EQ(84u, types.gray[0].id);
+  EXPECT_EQ(45u, types.gray[0].last_seen);
+
+  ASSERT_EQ(1u, types.anchor.size());
+  ASSERT_EQ(address_type::ipv4, types.anchor[0].adr.get_type_id());
+  EXPECT_EQ(999u, types.anchor[0].adr.template as<epee::net_utils::ipv4_network_address>().ip());
+  EXPECT_EQ(654u, types.anchor[0].adr.template as<epee::net_utils::ipv4_network_address>().port());
+  EXPECT_EQ(444u, types.anchor[0].id);
+  EXPECT_EQ(555u, types.anchor[0].first_seen);
+  {
+    std::ostringstream stream{};
+    EXPECT_TRUE(peers.store(stream, types));
+    buffer = stream.str();
+  }
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::i2p}));
+
+  types = peers.take_zone(zone::tor);
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::i2p, zone::tor}));
+
+  ASSERT_EQ(1u, types.white.size());
+  ASSERT_EQ(address_type::tor, types.white[0].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.white[0].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.white[0].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(64u, types.white[0].id);
+  EXPECT_EQ(75u, types.white[0].last_seen);
+
+  ASSERT_EQ(1u, types.gray.size());
+  ASSERT_EQ(address_type::tor, types.gray[0].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.gray[0].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.gray[0].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(99u, types.gray[0].id);
+  EXPECT_EQ(88u, types.gray[0].last_seen);
+
+  ASSERT_EQ(2u, types.anchor.size());
+  ASSERT_EQ(address_type::tor, types.anchor[0].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.anchor[0].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.anchor[0].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(14u, types.anchor[0].id);
+  EXPECT_EQ(33u, types.anchor[0].first_seen);
+  ASSERT_EQ(address_type::tor, types.anchor[1].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.anchor[1].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.anchor[1].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(24u, types.anchor[1].id);
+  EXPECT_EQ(22u, types.anchor[1].first_seen);
+
+  {
+    std::istringstream stream{buffer};
+    boost::optional<nodetool::peerlist_storage> read_peers =
+      nodetool::peerlist_storage::open(stream, true);
+    ASSERT_TRUE(bool(read_peers));
+    peers = std::move(*read_peers);
+  }
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::i2p}));
+
+  types = peers.take_zone(zone::public_);
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::i2p}));
+
+  ASSERT_EQ(1u, types.white.size());
+  ASSERT_EQ(address_type::ipv4, types.white[0].adr.get_type_id());
+  EXPECT_EQ(1000u, types.white[0].adr.template as<epee::net_utils::ipv4_network_address>().ip());
+  EXPECT_EQ(10u, types.white[0].adr.template as<epee::net_utils::ipv4_network_address>().port());
+  EXPECT_EQ(44u, types.white[0].id);
+  EXPECT_EQ(55u, types.white[0].last_seen);
+
+  ASSERT_EQ(1u, types.gray.size());
+  ASSERT_EQ(address_type::ipv4, types.gray[0].adr.get_type_id());
+  EXPECT_EQ(2000u, types.gray[0].adr.template as<epee::net_utils::ipv4_network_address>().ip());
+  EXPECT_EQ(20u, types.gray[0].adr.template as<epee::net_utils::ipv4_network_address>().port());
+  EXPECT_EQ(84u, types.gray[0].id);
+  EXPECT_EQ(45u, types.gray[0].last_seen);
+
+  ASSERT_EQ(1u, types.anchor.size());
+  ASSERT_EQ(address_type::ipv4, types.anchor[0].adr.get_type_id());
+  EXPECT_EQ(999u, types.anchor[0].adr.template as<epee::net_utils::ipv4_network_address>().ip());
+  EXPECT_EQ(654u, types.anchor[0].adr.template as<epee::net_utils::ipv4_network_address>().port());
+  EXPECT_EQ(444u, types.anchor[0].id);
+  EXPECT_EQ(555u, types.anchor[0].first_seen);
+
+  types = peers.take_zone(zone::tor);
+  EXPECT_TRUE(check_empty(peers, {zone::invalid, zone::public_, zone::i2p, zone::tor}));
+
+  ASSERT_EQ(1u, types.white.size());
+  ASSERT_EQ(address_type::tor, types.white[0].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.white[0].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.white[0].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(64u, types.white[0].id);
+  EXPECT_EQ(75u, types.white[0].last_seen);
+
+  ASSERT_EQ(1u, types.gray.size());
+  ASSERT_EQ(address_type::tor, types.gray[0].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.gray[0].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.gray[0].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(99u, types.gray[0].id);
+  EXPECT_EQ(88u, types.gray[0].last_seen);
+
+  ASSERT_EQ(2u, types.anchor.size());
+  ASSERT_EQ(address_type::tor, types.anchor[0].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.anchor[0].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.anchor[0].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(14u, types.anchor[0].id);
+  EXPECT_EQ(33u, types.anchor[0].first_seen);
+  ASSERT_EQ(address_type::tor, types.anchor[1].adr.get_type_id());
+  EXPECT_STREQ(net::tor_address::unknown_str(), types.anchor[1].adr.template as<net::tor_address>().host_str());
+  EXPECT_EQ(0u, types.anchor[1].adr.template as<net::tor_address>().port());
+  EXPECT_EQ(24u, types.anchor[1].id);
+  EXPECT_EQ(22u, types.anchor[1].first_seen);
 }