Merge pull request #5793

bdfc63a Add ref-counted buffer byte_slice. Currently used for sending TCP data. (vtnerd)
3b24b1d Added support for 'noise' over I1P/Tor to mask Tx transmission. (vtnerd)

ANONYMITY_NETWORKS.md

@@ -160,25 +160,6 @@ the system clock is noticeably off (and therefore more fingerprintable),
 linking the public IPv4/IPv6 connections with the anonymity networks will be
 more difficult.
 
-### Bandwidth Usage
-
-An ISP can passively monitor `monerod` connections from a node and observe when
-a transaction is sent over a Tor/I2P connection via timing analysis + size of
-data sent during that timeframe. I2P should provide better protection against
-this attack - its connections are not circuit based. However, if a node is
-only using I2P for broadcasting Monero transactions, the total aggregate of
-I2P data would also leak information.
-
-#### Mitigation
-
-There is no current mitigation for the user right now. This attack is fairly
-sophisticated, and likely requires support from the internet host of a Monero
-user.
-
-In the near future, "whitening" the amount of data sent over anonymity network
-connections will be performed. An attempt will be made to make a transaction
-broadcast indistinguishable from a peer timed sync command.
-
 ### Intermittent Monero Syncing
 
 If a user only runs `monerod` to send a transaction then quit, this can also
@@ -208,3 +189,36 @@ is a tradeoff in potential isses. Also, anyone attempting this strategy really
 wants to uncover a user, it seems unlikely that this would be performed against
 every Tor/I2P user.
 
+### I2P/Tor Stream Used Twice
+
+If a single I2P/Tor stream is used 2+ times for transmitting a transaction, the
+operator of the hidden service can conclude that both transactions came from the
+same source. If the subsequent transactions spend a change output from the
+earlier transactions, this will also reveal the "real" spend in the ring
+signature. This issue was (primarily) raised by @secparam on Twitter.
+
+#### Mitigation
+
+`monerod` currently selects two outgoing connections every 5 minutes for
+transmitting transactions over I2P/Tor. Using outgoing connections prevents an
+adversary from making many incoming connections to obtain information (this
+technique was taken from Dandelion). Outgoing connections also do not have a
+persistent public key identity - the creation of a new circuit will generate
+a new public key identity. The lock time on a change address is ~20 minutes, so
+`monerod` will have rotated its selected outgoing connections several times in
+most cases. However, the number of outgoing connections is typically a small
+fixed number, so there is a decent probability of re-use with the same public
+key identity.
+
+@secparam (twitter) recommended changing circuits (Tor) as an additional
+precaution. This is likely not a good idea - forcibly requesting Tor to change
+circuits is observable by the ISP. Instead, `monerod` should likely disconnect
+from peers ocassionally. Tor will rotate circuits every ~10 minutes, so
+establishing new connections will use a new public key identity and make it
+more difficult for the hidden service to link information. This process will
+have to be done carefully because closing/reconnecting connections can also
+leak information to hidden services if done improperly.
+
+At the current time, if users need to frequently make transactions, I2P/Tor
+will improve privacy from ISPs and other common adversaries, but still have
+some metadata leakages to unknown hidden service operators.

LEVIN_PROTOCOL.md

@@ -0,0 +1,165 @@
+# Levin Protocol
+This is a document explaining the current design of the levin protocol, as
+used by Monero. The protocol is largely inherited from cryptonote, but has
+undergone some changes.
+
+This document also may differ from the `struct bucket_head2` in Monero's
+code slightly - the spec here is slightly more strict to allow for
+extensibility.
+
+One of the goals of this document is to clearly indicate what is being sent
+"on the wire" to identify metadata that could de-anonymize users over I2P/Tor.
+These issues will be addressed as they are found. See `ANONMITY_NETWORKS.md` in
+the top-level folder for any outstanding issues.
+
+> This document does not currently list all data being sent by the monero
+> protocol, that portion is a work-in-progress. Please take the time to do it
+> if interested in learning about Monero p2p traffic!
+
+
+## Header
+This header is sent for every Monero p2p message.
+
+```
+ 0               1               2               3
+ 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+|      0x01     |      0x21     |      0x01     |      0x01     |
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+|      0x01     |      0x01     |      0x01     |      0x01     |
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+|                             Length                            |
+|                                                               |
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+|  E. Response  |                   Command
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                |                 Return Code
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                |Q|S|B|E|               Reserved
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                |      0x01     |      0x00     |      0x00     |
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+|     0x00      |
++-+-+-+-+-+-+-+-+
+```
+
+### Signature
+The first 8 bytes are the "signature" which helps identify the protocol (in
+case someone connected to the wrong port, etc). The comments indicate that byte
+sequence is from "benders nightmare".
+
+This also can be used by deep packet inspection (DPI) engines to identify
+Monero when the link is not encrypted. SSL has been proposed as a means to
+mitigate this issue, but BIP-151 or the Noise protocol should also be considered.
+
+### Length
+The length is an unsigned 64-bit little endian integer. The length does _not_
+include the header.
+
+The implementation currently rejects received messages that exceed 100 MB
+(base 10) by default.
+
+### Expect Response
+A zero-byte if no response is expected from the peer, and a non-zero byte if a
+response is expected from the peer. Peers must respond to requests with this
+flag in the same order that they were received, however, other messages can be
+sent between responses.
+
+There are some commands in the
+[cryptonote protocol](#cryptonote-protocol-commands) where a response is
+expected from the peer, but this flag is not set. Those responses are returned
+as notify messages and can be sent in any order by the peer.
+
+### Command
+An unsigned 32-bit little endian integer representing the Monero specific
+command being invoked.
+
+### Return Code
+A signed 32-bit little integer integer representing the response from the peer
+from the last command that was invoked. This is `0` for request messages.
+
+### Flags
+ * `Q` - Bit is set if the message is a request.
+ * `S` - Bit is set if the message is a response.
+ * `B` - Bit is set if this is a the beginning of a [fragmented message](#fragmented-messages).
+ * `E` - Bit is set if this is the end of a [fragmented message](#fragmented-messages).
+
+### Version
+A fixed value of `1` as an unsigned 32-bit little endian integer.
+
+
+## Message Flow
+The protocol can be subdivided into: (1) notifications, (2) requests,
+(3) responses, (4) fragmented messages, and (5) dummy messages. Response
+messages must be sent in the same order that a peer issued a request message.
+A peer does not have to send a response immediately following a request - any
+other message type can be sent instead.
+
+### Notifications
+Notifications are one-way messages that can be sent at any time without
+an expectation of a response from the peer. The `Q` bit must be set, the `S`,
+`B` and `E` bits must be unset, and the `Expect Response` field must be zeroed.
+
+Some notifications must be in response to other notifications. This is not
+part of the levin messaging layer, and is described in the
+[commands](#commands) section.
+
+### Requests
+Requests are the basis of the admin protocol for Monero. The `Q` bit must be
+set, the `S`, `B` and `E` bits must be unset, and the `Expect Response` field
+must be non-zero. The peer is expected to send a response message with the same
+`command` number.
+
+### Responses
+Response message can only be sent after a peer first issues a request message.
+Responses must have the `S` bit set, the `Q`, `B` and `E` bits unset, and have
+a zeroed `Expect Response` field. The `Command` field must be the same value
+that was sent in the request message. The `Return Code` is specific to the
+`Command` being issued (see [commands])(#commands)).
+
+### Fragmented
+Fragmented messages were introduced for the "white noise" feature for i2p/tor.
+A transaction can be sent in fragments to conceal when "real" data is being
+sent instead of dummy messages. Only one fragmented message can be sent at a
+time, and bits `B` and `E` are never set at the same time
+(see [dummy messages](#dummy)). The re-constructed message must contain a
+levin header for a different (non-fragment) message type.
+
+The `Q` and `S` bits are never set and the `Expect Response` field must always
+be zero. The first fragment has the `B` bit set, neither `B` nor `E` is set for
+"middle" fragments, and `E` is set for the last fragment.
+
+### Dummy
+Dummy messages have the `B` and `E` bits set, the `Q` and `S` bits unset, and
+the `Expect Reponse` field zeroed. When a message of this type is received, the
+contents can be safely ignored.
+
+
+## Commands
+### P2P (Admin) Commands
+
+#### (`1001` Request) Handshake
+#### (`1001` Response) Handshake
+#### (`1002` Request) Timed Sync
+#### (`1002` Response) Timed Sync
+#### (`1003` Request) Ping
+#### (`1003` Response) Ping
+#### (`1004` Request) Stat Info
+#### (`1004` Response) Stat Info
+#### (`1005` Request) Network State
+#### (`1005` Response) Network State
+#### (`1006` Request) Peer ID
+#### (`1006` Reponse) Peer ID
+#### (`1007` Request) Support Flags
+#### (`1007` Response) Support Flags
+
+### Cryptonote Protocol Commands
+
+#### (`2001` Notification) New Block
+#### (`2002` Notification) New Transactions
+#### (`2003` Notification) Request Get Objects
+#### (`2004` Notification) Response Get Objects
+#### (`2006` Notification) Request Chain
+#### (`2007` Notification) Response Chain Entry
+#### (`2008` Notification) New Fluffy Block
+#### (`2009` Notification) Request Fluffy Missing TX

contrib/epee/include/byte_slice.h

@@ -0,0 +1,145 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "span.h"
+
+namespace epee
+{
+  struct byte_slice_data;
+
+  struct release_byte_slice
+  {
+    void operator()(byte_slice_data*) const noexcept;
+  };
+
+  /*! Inspired by slices in golang. Storage is thread-safe reference counted,
+      allowing for cheap copies or range selection on the bytes. The bytes
+      owned by this class are always immutable.
+
+      The functions `operator=`, `take_slice` and `remove_prefix` may alter the
+      reference count for the backing store, which will invalidate pointers
+      previously returned if the reference count is zero. Be careful about
+      "caching" pointers in these circumstances. */
+  class byte_slice
+  {
+    /* A custom reference count is used instead of shared_ptr because it allows
+       for an allocation optimization for the span constructor. This also
+       reduces the size of this class by one pointer. */
+    std::unique_ptr<byte_slice_data, release_byte_slice> storage_;
+    span<const std::uint8_t> portion_; // within storage_
+
+    //! Internal use only; use to increase `storage` reference count.
+    byte_slice(byte_slice_data* storage, span<const std::uint8_t> portion) noexcept;
+
+    struct adapt_buffer{};
+
+    template<typename T>
+    explicit byte_slice(const adapt_buffer, T&& buffer);
+
+  public:
+    using value_type = std::uint8_t;
+    using size_type = std::size_t;
+    using difference_type = std::ptrdiff_t;
+    using pointer = const std::uint8_t*;
+    using const_pointer = const std::uint8_t*;
+    using reference = std::uint8_t;
+    using const_reference = std::uint8_t;
+    using iterator = pointer;
+    using const_iterator = const_pointer;
+
+    //! Construct empty slice.
+    byte_slice() noexcept
+      : storage_(nullptr), portion_()
+    {}
+
+    //! Construct empty slice
+    byte_slice(std::nullptr_t) noexcept
+      : byte_slice()
+    {}
+
+    //! Scatter-gather (copy) multiple `sources` into a single allocated slice.
+    explicit byte_slice(std::initializer_list<span<const std::uint8_t>> sources);
+
+    //! Convert `buffer` into a slice using one allocation for shared count.
+    explicit byte_slice(std::vector<std::uint8_t>&& buffer);
+
+    //! Convert `buffer` into a slice using one allocation for shared count.
+    explicit byte_slice(std::string&& buffer);
+
+    byte_slice(byte_slice&& source) noexcept;
+    ~byte_slice() noexcept = default;
+
+    //! \note May invalidate previously retrieved pointers.
+    byte_slice& operator=(byte_slice&&) noexcept;
+
+    //! \return A shallow (cheap) copy of the data from `this` slice.
+    byte_slice clone() const noexcept { return {storage_.get(), portion_}; }
+
+    iterator begin() const noexcept { return portion_.begin(); }
+    const_iterator cbegin() const noexcept { return portion_.begin(); }
+
+    iterator end() const noexcept { return portion_.end(); }
+    const_iterator cend() const noexcept { return portion_.end(); }
+
+    bool empty() const noexcept { return storage_ == nullptr; }
+    const std::uint8_t* data() const noexcept { return portion_.data(); }
+    std::size_t size() const noexcept { return portion_.size(); }
+
+    /*! Drop bytes from the beginning of `this` slice.
+
+        \note May invalidate previously retrieved pointers.
+        \post `this->size() = this->size() - std::min(this->size(), max_bytes)`
+        \post `if (this->size() <= max_bytes) this->data() = nullptr`
+        \return Number of bytes removed. */
+    std::size_t remove_prefix(std::size_t max_bytes) noexcept;
+
+    /*! "Take" bytes from the beginning of `this` slice.
+
+        \note May invalidate previously retrieved pointers.
+        \post `this->size() = this->size() - std::min(this->size(), max_bytes)`
+        \post `if (this->size() <= max_bytes) this->data() = nullptr`
+        \return Slice containing the bytes removed from `this` slice. */
+    byte_slice take_slice(std::size_t max_bytes) noexcept;
+
+    /*! Return a shallow (cheap) copy of a slice from `begin` and `end` offsets.
+
+        \throw std::out_of_range If `end < begin`.
+        \throw std::out_of_range If `size() < end`.
+        \return Slice starting at `data() + begin` of size `end - begin`. */
+    byte_slice get_slice(std::size_t begin, std::size_t end) const;
+  };
+} // epee
+

contrib/epee/include/net/abstract_tcp_server2.h

@@ -49,10 +49,12 @@
 #include <boost/asio/ssl.hpp>
 #include <boost/array.hpp>
 #include <boost/noncopyable.hpp>
-#include <boost/shared_ptr.hpp>
+#include <boost/shared_ptr.hpp> //! \TODO Convert to std::shared_ptr
 #include <boost/enable_shared_from_this.hpp>
 #include <boost/interprocess/detail/atomic.hpp>
 #include <boost/thread/thread.hpp>
+#include <memory>
+#include "byte_slice.h"
 #include "net_utils_base.h"
 #include "syncobj.h"
 #include "connection_basic.hpp"
@@ -90,25 +92,24 @@ namespace net_utils
   public:
     typedef typename t_protocol_handler::connection_context t_connection_context;
 
-    struct shared_state : connection_basic_shared_state
+    struct shared_state : connection_basic_shared_state, t_protocol_handler::config_type
     {
       shared_state()
-        : connection_basic_shared_state(), pfilter(nullptr), config(), stop_signal_sent(false)
+        : connection_basic_shared_state(), t_protocol_handler::config_type(), pfilter(nullptr), stop_signal_sent(false)
       {}
 
       i_connection_filter* pfilter;
-      typename t_protocol_handler::config_type config;
       bool stop_signal_sent;
     };
 
     /// Construct a connection with the given io_service.
     explicit connection( boost::asio::io_service& io_service,
-                        boost::shared_ptr<shared_state> state,
+                        std::shared_ptr<shared_state> state,
 			t_connection_type connection_type,
 			epee::net_utils::ssl_support_t ssl_support);
 
     explicit connection( boost::asio::ip::tcp::socket&& sock,
-			 boost::shared_ptr<shared_state> state,
+			 std::shared_ptr<shared_state> state,
 			t_connection_type connection_type,
 			epee::net_utils::ssl_support_t ssl_support);
 
@@ -135,8 +136,7 @@ namespace net_utils
     
   private:
     //----------------- i_service_endpoint ---------------------
-    virtual bool do_send(const void* ptr, size_t cb); ///< (see do_send from i_service_endpoint)
+    virtual bool do_send(byte_slice message); ///< (see do_send from i_service_endpoint)
-    virtual bool do_send_chunk(const void* ptr, size_t cb); ///< will send (or queue) a part of data
     virtual bool send_done();
     virtual bool close();
     virtual bool call_run_once_service_io();
@@ -145,6 +145,8 @@ namespace net_utils
     virtual bool add_ref();
     virtual bool release();
     //------------------------------------------------------
+    bool do_send_chunk(byte_slice chunk); ///< will send (or queue) a part of data. internal use only
+
     boost::shared_ptr<connection<t_protocol_handler> > safe_shared_from_this();
     bool shutdown();
     /// Handle completion of a receive operation.
@@ -269,7 +271,13 @@ namespace net_utils
     typename t_protocol_handler::config_type& get_config_object()
     {
       assert(m_state != nullptr); // always set in constructor
-      return m_state->config;
+      return *m_state;
+    }
+
+    std::shared_ptr<typename t_protocol_handler::config_type> get_config_shared()
+    {
+      assert(m_state != nullptr); // always set in constructor
+      return {m_state};
     }
 
     int get_binded_port(){return m_port;}
@@ -350,7 +358,7 @@ namespace net_utils
 
     bool is_thread_worker();
 
-    const boost::shared_ptr<typename connection<t_protocol_handler>::shared_state> m_state;
+    const std::shared_ptr<typename connection<t_protocol_handler>::shared_state> m_state;
 
     /// The io_service used to perform asynchronous operations.
     struct worker

contrib/epee/include/net/abstract_tcp_server2.inl

@@ -68,7 +68,7 @@ namespace epee
 namespace net_utils
 {
   template<typename T>
-  T& check_and_get(boost::shared_ptr<T>& ptr)
+  T& check_and_get(std::shared_ptr<T>& ptr)
   {
     CHECK_AND_ASSERT_THROW_MES(bool(ptr), "shared_state cannot be null");
     return *ptr;
@@ -81,7 +81,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   connection<t_protocol_handler>::connection( boost::asio::io_service& io_service,
-                boost::shared_ptr<shared_state> state,
+                std::shared_ptr<shared_state> state,
 		t_connection_type connection_type,
 		ssl_support_t ssl_support
 	)
@@ -91,13 +91,13 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   connection<t_protocol_handler>::connection( boost::asio::ip::tcp::socket&& sock,
-                boost::shared_ptr<shared_state> state,
+                std::shared_ptr<shared_state> state,
 		t_connection_type connection_type,
 		ssl_support_t ssl_support
 	)
 	: 
 		connection_basic(std::move(sock), state, ssl_support),
-		m_protocol_handler(this, check_and_get(state).config, context),
+		m_protocol_handler(this, check_and_get(state), context),
 		buffer_ssl_init_fill(0),
 		m_connection_type( connection_type ),
 		m_throttle_speed_in("speed_in", "throttle_speed_in"),
@@ -378,7 +378,6 @@ PRAGMA_WARNING_DISABLE_VS(4355)
       if(!recv_res)
       {  
         //_info("[sock " << socket().native_handle() << "] protocol_want_close");
-
         //some error in protocol, protocol handler ask to close connection
         boost::interprocess::ipcdetail::atomic_write32(&m_want_close_connection, 1);
         bool do_shutdown = false;
@@ -520,7 +519,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
   }
   //---------------------------------------------------------------------------------
     template<class t_protocol_handler>
-  bool connection<t_protocol_handler>::do_send(const void* ptr, size_t cb) {
+  bool connection<t_protocol_handler>::do_send(byte_slice message) {
     TRY_ENTRY();
 
     // Use safe_shared_from_this, because of this is public method and it can be called on the object being deleted
@@ -529,6 +528,9 @@ PRAGMA_WARNING_DISABLE_VS(4355)
     if (m_was_shutdown) return false;
 		// TODO avoid copy
 
+		std::uint8_t const* const message_data = message.data();
+		const std::size_t message_size = message.size();
+
 		const double factor = 32; // TODO config
 		typedef long long signed int t_safe; // my t_size to avoid any overunderflow in arithmetic
 		const t_safe chunksize_good = (t_safe)( 1024 * std::max(1.0,factor) );
@@ -538,13 +540,11 @@ PRAGMA_WARNING_DISABLE_VS(4355)
         CHECK_AND_ASSERT_MES(! (chunksize_max<0), false, "Negative chunksize_max" ); // make sure it is unsigned before removin sign with cast:
         long long unsigned int chunksize_max_unsigned = static_cast<long long unsigned int>( chunksize_max ) ;
 
-        if (allow_split && (cb > chunksize_max_unsigned)) {
+        if (allow_split && (message_size > chunksize_max_unsigned)) {
 			{ // LOCK: chunking
     		epee::critical_region_t<decltype(m_chunking_lock)> send_guard(m_chunking_lock); // *** critical *** 
 
-				MDEBUG("do_send() will SPLIT into small chunks, from packet="<<cb<<" B for ptr="<<ptr);
+				MDEBUG("do_send() will SPLIT into small chunks, from packet="<<message_size<<" B for ptr="<<message_data);
-				t_safe all = cb; // all bytes to send 
-				t_safe pos = 0; // current sending position
 				// 01234567890 
 				// ^^^^        (pos=0, len=4)     ;   pos:=pos+len, pos=4
 				//     ^^^^    (pos=4, len=4)     ;   pos:=pos+len, pos=8
@@ -554,40 +554,25 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 				// char* buf = new char[ bufsize ];
 
 				bool all_ok = true;
-				while (pos < all) {
+				while (!message.empty()) {
-					t_safe lenall = all-pos; // length from here to end
+					byte_slice chunk = message.take_slice(chunksize_good);
-					t_safe len = std::min( chunksize_good , lenall); // take a smaller part
-					CHECK_AND_ASSERT_MES(len<=chunksize_good, false, "len too large");
-					// pos=8; len=4; all=10;	len=3;
 
-                    CHECK_AND_ASSERT_MES(! (len<0), false, "negative len"); // check before we cast away sign:
+					MDEBUG("chunk_start="<<chunk.data()<<" ptr="<<message_data<<" pos="<<(chunk.data() - message_data));
-                    unsigned long long int len_unsigned = static_cast<long long int>( len );
+					MDEBUG("part of " << message.size() << ": pos="<<(chunk.data() - message_data) << " len="<<chunk.size());
-                    CHECK_AND_ASSERT_MES(len>0, false, "len not strictly positive"); // (redundant)
-                    CHECK_AND_ASSERT_MES(len_unsigned < std::numeric_limits<size_t>::max(), false, "Invalid len_unsigned");   // yeap we want strong < then max size, to be sure
 
-					void *chunk_start = ((char*)ptr) + pos;
+					bool ok = do_send_chunk(std::move(chunk)); // <====== ***
-					MDEBUG("chunk_start="<<chunk_start<<" ptr="<<ptr<<" pos="<<pos);
-					CHECK_AND_ASSERT_MES(chunk_start >= ptr, false, "Pointer wraparound"); // not wrapped around address?
-					//std::memcpy( (void*)buf, chunk_start, len);
-
-					MDEBUG("part of " << lenall << ": pos="<<pos << " len="<<len);
-
-					bool ok = do_send_chunk(chunk_start, len); // <====== ***
 
 					all_ok = all_ok && ok;
 					if (!all_ok) {
-						MDEBUG("do_send() DONE ***FAILED*** from packet="<<cb<<" B for ptr="<<ptr);
+						MDEBUG("do_send() DONE ***FAILED*** from packet="<<message_size<<" B for ptr="<<message_data);
 						MDEBUG("do_send() SEND was aborted in middle of big package - this is mostly harmless "
-							<< " (e.g. peer closed connection) but if it causes trouble tell us at #monero-dev. " << cb);
+							<< " (e.g. peer closed connection) but if it causes trouble tell us at #monero-dev. " << message_size);
 						return false; // partial failure in sending
 					}
-					pos = pos+len;
-					CHECK_AND_ASSERT_MES(pos >0, false, "pos <= 0");
-
 					// (in catch block, or uniq pointer) delete buf;
 				} // each chunk
 
-				MDEBUG("do_send() DONE SPLIT from packet="<<cb<<" B for ptr="<<ptr);
+				MDEBUG("do_send() DONE SPLIT from packet="<<message_size<<" B for ptr="<<message_data);
 
                 MDEBUG("do_send() m_connection_type = " << m_connection_type);
 
@@ -595,7 +580,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 			} // LOCK: chunking
 		} // a big block (to be chunked) - all chunks
 		else { // small block
-			return do_send_chunk(ptr,cb); // just send as 1 big chunk
+			return do_send_chunk(std::move(message)); // just send as 1 big chunk
 		}
 
     CATCH_ENTRY_L0("connection<t_protocol_handler>::do_send", false);
@@ -603,7 +588,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   //---------------------------------------------------------------------------------
   template<class t_protocol_handler>
-  bool connection<t_protocol_handler>::do_send_chunk(const void* ptr, size_t cb)
+  bool connection<t_protocol_handler>::do_send_chunk(byte_slice chunk)
   {
     TRY_ENTRY();
     // Use safe_shared_from_this, because of this is public method and it can be called on the object being deleted
@@ -615,7 +600,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
     double current_speed_up;
     {
 		CRITICAL_REGION_LOCAL(m_throttle_speed_out_mutex);
-		m_throttle_speed_out.handle_trafic_exact(cb);
+		m_throttle_speed_out.handle_trafic_exact(chunk.size());
 		current_speed_up = m_throttle_speed_out.get_current_speed();
 	}
     context.m_current_speed_up = current_speed_up;
@@ -623,7 +608,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
     //_info("[sock " << socket().native_handle() << "] SEND " << cb);
     context.m_last_send = time(NULL);
-    context.m_send_cnt += cb;
+    context.m_send_cnt += chunk.size();
     //some data should be wrote to stream
     //request complete
     
@@ -644,7 +629,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
         }*/
 
         long int ms = 250 + (rand()%50);
-        MDEBUG("Sleeping because QUEUE is FULL, in " << __FUNCTION__ << " for " << ms << " ms before packet_size="<<cb); // XXX debug sleep
+        MDEBUG("Sleeping because QUEUE is FULL, in " << __FUNCTION__ << " for " << ms << " ms before packet_size="<<chunk.size()); // XXX debug sleep
         m_send_que_lock.unlock();
         boost::this_thread::sleep(boost::posix_time::milliseconds( ms ) );
         m_send_que_lock.lock();
@@ -657,12 +642,11 @@ PRAGMA_WARNING_DISABLE_VS(4355)
         }
     }
 
-    m_send_que.resize(m_send_que.size()+1);
+    m_send_que.push_back(std::move(chunk));
-    m_send_que.back().assign((const char*)ptr, cb);
 
     if(m_send_que.size() > 1)
     { // active operation should be in progress, nothing to do, just wait last operation callback
-        auto size_now = cb;
+        auto size_now = m_send_que.back().size();
         MDEBUG("do_send_chunk() NOW just queues: packet="<<size_now<<" B, is added to queue-size="<<m_send_que.size());
         //do_send_handler_delayed( ptr , size_now ); // (((H))) // empty function
       
@@ -680,7 +664,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
         auto size_now = m_send_que.front().size();
         MDEBUG("do_send_chunk() NOW SENSD: packet="<<size_now<<" B");
         if (speed_limit_is_enabled())
-			do_send_handler_write( ptr , size_now ); // (((H)))
+			do_send_handler_write( m_send_que.back().data(), m_send_que.back().size() ); // (((H)))
 
         CHECK_AND_ASSERT_MES( size_now == m_send_que.front().size(), false, "Unexpected queue size");
         reset_timer(get_default_timeout(), false);
@@ -908,7 +892,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   boosted_tcp_server<t_protocol_handler>::boosted_tcp_server( t_connection_type connection_type ) :
-    m_state(boost::make_shared<typename connection<t_protocol_handler>::shared_state>()),
+    m_state(std::make_shared<typename connection<t_protocol_handler>::shared_state>()),
     m_io_service_local_instance(new worker()),
     io_service_(m_io_service_local_instance->io_service),
     acceptor_(io_service_),
@@ -927,7 +911,7 @@ PRAGMA_WARNING_DISABLE_VS(4355)
 
   template<class t_protocol_handler>
   boosted_tcp_server<t_protocol_handler>::boosted_tcp_server(boost::asio::io_service& extarnal_io_service, t_connection_type connection_type) :
-    m_state(boost::make_shared<typename connection<t_protocol_handler>::shared_state>()),
+    m_state(std::make_shared<typename connection<t_protocol_handler>::shared_state>()),
     io_service_(extarnal_io_service),
     acceptor_(io_service_),
     acceptor_ipv6(io_service_),

contrib/epee/include/net/connection_basic.hpp

@@ -49,6 +49,7 @@
 #include <boost/asio.hpp>
 #include <boost/asio/ssl.hpp>
 
+#include "byte_slice.h"
 #include "net/net_utils_base.h"
 #include "net/net_ssl.h"
 #include "syncobj.h"
@@ -99,7 +100,7 @@ class connection_basic_pimpl; // PIMPL for this class
 
 class connection_basic { // not-templated base class for rapid developmet of some code parts
 		// beware of removing const, net_utils::connection is sketchily doing a cast to prevent storing ptr twice
-		const boost::shared_ptr<connection_basic_shared_state> m_state;
+		const std::shared_ptr<connection_basic_shared_state> m_state;
 	public:
 
 		std::unique_ptr< connection_basic_pimpl > mI; // my Implementation
@@ -108,7 +109,7 @@ class connection_basic { // not-templated base class for rapid developmet of som
     volatile uint32_t m_want_close_connection;
     std::atomic<bool> m_was_shutdown;
     critical_section m_send_que_lock;
-    std::list<std::string> m_send_que;
+    std::deque<byte_slice> m_send_que;
     volatile bool m_is_multithreaded;
     /// Strand to ensure the connection's handlers are not called concurrently.
     boost::asio::io_service::strand strand_;
@@ -118,8 +119,8 @@ class connection_basic { // not-templated base class for rapid developmet of som
 
 	public:
 		// first counter is the ++/-- count of current sockets, the other socket_number is only-increasing ++ number generator
-		connection_basic(boost::asio::ip::tcp::socket&& socket, boost::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support);
+		connection_basic(boost::asio::ip::tcp::socket&& socket, std::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support);
-		connection_basic(boost::asio::io_service &io_service, boost::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support);
+		connection_basic(boost::asio::io_service &io_service, std::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support);
 
 		virtual ~connection_basic() noexcept(false);
 

contrib/epee/include/net/enums.h

@@ -49,7 +49,7 @@ namespace net_utils
 	{
 		invalid = 0,
 		public_ = 1, // public is keyword
-		i2p = 2,
+		i2p = 2,     // order from here changes priority of selection for origin TXes
 		tor = 3
 	};
 

contrib/epee/include/net/http_protocol_handler.inl

@@ -593,9 +593,10 @@ namespace net_utils
 
 		LOG_PRINT_L3("HTTP_RESPONSE_HEAD: << \r\n" << response_data);
 
-		m_psnd_hndlr->do_send((void*)response_data.data(), response_data.size());
 		if ((response.m_body.size() && (query_info.m_http_method != http::http_method_head)) || (query_info.m_http_method == http::http_method_options))
-			m_psnd_hndlr->do_send((void*)response.m_body.data(), response.m_body.size());
+			response_data += response.m_body;
+
+		m_psnd_hndlr->do_send(byte_slice{std::move(response_data)});
 		m_psnd_hndlr->send_done();
 		return res;
 	}

contrib/epee/include/net/levin_base.h

@@ -29,7 +29,11 @@
 #ifndef _LEVIN_BASE_H_
 #define _LEVIN_BASE_H_
 
+#include <cstdint>
+
+#include "byte_slice.h"
 #include "net_utils_base.h"
+#include "span.h"
 
 #define LEVIN_SIGNATURE  0x0101010101012101LL  //Bender's nightmare
 
@@ -72,6 +76,8 @@ namespace levin
 
 #define LEVIN_PACKET_REQUEST			0x00000001
 #define LEVIN_PACKET_RESPONSE		0x00000002
+#define LEVIN_PACKET_BEGIN		0x00000004
+#define LEVIN_PACKET_END		0x00000008
   
 
 #define LEVIN_PROTOCOL_VER_0         0
@@ -118,9 +124,30 @@ namespace levin
     }
   }
 
+  //! \return Intialized levin header.
+  bucket_head2 make_header(uint32_t command, uint64_t msg_size, uint32_t flags, bool expect_response) noexcept;
 
+  //! \return A levin notification message.
+  byte_slice make_notify(int command, epee::span<const std::uint8_t> payload);
+
+  /*! Generate a dummy levin message.
+
+      \param noise_bytes Total size of the returned `byte_slice`.
+      \return `nullptr` if `noise_size` is smaller than the levin header.
+        Otherwise, a dummy levin message. */
+  byte_slice make_noise_notify(std::size_t noise_bytes);
+
+  /*! Generate 1+ levin messages that are identical to the noise message size.
+
+   \param noise Each levin message will be identical to the size of this
+      message. The bytes from this message will be used for padding.
+   \return `nullptr` if `noise.size()` is less than the levin header size.
+      Otherwise, a levin notification message OR 2+ levin fragment messages.
+      Each message is `noise.size()` in length. */
+  byte_slice make_fragmented_notify(const byte_slice& noise, int command, epee::span<const std::uint8_t> payload);
 }
 }
 
 
 #endif //_LEVIN_BASE_H_
+

contrib/epee/include/net/levin_protocol_handler.h

@@ -157,10 +157,9 @@ namespace levin
 						m_current_head.m_return_code = m_config.m_pcommands_handler->invoke(m_current_head.m_command, buff_to_invoke, return_buff, m_conn_context);
 						m_current_head.m_cb = return_buff.size();
 						m_current_head.m_have_to_return_data = false;
-						std::string send_buff((const char*)&m_current_head, sizeof(m_current_head));
-						send_buff += return_buff;
 
-						if(!m_psnd_hndlr->do_send(send_buff.data(), send_buff.size()))
+						return_buff.insert(0, (const char*)&m_current_head, sizeof(m_current_head));
+						if(!m_psnd_hndlr->do_send(byte_slice{std::move(return_buff)}))
 							return false;
 
 					}

contrib/epee/include/net/levin_protocol_handler_async.h

@@ -32,6 +32,7 @@
 #include <boost/smart_ptr/make_shared.hpp>
 
 #include <atomic>
+#include <deque>
 
 #include "levin_base.h"
 #include "buffer.h"
@@ -91,6 +92,7 @@ public:
   int invoke_async(int command, const epee::span<const uint8_t> in_buff, boost::uuids::uuid connection_id, const callback_t &cb, size_t timeout = LEVIN_DEFAULT_TIMEOUT_PRECONFIGURED);
 
   int notify(int command, const epee::span<const uint8_t> in_buff, boost::uuids::uuid connection_id);
+  int send(epee::byte_slice message, const boost::uuids::uuid& connection_id);
   bool close(boost::uuids::uuid connection_id);
   bool update_connection_context(const t_connection_context& contxt);
   bool request_callback(boost::uuids::uuid connection_id);
@@ -117,6 +119,22 @@ public:
 template<class t_connection_context = net_utils::connection_context_base>
 class async_protocol_handler
 {
+  std::string m_fragment_buffer;
+
+  bool send_message(uint32_t command, epee::span<const uint8_t> in_buff, uint32_t flags, bool expect_response)
+  {
+    const bucket_head2 head = make_header(command, in_buff.size(), flags, expect_response);
+    if(!m_pservice_endpoint->do_send(byte_slice{as_byte_span(head), in_buff}))
+      return false;
+
+    MDEBUG(m_connection_context << "LEVIN_PACKET_SENT. [len=" << head.m_cb
+        << ", flags" << head.m_flags
+        << ", r?=" << head.m_have_to_return_data
+        <<", cmd = " << head.m_command
+        << ", ver=" << head.m_protocol_version);
+    return true;
+  }
+
 public:
   typedef t_connection_context connection_context;
   typedef async_protocol_handler_config<t_connection_context> config_type;
@@ -136,7 +154,6 @@ public:
   critical_section m_local_inv_buff_lock;
   std::string m_local_inv_buff;
 
-  critical_section m_send_lock;
   critical_section m_call_lock;
 
   volatile uint32_t m_wait_count;
@@ -376,7 +393,12 @@ public:
       return false;
     }
 
-    if(m_cache_in_buffer.size() +  cb > m_config.m_max_packet_size)
+    // these should never fail, but do runtime check for safety
+    CHECK_AND_ASSERT_MES(m_config.m_max_packet_size >= m_cache_in_buffer.size(), false, "Bad m_cache_in_buffer.size()");
+    CHECK_AND_ASSERT_MES(m_config.m_max_packet_size - m_cache_in_buffer.size() >= m_fragment_buffer.size(), false, "Bad m_cache_in_buffer.size() + m_fragment_buffer.size()");
+
+    // flipped to subtraction; prevent overflow since m_max_packet_size is variable and public
+    if(cb > m_config.m_max_packet_size - m_cache_in_buffer.size() - m_fragment_buffer.size())
     {
       MWARNING(m_connection_context << "Maximum packet size exceed!, m_max_packet_size = " << m_config.m_max_packet_size
                           << ", packet received " << m_cache_in_buffer.size() +  cb 
@@ -408,8 +430,38 @@ public:
           }
           break;
         }
+
         {
+          std::string temp{};
           epee::span<const uint8_t> buff_to_invoke = m_cache_in_buffer.carve((std::string::size_type)m_current_head.m_cb);
+          m_state = stream_state_head;
+
+          // abstract_tcp_server2.h manages max bandwidth for a p2p link
+          if (!(m_current_head.m_flags & (LEVIN_PACKET_REQUEST | LEVIN_PACKET_RESPONSE)))
+          {
+            // special noise/fragment command
+            static constexpr const uint32_t both_flags = (LEVIN_PACKET_BEGIN | LEVIN_PACKET_END);
+            if ((m_current_head.m_flags & both_flags) == both_flags)
+              break; // noise message, skip to next message
+
+            if (m_current_head.m_flags & LEVIN_PACKET_BEGIN)
+              m_fragment_buffer.clear();
+
+            m_fragment_buffer.append(reinterpret_cast<const char*>(buff_to_invoke.data()), buff_to_invoke.size());
+            if (!(m_current_head.m_flags & LEVIN_PACKET_END))
+              break; // skip to next message
+
+            if (m_fragment_buffer.size() < sizeof(bucket_head2))
+            {
+              MERROR(m_connection_context << "Fragmented data too small for levin header");
+              return false;
+            }
+
+            temp = std::move(m_fragment_buffer);
+            m_fragment_buffer.clear();
+            std::memcpy(std::addressof(m_current_head), std::addressof(temp[0]), sizeof(bucket_head2));
+            buff_to_invoke = {reinterpret_cast<const uint8_t*>(temp.data()) + sizeof(bucket_head2), temp.size() - sizeof(bucket_head2)};
+          }
 
           bool is_response = (m_oponent_protocol_ver == LEVIN_PROTOCOL_VER_1 && m_current_head.m_flags&LEVIN_PACKET_RESPONSE);
 
@@ -458,43 +510,33 @@ public:
             if(m_current_head.m_have_to_return_data)
             {
               std::string return_buff;
-              m_current_head.m_return_code = m_config.m_pcommands_handler->invoke(
+              const uint32_t return_code = m_config.m_pcommands_handler->invoke(
-                                                                  m_current_head.m_command, 
+                m_current_head.m_command, buff_to_invoke, return_buff, m_connection_context
-                                                                  buff_to_invoke, 
+              );
-                                                                  return_buff, 
+
-                                                                  m_connection_context);
+              bucket_head2 head = make_header(m_current_head.m_command, return_buff.size(), LEVIN_PACKET_RESPONSE, false);
-              m_current_head.m_cb = return_buff.size();
+              head.m_return_code = SWAP32LE(return_code);
-              m_current_head.m_have_to_return_data = false;
+              return_buff.insert(0, reinterpret_cast<const char*>(&head), sizeof(head));
-              m_current_head.m_protocol_version = LEVIN_PROTOCOL_VER_1;
+
-              m_current_head.m_flags = LEVIN_PACKET_RESPONSE;
+              if(!m_pservice_endpoint->do_send(byte_slice{std::move(return_buff)}))
-#if BYTE_ORDER == LITTLE_ENDIAN
-              std::string send_buff((const char*)&m_current_head, sizeof(m_current_head));
-#else
-              bucket_head2 head = m_current_head;
-              head.m_signature = SWAP64LE(head.m_signature);
-              head.m_cb = SWAP64LE(head.m_cb);
-              head.m_command = SWAP32LE(head.m_command);
-              head.m_return_code = SWAP32LE(head.m_return_code);
-              head.m_flags = SWAP32LE(head.m_flags);
-              head.m_protocol_version = SWAP32LE(head.m_protocol_version);
-              std::string send_buff((const char*)&head, sizeof(head));
-#endif
-              send_buff += return_buff;
-              CRITICAL_REGION_BEGIN(m_send_lock);
-              if(!m_pservice_endpoint->do_send(send_buff.data(), send_buff.size()))
                 return false;
-              CRITICAL_REGION_END();
+
-              MDEBUG(m_connection_context << "LEVIN_PACKET_SENT. [len=" << m_current_head.m_cb
+              MDEBUG(m_connection_context << "LEVIN_PACKET_SENT. [len=" << head.m_cb
-                << ", flags" << m_current_head.m_flags 
+                << ", flags" << head.m_flags
-                << ", r?=" << m_current_head.m_have_to_return_data 
+                << ", r?=" << head.m_have_to_return_data
-                <<", cmd = " << m_current_head.m_command 
+                <<", cmd = " << head.m_command
-                << ", ver=" << m_current_head.m_protocol_version);
+                << ", ver=" << head.m_protocol_version);
             }
             else
               m_config.m_pcommands_handler->notify(m_current_head.m_command, buff_to_invoke, m_connection_context);
           }
+          // reuse small buffer
+          if (!temp.empty() && temp.capacity() <= 64 * 1024)
+          {
+            temp.clear();
+            m_fragment_buffer = std::move(temp);
+          }
         }
-        m_state = stream_state_head;
         break;
       case stream_state_head:
         {
@@ -584,26 +626,10 @@ public:
         break;
       }
 
-      bucket_head2 head = {0};
-      head.m_signature = SWAP64LE(LEVIN_SIGNATURE);
-      head.m_cb = SWAP64LE(in_buff.size());
-      head.m_have_to_return_data = true;
-
-      head.m_flags = SWAP32LE(LEVIN_PACKET_REQUEST);
-      head.m_command = SWAP32LE(command);
-      head.m_protocol_version = SWAP32LE(LEVIN_PROTOCOL_VER_1);
-
       boost::interprocess::ipcdetail::atomic_write32(&m_invoke_buf_ready, 0);
-      CRITICAL_REGION_BEGIN(m_send_lock);
+      CRITICAL_REGION_BEGIN(m_invoke_response_handlers_lock);
-      CRITICAL_REGION_LOCAL1(m_invoke_response_handlers_lock);
-      if(!m_pservice_endpoint->do_send(&head, sizeof(head)))
-      {
-        LOG_ERROR_CC(m_connection_context, "Failed to do_send");
-        err_code = LEVIN_ERROR_CONNECTION;
-        break;
-      }
 
-      if(!m_pservice_endpoint->do_send(in_buff.data(), in_buff.size()))
+      if(!send_message(command, in_buff, LEVIN_PACKET_REQUEST, true))
       {
         LOG_ERROR_CC(m_connection_context, "Failed to do_send");
         err_code = LEVIN_ERROR_CONNECTION;
@@ -620,7 +646,7 @@ public:
 
     if (LEVIN_OK != err_code)
     {
-      epee::span<const uint8_t> stub_buff{(const uint8_t*)"", 0};
+      epee::span<const uint8_t> stub_buff = nullptr;
       // Never call callback inside critical section, that can cause deadlock
       cb(err_code, stub_buff, m_connection_context);
       return false;
@@ -642,36 +668,14 @@ public:
     if(m_deletion_initiated)
       return LEVIN_ERROR_CONNECTION_DESTROYED;
 
-    bucket_head2 head = {0};
-    head.m_signature = SWAP64LE(LEVIN_SIGNATURE);
-    head.m_cb = SWAP64LE(in_buff.size());
-    head.m_have_to_return_data = true;
-
-    head.m_flags = SWAP32LE(LEVIN_PACKET_REQUEST);
-    head.m_command = SWAP32LE(command);
-    head.m_protocol_version = SWAP32LE(LEVIN_PROTOCOL_VER_1);
-
     boost::interprocess::ipcdetail::atomic_write32(&m_invoke_buf_ready, 0);
-    CRITICAL_REGION_BEGIN(m_send_lock);
+
-    if(!m_pservice_endpoint->do_send(&head, sizeof(head)))
+    if (!send_message(command, in_buff, LEVIN_PACKET_REQUEST, true))
     {
-      LOG_ERROR_CC(m_connection_context, "Failed to do_send");
+      LOG_ERROR_CC(m_connection_context, "Failed to send request");
       return LEVIN_ERROR_CONNECTION;
     }
 
-    if(!m_pservice_endpoint->do_send(in_buff.data(), in_buff.size()))
-    {
-      LOG_ERROR_CC(m_connection_context, "Failed to do_send");
-      return LEVIN_ERROR_CONNECTION;
-    }
-    CRITICAL_REGION_END();
-
-    MDEBUG(m_connection_context << "LEVIN_PACKET_SENT. [len=" << head.m_cb
-                            << ", f=" << head.m_flags 
-                            << ", r?=" << head.m_have_to_return_data 
-                            << ", cmd = " << head.m_command 
-                            << ", ver=" << head.m_protocol_version);
-
     uint64_t ticks_start = misc_utils::get_tick_count();
     size_t prev_size = 0;
 
@@ -716,33 +720,38 @@ public:
     if(m_deletion_initiated)
       return LEVIN_ERROR_CONNECTION_DESTROYED;
 
-    bucket_head2 head = {0};
+    if (!send_message(command, in_buff, LEVIN_PACKET_REQUEST, false))
-    head.m_signature = SWAP64LE(LEVIN_SIGNATURE);
-    head.m_have_to_return_data = false;
-    head.m_cb = SWAP64LE(in_buff.size());
-
-    head.m_command = SWAP32LE(command);
-    head.m_protocol_version = SWAP32LE(LEVIN_PROTOCOL_VER_1);
-    head.m_flags = SWAP32LE(LEVIN_PACKET_REQUEST);
-    CRITICAL_REGION_BEGIN(m_send_lock);
-    if(!m_pservice_endpoint->do_send(&head, sizeof(head)))
     {
-      LOG_ERROR_CC(m_connection_context, "Failed to do_send()");
+      LOG_ERROR_CC(m_connection_context, "Failed to send notify message");
       return -1;
     }
 
-    if(!m_pservice_endpoint->do_send(in_buff.data(), in_buff.size()))
+    return 1;
+  }
+
+  /*! Sends `message` without adding a levin header. The message must have
+      been created with `make_notify`, `make_noise_notify` or
+      `make_fragmented_notify`. See additional instructions for
+      `make_fragmented_notify`.
+
+      \return 1 on success */
+  int send(byte_slice message)
   {
-      LOG_ERROR_CC(m_connection_context, "Failed to do_send()");
+    const misc_utils::auto_scope_leave_caller scope_exit_handler = misc_utils::create_scope_leave_handler(
+      boost::bind(&async_protocol_handler::finish_outer_call, this)
+    );
+
+    if(m_deletion_initiated)
+      return LEVIN_ERROR_CONNECTION_DESTROYED;
+
+    const std::size_t length = message.size();
+    if (!m_pservice_endpoint->do_send(std::move(message)))
+    {
+      LOG_ERROR_CC(m_connection_context, "Failed to send message, dropping it");
       return -1;
     }
-    CRITICAL_REGION_END();
-    LOG_DEBUG_CC(m_connection_context, "LEVIN_PACKET_SENT. [len=" << head.m_cb <<
-      ", f=" << head.m_flags << 
-      ", r?=" << head.m_have_to_return_data <<
-      ", cmd = " << head.m_command << 
-      ", ver=" << head.m_protocol_version);
 
+    MDEBUG(m_connection_context << "LEVIN_PACKET_SENT. [len=" << (length - sizeof(bucket_head2)) << ", r?=0]");
     return 1;
   }
   //------------------------------------------------------------------------------------------
@@ -923,6 +932,14 @@ int async_protocol_handler_config<t_connection_context>::notify(int command, con
 }
 //------------------------------------------------------------------------------------------
 template<class t_connection_context>
+int async_protocol_handler_config<t_connection_context>::send(byte_slice message, const boost::uuids::uuid& connection_id)
+{
+  async_protocol_handler<t_connection_context>* aph;
+  int r = find_and_lock_connection(connection_id, aph);
+  return LEVIN_OK == r ? aph->send(std::move(message)) : 0;
+}
+//------------------------------------------------------------------------------------------
+template<class t_connection_context>
 bool async_protocol_handler_config<t_connection_context>::close(boost::uuids::uuid connection_id)
 {
   CRITICAL_REGION_LOCAL(m_connects_lock);

contrib/epee/include/net/net_helper.h

@@ -31,6 +31,7 @@
 
 //#include <Winsock2.h>
 //#include <Ws2tcpip.h>
+#include <atomic>
 #include <string>
 #include <boost/version.hpp>
 #include <boost/asio/io_service.hpp>
@@ -154,7 +155,7 @@ namespace net_utils
     }
 
     inline
-			try_connect_result_t try_connect(const std::string& addr, const std::string& port, std::chrono::milliseconds timeout, epee::net_utils::ssl_support_t ssl_support)
+			try_connect_result_t try_connect(const std::string& addr, const std::string& port, std::chrono::milliseconds timeout)
 		{
 				m_deadline.expires_from_now(timeout);
 				boost::unique_future<boost::asio::ip::tcp::socket> connection = m_connector(addr, port, m_deadline);
@@ -174,11 +175,11 @@ namespace net_utils
 					m_connected = true;
 					m_deadline.expires_at(std::chrono::steady_clock::time_point::max());
 					// SSL Options
-					if (ssl_support == epee::net_utils::ssl_support_t::e_ssl_support_enabled || ssl_support == epee::net_utils::ssl_support_t::e_ssl_support_autodetect)
+					if (m_ssl_options.support == epee::net_utils::ssl_support_t::e_ssl_support_enabled || m_ssl_options.support == epee::net_utils::ssl_support_t::e_ssl_support_autodetect)
 					{
 						if (!m_ssl_options.handshake(*m_ssl_socket, boost::asio::ssl::stream_base::client, addr))
 						{
-							if (ssl_support == epee::net_utils::ssl_support_t::e_ssl_support_autodetect)
+							if (m_ssl_options.support == epee::net_utils::ssl_support_t::e_ssl_support_autodetect)
 							{
 								boost::system::error_code ignored_ec;
 								m_ssl_socket->next_layer().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ignored_ec);
@@ -217,7 +218,7 @@ namespace net_utils
 
 				// Get a list of endpoints corresponding to the server name.
 
-				try_connect_result_t try_connect_result = try_connect(addr, port, timeout, m_ssl_options.support);
+				try_connect_result_t try_connect_result = try_connect(addr, port, timeout);
 				if (try_connect_result == CONNECT_FAILURE)
 					return false;
 				if (m_ssl_options.support == epee::net_utils::ssl_support_t::e_ssl_support_autodetect)
@@ -226,7 +227,7 @@ namespace net_utils
 					{
 						MERROR("SSL handshake failed on an autodetect connection, reconnecting without SSL");
 						m_ssl_options.support = epee::net_utils::ssl_support_t::e_ssl_support_disabled;
-						if (try_connect(addr, port, timeout, m_ssl_options.support) != CONNECT_SUCCESS)
+						if (try_connect(addr, port, timeout) != CONNECT_SUCCESS)
 							return false;
 					}
 				}
@@ -562,7 +563,7 @@ namespace net_utils
 		{
 			m_deadline.cancel();
 			boost::system::error_code ec;
-			if(m_ssl_options.support != ssl_support_t::e_ssl_support_disabled)
+			if(m_ssl_options)
 				shutdown_ssl();
 			m_ssl_socket->next_layer().cancel(ec);
 			if(ec)

contrib/epee/include/net/net_utils_base.h

@@ -34,9 +34,10 @@
 #include <boost/asio/ip/address_v6.hpp>
 #include <typeinfo>
 #include <type_traits>
+#include "byte_slice.h"
 #include "enums.h"
-#include "serialization/keyvalue_serialization.h"
 #include "misc_log_ex.h"
+#include "serialization/keyvalue_serialization.h"
 
 #undef MONERO_DEFAULT_LOG_CATEGORY
 #define MONERO_DEFAULT_LOG_CATEGORY "net"
@@ -424,7 +425,7 @@ namespace net_utils
 	/************************************************************************/
 	struct i_service_endpoint
 	{
-		virtual bool do_send(const void* ptr, size_t cb)=0;
+		virtual bool do_send(byte_slice message)=0;
     virtual bool close()=0;
     virtual bool send_done()=0;
     virtual bool call_run_once_service_io()=0;

contrib/epee/src/CMakeLists.txt

@@ -26,8 +26,8 @@
 # STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
 # THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-add_library(epee STATIC hex.cpp http_auth.cpp mlog.cpp net_helper.cpp net_utils_base.cpp string_tools.cpp wipeable_string.cpp memwipe.c
+add_library(epee STATIC byte_slice.cpp hex.cpp http_auth.cpp mlog.cpp net_helper.cpp net_utils_base.cpp string_tools.cpp wipeable_string.cpp
-    connection_basic.cpp network_throttle.cpp network_throttle-detail.cpp mlocker.cpp buffer.cpp net_ssl.cpp)
+    levin_base.cpp memwipe.c connection_basic.cpp network_throttle.cpp network_throttle-detail.cpp mlocker.cpp buffer.cpp net_ssl.cpp)
 
 if (USE_READLINE AND GNU_READLINE_FOUND)
   add_library(epee_readline STATIC readline_buffer.cpp)

contrib/epee/src/byte_slice.cpp

@@ -0,0 +1,209 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <atomic>
+#include <cstdlib>
+#include <cstring>
+#include <limits>
+#include <stdexcept>
+#include <utility>
+
+#include "byte_slice.h"
+
+namespace epee
+{
+  struct byte_slice_data
+  {
+    byte_slice_data() noexcept
+      : ref_count(1)
+    {}
+
+    virtual ~byte_slice_data() noexcept
+    {}
+
+    std::atomic<std::size_t> ref_count;
+  };
+
+  void release_byte_slice::operator()(byte_slice_data* ptr) const noexcept
+  {
+    if (ptr && --(ptr->ref_count) == 0)
+    {
+      ptr->~byte_slice_data();
+      free(ptr);
+    }
+  }
+
+  namespace
+  {
+    template<typename T>
+    struct adapted_byte_slice final : byte_slice_data
+    {
+      explicit adapted_byte_slice(T&& buffer)
+        : byte_slice_data(), buffer(std::move(buffer))
+      {}
+
+      virtual ~adapted_byte_slice() noexcept final override
+      {}
+
+      const T buffer;
+    };
+
+    // bytes "follow" this structure in memory slab
+    struct raw_byte_slice final : byte_slice_data
+    {
+      raw_byte_slice() noexcept
+        : byte_slice_data()
+      {}
+
+      virtual ~raw_byte_slice() noexcept final override
+      {}
+    };
+
+    /* This technique is not-standard, but allows for the reference count and
+       memory for the bytes (when given a list of spans) to be allocated in a
+       single call. In that situation, the dynamic sized bytes are after/behind
+       the raw_byte_slice class. The C runtime has to track the number of bytes
+       allocated regardless, so free'ing is relatively easy. */
+
+    template<typename T, typename... U>
+    std::unique_ptr<T, release_byte_slice> allocate_slice(std::size_t extra_bytes, U&&... args)
+    {
+      if (std::numeric_limits<std::size_t>::max() - sizeof(T) < extra_bytes)
+        throw std::bad_alloc{};
+
+      void* const ptr = malloc(sizeof(T) + extra_bytes);
+      if (ptr == nullptr)
+        throw std::bad_alloc{};
+
+      try
+      {
+        new (ptr) T{std::forward<U>(args)...};
+      }
+      catch (...)
+      {
+        free(ptr);
+        throw;
+      }
+      return std::unique_ptr<T, release_byte_slice>{reinterpret_cast<T*>(ptr)};
+    }
+  } // anonymous
+
+  byte_slice::byte_slice(byte_slice_data* storage, span<const std::uint8_t> portion) noexcept
+    : storage_(storage), portion_(portion)
+  {
+    if (storage_)
+      ++(storage_->ref_count);
+  }
+
+  template<typename T>
+  byte_slice::byte_slice(const adapt_buffer, T&& buffer)
+    : storage_(nullptr), portion_(to_byte_span(to_span(buffer)))
+  {
+    if (!buffer.empty())
+      storage_ = allocate_slice<adapted_byte_slice<T>>(0, std::move(buffer));
+  }
+
+  byte_slice::byte_slice(std::initializer_list<span<const std::uint8_t>> sources)
+    : byte_slice()
+  {
+    std::size_t space_needed = 0;
+    for (const auto source : sources)
+      space_needed += source.size();
+
+    if (space_needed)
+    {
+      auto storage = allocate_slice<raw_byte_slice>(space_needed);
+      span<std::uint8_t> out{reinterpret_cast<std::uint8_t*>(storage.get() + 1), space_needed};
+      portion_ = {out.data(), out.size()};
+
+      for (const auto source : sources)
+      {
+        std::memcpy(out.data(), source.data(), source.size());
+        if (out.remove_prefix(source.size()) < source.size())
+          throw std::bad_alloc{}; // size_t overflow on space_needed
+      }
+      storage_ = std::move(storage);
+    }
+  }
+
+  byte_slice::byte_slice(std::string&& buffer)
+    : byte_slice(adapt_buffer{}, std::move(buffer))
+  {}
+
+  byte_slice::byte_slice(std::vector<std::uint8_t>&& buffer)
+    : byte_slice(adapt_buffer{}, std::move(buffer))
+  {}
+
+  byte_slice::byte_slice(byte_slice&& source) noexcept
+    : storage_(std::move(source.storage_)), portion_(source.portion_)
+  {
+    source.portion_ = epee::span<const std::uint8_t>{};
+  }
+
+  byte_slice& byte_slice::operator=(byte_slice&& source) noexcept
+  {
+    storage_ = std::move(source.storage_);
+    portion_ = source.portion_;
+    if (source.storage_ == nullptr)
+      source.portion_ = epee::span<const std::uint8_t>{};
+
+    return *this;
+  }
+
+  std::size_t byte_slice::remove_prefix(std::size_t max_bytes) noexcept
+  {
+    max_bytes = portion_.remove_prefix(max_bytes);
+    if (portion_.empty())
+      storage_ = nullptr;
+    return max_bytes;
+  }
+
+  byte_slice byte_slice::take_slice(const std::size_t max_bytes) noexcept
+  {
+    byte_slice out{};
+    std::uint8_t const* const ptr = data();
+    out.portion_ = {ptr, portion_.remove_prefix(max_bytes)};
+
+    if (portion_.empty())
+      out.storage_ = std::move(storage_); // no atomic inc/dec
+    else
+      out = {storage_.get(), out.portion_};
+
+    return out;
+  }
+
+  byte_slice byte_slice::get_slice(const std::size_t begin, const std::size_t end) const
+  {
+    if (end < begin || portion_.size() < end)
+      throw std::out_of_range{"bad slice range"};
+
+    if (begin == end)
+      return {};
+    return {storage_.get(), {portion_.begin() + begin, end - begin}};
+  }
+} // epee

contrib/epee/src/connection_basic.cpp

@@ -128,7 +128,7 @@ connection_basic_pimpl::connection_basic_pimpl(const std::string &name) : m_thro
 int connection_basic_pimpl::m_default_tos;
 
 // methods:
-connection_basic::connection_basic(boost::asio::ip::tcp::socket&& sock, boost::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support)
+connection_basic::connection_basic(boost::asio::ip::tcp::socket&& sock, std::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support)
 	:
 	m_state(std::move(state)),
 	mI( new connection_basic_pimpl("peer") ),
@@ -152,7 +152,7 @@ connection_basic::connection_basic(boost::asio::ip::tcp::socket&& sock, boost::s
 	_note("Spawned connection #"<<mI->m_peer_number<<" to " << remote_addr_str << " currently we have sockets count:" << m_state->sock_count);
 }
 
-connection_basic::connection_basic(boost::asio::io_service &io_service, boost::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support)
+connection_basic::connection_basic(boost::asio::io_service &io_service, std::shared_ptr<connection_basic_shared_state> state, ssl_support_t ssl_support)
 	:
 	m_state(std::move(state)),
 	mI( new connection_basic_pimpl("peer") ),

contrib/epee/src/levin_base.cpp

@@ -0,0 +1,128 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "net/levin_base.h"
+
+#include "int-util.h"
+
+namespace epee
+{
+namespace levin
+{
+  bucket_head2 make_header(uint32_t command, uint64_t msg_size, uint32_t flags, bool expect_response) noexcept
+  {
+    bucket_head2 head = {0};
+    head.m_signature = SWAP64LE(LEVIN_SIGNATURE);
+    head.m_have_to_return_data = expect_response;
+    head.m_cb = SWAP64LE(msg_size);
+
+    head.m_command = SWAP32LE(command);
+    head.m_protocol_version = SWAP32LE(LEVIN_PROTOCOL_VER_1);
+    head.m_flags = SWAP32LE(flags);
+    return head;
+  }
+
+  byte_slice make_notify(int command, epee::span<const std::uint8_t> payload)
+  {
+    const bucket_head2 head = make_header(command, payload.size(), LEVIN_PACKET_REQUEST, false);
+    return byte_slice{epee::as_byte_span(head), payload};
+  }
+
+  byte_slice make_noise_notify(const std::size_t noise_bytes)
+  {
+    static constexpr const std::uint32_t flags =
+        LEVIN_PACKET_BEGIN | LEVIN_PACKET_END;
+
+    if (noise_bytes < sizeof(bucket_head2))
+      return nullptr;
+
+    std::string buffer(noise_bytes, char(0));
+    const bucket_head2 head = make_header(0, noise_bytes - sizeof(bucket_head2), flags, false);
+    std::memcpy(std::addressof(buffer[0]), std::addressof(head), sizeof(head));
+
+    return byte_slice{std::move(buffer)};
+  }
+
+  byte_slice make_fragmented_notify(const byte_slice& noise_message, int command, epee::span<const std::uint8_t> payload)
+  {
+    const size_t noise_size = noise_message.size();
+    if (noise_size < sizeof(bucket_head2) * 2)
+      return nullptr;
+
+    if (payload.size() <= noise_size - sizeof(bucket_head2))
+    {
+      /* The entire message can be sent at once, and the levin binary parser
+         will ignore extra bytes. So just pad with zeroes and otherwise send
+         a "normal", not fragmented message. */
+      const size_t padding = noise_size - sizeof(bucket_head2) - payload.size();
+      const span<const uint8_t> padding_bytes{noise_message.end() - padding, padding};
+
+      const bucket_head2 head = make_header(command, noise_size - sizeof(bucket_head2), LEVIN_PACKET_REQUEST, false);
+      return byte_slice{as_byte_span(head), payload, padding_bytes};
+    }
+
+    // fragment message
+    const size_t payload_space = noise_size - sizeof(bucket_head2);
+    const size_t expected_fragments = ((payload.size() - 2) / payload_space) + 1;
+
+    std::string buffer{};
+    buffer.reserve((expected_fragments + 1) * noise_size); // +1 here overselects for internal bucket_head2 value
+
+    bucket_head2 head = make_header(0, noise_size - sizeof(bucket_head2), LEVIN_PACKET_BEGIN, false);
+    buffer.append(reinterpret_cast<const char*>(&head), sizeof(head));
+
+    head.m_command = command;
+    head.m_flags = LEVIN_PACKET_REQUEST;
+    head.m_cb = payload.size();
+    buffer.append(reinterpret_cast<const char*>(&head), sizeof(head));
+
+    size_t copy_size = payload.remove_prefix(payload_space - sizeof(bucket_head2));
+    buffer.append(reinterpret_cast<const char*>(payload.data()) - copy_size, copy_size);
+
+    head.m_command = 0;
+    head.m_flags = 0;
+    head.m_cb = noise_size - sizeof(bucket_head2);
+
+    while (!payload.empty())
+    {
+      copy_size = payload.remove_prefix(payload_space);
+
+      if (payload.empty())
+        head.m_flags = LEVIN_PACKET_END;
+
+      buffer.append(reinterpret_cast<const char*>(&head), sizeof(head));
+      buffer.append(reinterpret_cast<const char*>(payload.data()) - copy_size, copy_size);
+    }
+
+    const size_t padding = noise_size - copy_size - sizeof(bucket_head2);
+    buffer.append(reinterpret_cast<const char*>(noise_message.end()) - padding, padding);
+
+    return byte_slice{std::move(buffer)};
+  }
+} // levin
+} // epee

src/cryptonote_basic/connection_context.h

@@ -31,8 +31,10 @@
 #pragma once
 #include <unordered_set>
 #include <atomic>
+#include <boost/date_time/posix_time/posix_time.hpp>
 #include "net/net_utils_base.h"
 #include "copyable_atomic.h"
+#include "crypto/hash.h"
 
 namespace cryptonote
 {

src/cryptonote_config.h

@@ -100,6 +100,16 @@
 #define CRYPTONOTE_MEMPOOL_TX_LIVETIME                    (86400*3) //seconds, three days
 #define CRYPTONOTE_MEMPOOL_TX_FROM_ALT_BLOCK_LIVETIME     604800 //seconds, one week
 
+// see src/cryptonote_protocol/levin_notify.cpp
+#define CRYPTONOTE_NOISE_MIN_EPOCH                      5      // minutes
+#define CRYPTONOTE_NOISE_EPOCH_RANGE                    30     // seconds
+#define CRYPTONOTE_NOISE_MIN_DELAY                      10     // seconds
+#define CRYPTONOTE_NOISE_DELAY_RANGE                    5      // seconds
+#define CRYPTONOTE_NOISE_BYTES                          3*1024 // 3 KiB
+#define CRYPTONOTE_NOISE_CHANNELS                       2      // Max outgoing connections per zone used for noise/covert sending
+
+#define CRYPTONOTE_MAX_FRAGMENTS                        20 // ~20 * NOISE_BYTES max payload size for covert/noise send
+
 #define COMMAND_RPC_GET_BLOCKS_FAST_MAX_COUNT           1000
 
 #define P2P_LOCAL_WHITE_PEERLIST_LIMIT                  1000

src/cryptonote_protocol/cryptonote_protocol_handler.inl

@@ -2227,69 +2227,11 @@ skip:
   template<class t_core>
   bool t_cryptonote_protocol_handler<t_core>::relay_transactions(NOTIFY_NEW_TRANSACTIONS::request& arg, cryptonote_connection_context& exclude_context)
   {
-    const bool hide_tx_broadcast =
+    for(auto& tx_blob : arg.txs)
-      1 < m_p2p->get_zone_count() && exclude_context.m_remote_address.get_zone() == epee::net_utils::zone::invalid;
+      m_core.on_transaction_relayed(tx_blob);
-
-    if (hide_tx_broadcast)
-      MDEBUG("Attempting to conceal origin of tx via anonymity network connection(s)");
 
     // no check for success, so tell core they're relayed unconditionally
-    const bool pad_transactions = m_core.pad_transactions() || hide_tx_broadcast;
+    m_p2p->send_txs(std::move(arg.txs), exclude_context.m_remote_address.get_zone(), exclude_context.m_connection_id, m_core.pad_transactions());
-    size_t bytes = pad_transactions ? 9 /* header */ + 4 /* 1 + 'txs' */ + tools::get_varint_data(arg.txs.size()).size() : 0;
-    for(auto tx_blob_it = arg.txs.begin(); tx_blob_it!=arg.txs.end(); ++tx_blob_it)
-    {
-      m_core.on_transaction_relayed(*tx_blob_it);
-      if (pad_transactions)
-        bytes += tools::get_varint_data(tx_blob_it->size()).size() + tx_blob_it->size();
-    }
-
-    if (pad_transactions)
-    {
-      // stuff some dummy bytes in to stay safe from traffic volume analysis
-      static constexpr size_t granularity = 1024;
-      size_t padding = granularity - bytes % granularity;
-      const size_t overhead = 2 /* 1 + '_' */ + tools::get_varint_data(padding).size();
-      if (overhead > padding)
-        padding = 0;
-      else
-        padding -= overhead;
-      arg._ = std::string(padding, ' ');
-
-      std::string arg_buff;
-      epee::serialization::store_t_to_binary(arg, arg_buff);
-
-      // we probably lowballed the payload size a bit, so added a but too much. Fix this now.
-      size_t remove = arg_buff.size() % granularity;
-      if (remove > arg._.size())
-        arg._.clear();
-      else
-        arg._.resize(arg._.size() - remove);
-      // if the size of _ moved enough, we might lose byte in size encoding, we don't care
-    }
-
-    std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections;
-    m_p2p->for_each_connection([hide_tx_broadcast, &exclude_context, &connections](connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)
-    {
-      const epee::net_utils::zone current_zone = context.m_remote_address.get_zone();
-      const bool broadcast_to_peer =
-        peer_id &&
-        (hide_tx_broadcast != bool(current_zone == epee::net_utils::zone::public_)) &&
-	exclude_context.m_connection_id != context.m_connection_id;
-
-      if (broadcast_to_peer)
-        connections.push_back({current_zone, context.m_connection_id});
-
-      return true;
-    });
-
-    if (connections.empty())
-      MERROR("Transaction not relayed - no" << (hide_tx_broadcast ? " privacy": "") << " peers available");
-    else
-    {
-      std::string fullBlob;
-      epee::serialization::store_t_to_binary(arg, fullBlob);
-      m_p2p->relay_notify_to_list(NOTIFY_NEW_TRANSACTIONS::ID, epee::strspan<uint8_t>(fullBlob), std::move(connections));
-    }
     return true;
   }
   //------------------------------------------------------------------------------------------------------------------------

src/cryptonote_protocol/levin_notify.cpp

@@ -0,0 +1,574 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "levin_notify.h"
+
+#include <boost/asio/steady_timer.hpp>
+#include <boost/system/system_error.hpp>
+#include <chrono>
+#include <deque>
+#include <stdexcept>
+
+#include "common/expect.h"
+#include "common/varint.h"
+#include "cryptonote_config.h"
+#include "crypto/random.h"
+#include "cryptonote_basic/connection_context.h"
+#include "cryptonote_protocol/cryptonote_protocol_defs.h"
+#include "net/dandelionpp.h"
+#include "p2p/net_node.h"
+
+namespace cryptonote
+{
+namespace levin
+{
+  namespace
+  {
+    constexpr std::size_t connection_id_reserve_size = 100;
+
+    constexpr const std::chrono::minutes noise_min_epoch{CRYPTONOTE_NOISE_MIN_EPOCH};
+    constexpr const std::chrono::seconds noise_epoch_range{CRYPTONOTE_NOISE_EPOCH_RANGE};
+
+    constexpr const std::chrono::seconds noise_min_delay{CRYPTONOTE_NOISE_MIN_DELAY};
+    constexpr const std::chrono::seconds noise_delay_range{CRYPTONOTE_NOISE_DELAY_RANGE};
+
+    /*! Select a randomized duration from 0 to `range`. The precision will be to
+        the systems `steady_clock`. As an example, supplying 3 seconds to this
+        function will select a duration from [0, 3] seconds, and the increments
+        for the selection will be determined by the `steady_clock` precision
+        (typically nanoseconds).
+
+        \return A randomized duration from 0 to `range`. */
+    std::chrono::steady_clock::duration random_duration(std::chrono::steady_clock::duration range)
+    {
+      using rep = std::chrono::steady_clock::rep;
+      return std::chrono::steady_clock::duration{crypto::rand_range(rep(0), range.count())};
+    }
+
+    //! \return All outgoing connections supporting fragments in `connections`.
+    std::vector<boost::uuids::uuid> get_out_connections(connections& p2p)
+    {
+      std::vector<boost::uuids::uuid> outs;
+      outs.reserve(connection_id_reserve_size);
+
+      /* The foreach call is serialized with a lock, but should be quick due to
+         the reserve call so a strand is not used. Investigate if there is lots
+         of waiting in here. */
+
+      p2p.foreach_connection([&outs] (detail::p2p_context& context) {
+        if (!context.m_is_income)
+          outs.emplace_back(context.m_connection_id);
+        return true;
+      });
+
+      return outs;
+    }
+
+    std::string make_tx_payload(std::vector<blobdata>&& txs, const bool pad)
+    {
+      NOTIFY_NEW_TRANSACTIONS::request request{};
+      request.txs = std::move(txs);
+
+      if (pad)
+      {
+        size_t bytes = 9 /* header */ + 4 /* 1 + 'txs' */ + tools::get_varint_data(request.txs.size()).size();
+        for(auto tx_blob_it = request.txs.begin(); tx_blob_it!=request.txs.end(); ++tx_blob_it)
+          bytes += tools::get_varint_data(tx_blob_it->size()).size() + tx_blob_it->size();
+
+        // stuff some dummy bytes in to stay safe from traffic volume analysis
+        static constexpr const size_t granularity = 1024;
+        size_t padding = granularity - bytes % granularity;
+        const size_t overhead = 2 /* 1 + '_' */ + tools::get_varint_data(padding).size();
+        if (overhead > padding)
+          padding = 0;
+        else
+          padding -= overhead;
+        request._ = std::string(padding, ' ');
+
+        std::string arg_buff;
+        epee::serialization::store_t_to_binary(request, arg_buff);
+
+        // we probably lowballed the payload size a bit, so added a but too much. Fix this now.
+        size_t remove = arg_buff.size() % granularity;
+        if (remove > request._.size())
+          request._.clear();
+        else
+          request._.resize(request._.size() - remove);
+        // if the size of _ moved enough, we might lose byte in size encoding, we don't care
+      }
+
+      std::string fullBlob;
+      if (!epee::serialization::store_t_to_binary(request, fullBlob))
+        throw std::runtime_error{"Failed to serialize to epee binary format"};
+
+      return fullBlob;
+    }
+
+    /* The current design uses `asio::strand`s. The documentation isn't as clear
+       as it should be - a `strand` has an internal `mutex` and `bool`. The
+       `mutex` synchronizes thread access and the `bool` is set when a thread is
+       executing something "in the strand". Therefore, if a callback has lots of
+       work to do in a `strand`, asio can switch to some other task instead of
+       blocking 1+ threads to wait for the original thread to complete the task
+       (as is the case when client code has a `mutex` inside the callback). The
+       downside is that asio _always_ allocates for the callback, even if it can
+       be immediately executed. So if all work in a strand is minimal, a lock
+       may be better.
+
+       This code uses a strand per "zone" and a strand per "channel in a zone".
+       `dispatch` is used heavily, which means "execute immediately in _this_
+       thread if the strand is not in use, otherwise queue the callback to be
+       executed immediately after the strand completes its current task".
+       `post` is used where deferred execution to an `asio::io_service::run`
+       thread is preferred.
+
+       The strand per "zone" is useful because the levin
+       `foreach_connection` is blocked with a mutex anyway. So this primarily
+       helps with reducing blocking of a thread attempting a "flood"
+       notification. Updating/merging the outgoing connections in the
+       Dandelion++ map is also somewhat expensive.
+
+       The strand per "channel" may need a re-visit. The most "expensive" code
+       is figuring out the noise/notification to send. If levin code is
+       optimized further, it might be better to just use standard locks per
+       channel. */
+
+    //! A queue of levin messages for a noise i2p/tor link
+    struct noise_channel
+    {
+      explicit noise_channel(boost::asio::io_service& io_service)
+        : active(nullptr),
+          queue(),
+          strand(io_service),
+          next_noise(io_service),
+          connection(boost::uuids::nil_uuid())
+      {}
+
+      // `asio::io_service::strand` cannot be copied or moved
+      noise_channel(const noise_channel&) = delete;
+      noise_channel& operator=(const noise_channel&) = delete;
+
+      // Only read/write these values "inside the strand"
+
+      epee::byte_slice active;
+      std::deque<epee::byte_slice> queue;
+      boost::asio::io_service::strand strand;
+      boost::asio::steady_timer next_noise;
+      boost::uuids::uuid connection;
+    };
+  } // anonymous
+
+  namespace detail
+  {
+    struct zone
+    {
+      explicit zone(boost::asio::io_service& io_service, std::shared_ptr<connections> p2p, epee::byte_slice noise_in)
+        : p2p(std::move(p2p)),
+          noise(std::move(noise_in)),
+          next_epoch(io_service),
+          strand(io_service),
+          map(),
+          channels(),
+          connection_count(0)
+      {
+        for (std::size_t count = 0; !noise.empty() && count < CRYPTONOTE_NOISE_CHANNELS; ++count)
+          channels.emplace_back(io_service);
+      }
+
+      const std::shared_ptr<connections> p2p;
+      const epee::byte_slice noise; //!< `!empty()` means zone is using noise channels
+      boost::asio::steady_timer next_epoch;
+      boost::asio::io_service::strand strand;
+      net::dandelionpp::connection_map map;//!< Tracks outgoing uuid's for noise channels or Dandelion++ stems
+      std::deque<noise_channel> channels;  //!< Never touch after init; only update elements on `noise_channel.strand`
+      std::atomic<std::size_t> connection_count; //!< Only update in strand, can be read at any time
+    };
+  } // detail
+
+  namespace
+  {
+    //! Adds a message to the sending queue of the channel.
+    class queue_covert_notify
+    {
+      std::shared_ptr<detail::zone> zone_;
+      epee::byte_slice message_; // Requires manual copy constructor
+      const std::size_t destination_;
+
+    public:
+      queue_covert_notify(std::shared_ptr<detail::zone> zone, epee::byte_slice message, std::size_t destination)
+        : zone_(std::move(zone)), message_(std::move(message)), destination_(destination)
+      {}
+
+      queue_covert_notify(queue_covert_notify&&) = default;
+      queue_covert_notify(const queue_covert_notify& source)
+        : zone_(source.zone_), message_(source.message_.clone()), destination_(source.destination_)
+      {}
+
+      //! \pre Called within `zone_->channels[destionation_].strand`.
+      void operator()()
+      {
+        if (!zone_)
+          return;
+
+        noise_channel& channel = zone_->channels.at(destination_);
+        assert(channel.strand.running_in_this_thread());
+
+        if (!channel.connection.is_nil())
+          channel.queue.push_back(std::move(message_));
+      }
+    };
+
+    //! Sends a message to every active connection
+    class flood_notify
+    {
+      std::shared_ptr<detail::zone> zone_;
+      epee::byte_slice message_; // Requires manual copy
+      boost::uuids::uuid source_;
+
+    public:
+      explicit flood_notify(std::shared_ptr<detail::zone> zone, epee::byte_slice message, const boost::uuids::uuid& source)
+        : zone_(std::move(zone)), message_(message.clone()), source_(source)
+      {}
+
+      flood_notify(flood_notify&&) = default;
+      flood_notify(const flood_notify& source)
+        : zone_(source.zone_), message_(source.message_.clone()), source_(source.source_)
+      {}
+
+      void operator()() const
+      {
+        if (!zone_ || !zone_->p2p)
+          return;
+
+        assert(zone_->strand.running_in_this_thread());
+
+        /* The foreach should be quick, but then it iterates and acquires the
+           same lock for every connection. So do in a strand because two threads
+           will ping-pong each other with cacheline invalidations. Revisit if
+           algorithm changes or the locking strategy within the levin config
+           class changes. */
+
+        std::vector<boost::uuids::uuid> connections;
+        connections.reserve(connection_id_reserve_size);
+        zone_->p2p->foreach_connection([this, &connections] (detail::p2p_context& context) {
+          if (this->source_ != context.m_connection_id)
+            connections.emplace_back(context.m_connection_id);
+          return true;
+        });
+
+        for (const boost::uuids::uuid& connection : connections)
+          zone_->p2p->send(message_.clone(), connection);
+      }
+    };
+
+    //! Updates the connection for a channel.
+    struct update_channel
+    {
+      std::shared_ptr<detail::zone> zone_;
+      const std::size_t channel_;
+      const boost::uuids::uuid connection_;
+
+      //! \pre Called within `stem_.strand`.
+      void operator()() const
+      {
+        if (!zone_)
+          return;
+
+        noise_channel& channel = zone_->channels.at(channel_);
+        assert(channel.strand.running_in_this_thread());
+        static_assert(
+          CRYPTONOTE_MAX_FRAGMENTS <= (noise_min_epoch / (noise_min_delay + noise_delay_range)),
+          "Max fragments more than the max that can be sent in an epoch"
+        );
+
+        /* This clears the active message so that a message "in-flight" is
+           restarted. DO NOT try to send the remainder of the fragments, this
+           additional send time can leak that this node was sending out a real
+           notify (tx) instead of dummy noise. */
+
+        channel.connection = connection_;
+        channel.active = nullptr;
+
+        if (connection_.is_nil())
+          channel.queue.clear();
+      }
+    };
+
+    //! Merges `out_connections_` into the existing `zone_->map`.
+    struct update_channels
+    {
+      std::shared_ptr<detail::zone> zone_;
+      std::vector<boost::uuids::uuid> out_connections_;
+
+      //! \pre Called within `zone->strand`.
+      static void post(std::shared_ptr<detail::zone> zone)
+      {
+        if (!zone)
+          return;
+
+        assert(zone->strand.running_in_this_thread());
+
+        zone->connection_count = zone->map.size();
+        for (auto id = zone->map.begin(); id != zone->map.end(); ++id)
+        {
+          const std::size_t i = id - zone->map.begin();
+          zone->channels[i].strand.post(update_channel{zone, i, *id});
+        }
+      }
+
+      //! \pre Called within `zone_->strand`.
+      void operator()()
+      {
+        if (!zone_)
+          return;
+
+        assert(zone_->strand.running_in_this_thread());
+        if (zone_->map.update(std::move(out_connections_)))
+          post(std::move(zone_));
+      }
+    };
+
+    //! Swaps out noise channels entirely; new epoch start.
+    class change_channels
+    {
+      std::shared_ptr<detail::zone> zone_;
+      net::dandelionpp::connection_map map_; // Requires manual copy constructor
+
+    public:
+      explicit change_channels(std::shared_ptr<detail::zone> zone, net::dandelionpp::connection_map map)
+        : zone_(std::move(zone)), map_(std::move(map))
+      {}
+
+      change_channels(change_channels&&) = default;
+      change_channels(const change_channels& source)
+        : zone_(source.zone_), map_(source.map_.clone())
+      {}
+
+      //! \pre Called within `zone_->strand`.
+      void operator()()
+      {
+        if (!zone_)
+          return
+
+        assert(zone_->strand.running_in_this_thread());
+
+        zone_->map = std::move(map_);
+        update_channels::post(std::move(zone_));
+      }
+    };
+
+    //! Sends a noise packet or real notification and sets timer for next call.
+    struct send_noise
+    {
+      std::shared_ptr<detail::zone> zone_;
+      const std::size_t channel_;
+
+      static void wait(const std::chrono::steady_clock::time_point start, std::shared_ptr<detail::zone> zone, const std::size_t index)
+      {
+        if (!zone)
+          return;
+
+        noise_channel& channel = zone->channels.at(index);
+        channel.next_noise.expires_at(start + noise_min_delay + random_duration(noise_delay_range));
+        channel.next_noise.async_wait(
+          channel.strand.wrap(send_noise{std::move(zone), index})
+        );
+      }
+
+      //! \pre Called within `zone_->channels[channel_].strand`.
+      void operator()(boost::system::error_code error)
+      {
+        if (!zone_ || !zone_->p2p || zone_->noise.empty())
+          return;
+
+        if (error && error != boost::system::errc::operation_canceled)
+          throw boost::system::system_error{error, "send_noise timer failed"};
+
+        assert(zone_->channels.at(channel_).strand.running_in_this_thread());
+
+        const auto start = std::chrono::steady_clock::now();
+        noise_channel& channel = zone_->channels.at(channel_);
+
+        if (!channel.connection.is_nil())
+        {
+          epee::byte_slice message = nullptr;
+          if (!channel.active.empty())
+            message = channel.active.take_slice(zone_->noise.size());
+          else if (!channel.queue.empty())
+          {
+            channel.active = channel.queue.front().clone();
+            message = channel.active.take_slice(zone_->noise.size());
+          }
+          else
+            message = zone_->noise.clone();
+
+          if (zone_->p2p->send(std::move(message), channel.connection))
+          {
+            if (!channel.queue.empty() && channel.active.empty())
+              channel.queue.pop_front();
+          }
+          else
+          {
+            channel.active = nullptr;
+            channel.connection = boost::uuids::nil_uuid();
+            zone_->strand.post(
+              update_channels{zone_, get_out_connections(*zone_->p2p)}
+            );
+          }
+        }
+
+        wait(start, std::move(zone_), channel_);
+      }
+    };
+
+    //! Prepares connections for new channel epoch and sets timer for next epoch
+    struct start_epoch
+    {
+      // Variables allow for Dandelion++ extension
+      std::shared_ptr<detail::zone> zone_;
+      std::chrono::seconds min_epoch_;
+      std::chrono::seconds epoch_range_;
+      std::size_t count_;
+
+      //! \pre Should not be invoked within any strand to prevent blocking.
+      void operator()(const boost::system::error_code error = {})
+      {
+        if (!zone_ || !zone_->p2p)
+          return;
+
+        if (error && error != boost::system::errc::operation_canceled)
+          throw boost::system::system_error{error, "start_epoch timer failed"};
+
+        const auto start = std::chrono::steady_clock::now();
+        zone_->strand.dispatch(
+          change_channels{zone_, net::dandelionpp::connection_map{get_out_connections(*(zone_->p2p)), count_}}
+        );
+
+        detail::zone& alias = *zone_;
+        alias.next_epoch.expires_at(start + min_epoch_ + random_duration(epoch_range_));
+        alias.next_epoch.async_wait(start_epoch{std::move(*this)});
+      }
+    };
+  } // anonymous
+
+  notify::notify(boost::asio::io_service& service, std::shared_ptr<connections> p2p, epee::byte_slice noise)
+    : zone_(std::make_shared<detail::zone>(service, std::move(p2p), std::move(noise)))
+  {
+    if (!zone_->p2p)
+      throw std::logic_error{"cryptonote::levin::notify cannot have nullptr p2p argument"};
+
+    if (!zone_->noise.empty())
+    {
+      const auto now = std::chrono::steady_clock::now();
+      start_epoch{zone_, noise_min_epoch, noise_epoch_range, CRYPTONOTE_NOISE_CHANNELS}();
+      for (std::size_t channel = 0; channel < zone_->channels.size(); ++channel)
+        send_noise::wait(now, zone_, channel);
+    }
+  }
+
+  notify::~notify() noexcept
+  {}
+
+  notify::status notify::get_status() const noexcept
+  {
+    if (!zone_)
+      return {false, false};
+
+    return {!zone_->noise.empty(), CRYPTONOTE_NOISE_CHANNELS <= zone_->connection_count};
+  }
+
+  void notify::new_out_connection()
+  {
+    if (!zone_ || zone_->noise.empty() || CRYPTONOTE_NOISE_CHANNELS <= zone_->connection_count)
+      return;
+
+    zone_->strand.dispatch(
+      update_channels{zone_, get_out_connections(*(zone_->p2p))}
+    );
+  }
+
+  void notify::run_epoch()
+  {
+    if (!zone_)
+      return;
+    zone_->next_epoch.cancel();
+  }
+
+  void notify::run_stems()
+  {
+    if (!zone_)
+      return;
+
+    for (noise_channel& channel : zone_->channels)
+      channel.next_noise.cancel();
+  }
+
+  bool notify::send_txs(std::vector<cryptonote::blobdata> txs, const boost::uuids::uuid& source, const bool pad_txs)
+  {
+    if (!zone_)
+      return false;
+
+    if (!zone_->noise.empty() && !zone_->channels.empty())
+    {
+      // covert send in "noise" channel
+      static_assert(
+        CRYPTONOTE_MAX_FRAGMENTS * CRYPTONOTE_NOISE_BYTES <= LEVIN_DEFAULT_MAX_PACKET_SIZE, "most nodes will reject this fragment setting"
+      );
+
+      // padding is not useful when using noise mode
+      const std::string payload = make_tx_payload(std::move(txs), false);
+      epee::byte_slice message = epee::levin::make_fragmented_notify(
+        zone_->noise, NOTIFY_NEW_TRANSACTIONS::ID, epee::strspan<std::uint8_t>(payload)
+      );
+      if (CRYPTONOTE_MAX_FRAGMENTS * zone_->noise.size() < message.size())
+      {
+        MERROR("notify::send_txs provided message exceeding covert fragment size");
+        return false;
+      }
+
+      for (std::size_t channel = 0; channel < zone_->channels.size(); ++channel)
+      {
+        zone_->channels[channel].strand.dispatch(
+          queue_covert_notify{zone_, message.clone(), channel}
+        );
+      }
+    }
+    else
+    {
+      const std::string payload = make_tx_payload(std::move(txs), pad_txs);
+      epee::byte_slice message =
+        epee::levin::make_notify(NOTIFY_NEW_TRANSACTIONS::ID, epee::strspan<std::uint8_t>(payload));
+
+      // traditional monero send technique
+      zone_->strand.dispatch(flood_notify{zone_, std::move(message), source});
+    }
+
+    return true;
+  }
+} // levin
+} // net

src/cryptonote_protocol/levin_notify.h

@@ -0,0 +1,132 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/asio/io_service.hpp>
+#include <boost/uuid/uuid.hpp>
+#include <memory>
+#include <vector>
+
+#include "byte_slice.h"
+#include "cryptonote_basic/blobdatatype.h"
+#include "net/enums.h"
+#include "span.h"
+
+namespace epee
+{
+namespace levin
+{
+    template<typename> class async_protocol_handler_config;
+}
+}
+
+namespace nodetool
+{
+  template<typename> struct p2p_connection_context_t;
+}
+
+namespace cryptonote
+{
+  struct cryptonote_connection_context;
+}
+
+namespace cryptonote
+{
+namespace levin
+{
+  namespace detail
+  {
+    using p2p_context = nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context>;
+    struct zone; //!< Internal data needed for zone notifications
+  } // detail
+
+  using connections = epee::levin::async_protocol_handler_config<detail::p2p_context>;
+
+  //! Provides tx notification privacy
+  class notify
+  {
+    std::shared_ptr<detail::zone> zone_;
+
+  public:
+    struct status
+    {
+      bool has_noise;
+      bool connections_filled;
+    };
+
+    //! Construct an instance that cannot notify.
+    notify() noexcept
+      : zone_(nullptr)
+    {}
+
+    //! Construct an instance with available notification `zones`.
+    explicit notify(boost::asio::io_service& service, std::shared_ptr<connections> p2p, epee::byte_slice noise);
+
+    notify(const notify&) = delete;
+    notify(notify&&) = default;
+
+    ~notify() noexcept;
+
+    notify& operator=(const notify&) = delete;
+    notify& operator=(notify&&) = default;
+
+    //! \return Status information for zone selection.
+    status get_status() const noexcept;
+
+    //! Probe for new outbound connection - skips if not needed.
+    void new_out_connection();
+
+    //! Run the logic for the next epoch immediately. Only use in testing.
+    void run_epoch();
+
+    //! Run the logic for the next stem timeout imemdiately. Only use in  testing.
+    void run_stems();
+
+    /*! Send txs using `cryptonote_protocol_defs.h` payload format wrapped in a
+        levin header. The message will be sent in a "discreet" manner if
+        enabled - if `!noise.empty()` then the `command`/`payload` will be
+        queued to send at the next available noise interval. Otherwise, a
+        standard Monero flood notification will be used.
+
+        \note Eventually Dandelion++ stem sending will be used here when
+          enabled.
+
+        \param txs The transactions that need to be serialized and relayed.
+        \param source The source of the notification. `is_nil()` indicates this
+          node is the source. Dandelion++ will use this to map a source to a
+          particular stem.
+        \param pad_txs A request to pad txs to help conceal origin via
+          statistical analysis. Ignored if noise was enabled during
+          construction.
+
+      \return True iff the notification is queued for sending. */
+    bool send_txs(std::vector<blobdata> txs, const boost::uuids::uuid& source, bool pad_txs);
+  };
+} // levin
+} // net

src/net/CMakeLists.txt

@@ -26,8 +26,8 @@
 # STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
 # THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-set(net_sources error.cpp i2p_address.cpp parse.cpp socks.cpp socks_connect.cpp tor_address.cpp)
+set(net_sources dandelionpp.cpp error.cpp i2p_address.cpp parse.cpp socks.cpp socks_connect.cpp tor_address.cpp)
-set(net_headers error.h i2p_address.h parse.h socks.h socks_connect.h tor_address.h)
+set(net_headers dandelionpp.h error.h i2p_address.h parse.h socks.h socks_connect.h tor_address.h)
 
 monero_add_library(net ${net_sources} ${net_headers})
 target_link_libraries(net common epee ${Boost_ASIO_LIBRARY})

src/net/dandelionpp.cpp

@@ -0,0 +1,212 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "dandelionpp.h"
+
+#include <boost/container/small_vector.hpp>
+#include <boost/uuid/nil_generator.hpp>
+#include <chrono>
+
+#include "common/expect.h"
+#include "cryptonote_config.h"
+#include "crypto/crypto.h"
+
+namespace net
+{
+namespace dandelionpp
+{
+    namespace
+    {
+        constexpr const std::size_t expected_max_channels = CRYPTONOTE_NOISE_CHANNELS;
+
+        // could be in util somewhere
+        struct key_less
+        {
+            template<typename K, typename V>
+            bool operator()(const std::pair<K, V>& left, const K& right) const
+            {
+                return left.first < right;
+            }
+
+            template<typename K, typename V>
+            bool operator()(const K& left, const std::pair<K, V>& right) const
+            {
+                return left < right.first;
+            }
+        };
+
+        std::size_t select_stem(epee::span<const std::size_t> usage, epee::span<const boost::uuids::uuid> out_map)
+        {
+            assert(usage.size() < std::numeric_limits<std::size_t>::max()); // prevented in constructor
+            if (usage.size() < out_map.size())
+                return std::numeric_limits<std::size_t>::max();
+
+            // small_vector uses stack space if `expected_max_channels < capacity()`
+            std::size_t lowest = std::numeric_limits<std::size_t>::max();
+            boost::container::small_vector<std::size_t, expected_max_channels> choices;
+            static_assert(sizeof(choices) < 256, "choices is too large based on current configuration");
+
+            for (const boost::uuids::uuid& out : out_map)
+            {
+                if (!out.is_nil())
+                {
+                    const std::size_t location = std::addressof(out) - out_map.begin();
+                    if (usage[location] < lowest)
+                    {
+                        lowest = usage[location];
+                        choices = {location};
+                    }
+                    else if (usage[location] == lowest)
+                        choices.push_back(location);
+                }
+            }
+
+            switch (choices.size())
+            {
+            case 0:
+                return std::numeric_limits<std::size_t>::max();
+            case 1:
+                return choices[0];
+            default:
+                break;
+            }
+
+            return choices[crypto::rand_idx(choices.size())];
+        }
+    } // anonymous
+
+    connection_map::connection_map(std::vector<boost::uuids::uuid> out_connections, const std::size_t stems)
+      : out_mapping_(std::move(out_connections)),
+        in_mapping_(),
+        usage_count_()
+    {
+        // max value is used by `select_stem` as error case
+        if (stems == std::numeric_limits<std::size_t>::max())
+            MONERO_THROW(common_error::kInvalidArgument, "stems value cannot be max size_t");
+
+        usage_count_.resize(stems);
+        if (stems < out_mapping_.size())
+        {
+            for (unsigned i = 0; i < stems; ++i)
+                std::swap(out_mapping_[i], out_mapping_.at(i + crypto::rand_idx(out_mapping_.size() - i)));
+
+            out_mapping_.resize(stems);
+        }
+        else
+        {
+            std::shuffle(out_mapping_.begin(), out_mapping_.end(), crypto::random_device{});
+        }
+    }
+
+    connection_map::~connection_map() noexcept
+    {}
+
+    connection_map connection_map::clone() const
+    {
+        return {*this};
+    }
+
+    bool connection_map::update(std::vector<boost::uuids::uuid> current)
+    {
+        std::sort(current.begin(), current.end());
+
+        bool replace = false;
+        for (auto& existing_out : out_mapping_)
+        {
+            const auto elem = std::lower_bound(current.begin(), current.end(), existing_out);
+            if (elem == current.end() || *elem != existing_out)
+            {
+                existing_out = boost::uuids::nil_uuid();
+                replace = true;
+            }
+            else // already using connection, remove it from candidate list
+                current.erase(elem);
+        }
+
+        if (!replace && out_mapping_.size() == usage_count_.size())
+            return false;
+
+        const std::size_t existing_outs = out_mapping_.size();
+        for (std::size_t i = 0; i < usage_count_.size() && !current.empty(); ++i)
+        {
+            const bool increase_stems = out_mapping_.size() <= i;
+            if (increase_stems || out_mapping_[i].is_nil())
+            {
+                std::swap(current.back(), current.at(crypto::rand_idx(current.size())));
+                if (increase_stems)
+                    out_mapping_.push_back(current.back());
+                else
+                    out_mapping_[i] = current.back();
+                current.pop_back();
+            }
+        }
+
+        return replace || existing_outs < out_mapping_.size();
+    }
+
+    std::size_t connection_map::size() const noexcept
+    {
+        std::size_t count = 0;
+        for (const boost::uuids::uuid& connection : out_mapping_)
+        {
+            if (!connection.is_nil())
+                ++count;
+        }
+        return count;
+    }
+
+    boost::uuids::uuid connection_map::get_stem(const boost::uuids::uuid& source)
+    {
+        auto elem = std::lower_bound(in_mapping_.begin(), in_mapping_.end(), source, key_less{});
+        if (elem == in_mapping_.end() || elem->first != source)
+        {
+            const std::size_t index = select_stem(epee::to_span(usage_count_), epee::to_span(out_mapping_));
+            if (out_mapping_.size() < index)
+                return boost::uuids::nil_uuid();
+
+            elem = in_mapping_.emplace(elem, source, index);
+            usage_count_[index]++;
+        }
+        else if (out_mapping_.at(elem->second).is_nil()) // stem connection disconnected after mapping
+        {
+            usage_count_.at(elem->second)--;
+            const std::size_t index = select_stem(epee::to_span(usage_count_), epee::to_span(out_mapping_));
+            if (out_mapping_.size() < index)
+            {
+                in_mapping_.erase(elem);
+                return boost::uuids::nil_uuid();
+            }
+
+            elem->second = index;
+            usage_count_[index]++;
+        }
+
+        return out_mapping_[elem->second];
+    }
+} // dandelionpp
+} // net

src/net/dandelionpp.h

@@ -0,0 +1,106 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/uuid/uuid.hpp>
+#include <cstddef>
+#include <memory>
+#include <utility>
+#include <vector>
+
+#include "span.h"
+
+namespace net
+{
+namespace dandelionpp
+{
+    //! Assists with mapping source -> stem and tracking connections for stem.
+    class connection_map
+    {
+        // Make sure to update clone method if changing members
+        std::vector<boost::uuids::uuid> out_mapping_; //<! Current outgoing uuid connection at index.
+        std::vector<std::pair<boost::uuids::uuid, std::size_t>> in_mapping_; //<! uuid source to an `out_mapping_` index.
+        std::vector<std::size_t> usage_count_;
+
+        // Use clone method to prevent "hidden" copies.
+        connection_map(const connection_map&) = default;
+
+    public:
+        using value_type = boost::uuids::uuid;
+        using size_type = std::vector<boost::uuids::uuid>::size_type;
+        using difference_type = std::vector<boost::uuids::uuid>::difference_type;
+        using reference = const boost::uuids::uuid&;
+        using const_reference = reference;
+        using iterator = std::vector<boost::uuids::uuid>::const_iterator;
+        using const_iterator = iterator;
+
+        //! Initialized with zero stem connections.
+        explicit connection_map()
+          : connection_map(std::vector<boost::uuids::uuid>{}, 0)
+        {}
+
+        //! Initialized with `out_connections` and `stem_count`.
+        explicit connection_map(std::vector<boost::uuids::uuid> out_connections, std::size_t stems);
+
+        connection_map(connection_map&&) = default;
+        ~connection_map() noexcept;
+        connection_map& operator=(connection_map&&) = default;
+        connection_map& operator=(const connection_map&) = delete;
+
+        //! \return An exact duplicate of `this` map.
+        connection_map clone() const;
+
+        //! \return First stem connection.
+        const_iterator begin() const noexcept
+        {
+            return out_mapping_.begin();
+        }
+
+        //! \return One-past the last stem connection.
+        const_iterator end() const noexcept
+        {
+            return out_mapping_.end();
+        }
+
+        /*! Merges in current connections with the previous set of connections.
+            If a connection died, a new one will take its place in the stem or
+            the stem is marked as dead.
+
+            \param connections Current outbound connection ids.
+            \return True if any updates to `get_connections()` was made. */
+        bool update(std::vector<boost::uuids::uuid> current);
+
+        //! \return Number of outgoing connections in use.
+        std::size_t size() const noexcept;
+
+        //! \return Current stem mapping for `source` or `nil_uuid()` if none is possible.
+        boost::uuids::uuid get_stem(const boost::uuids::uuid& source);
+    };
+} // dandelionpp
+} // net

src/p2p/net_node.cpp

@@ -144,7 +144,7 @@ namespace nodetool
     const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_add_exclusive_node   = {"add-exclusive-node", "Specify list of peers to connect to only."
                                                                                                   " If this option is given the options add-priority-node and seed-node are ignored"};
     const command_line::arg_descriptor<std::vector<std::string> > arg_p2p_seed_node   = {"seed-node", "Connect to a node to retrieve peer addresses, and disconnect"};
-    const command_line::arg_descriptor<std::vector<std::string> > arg_proxy = {"proxy", "<network-type>,<socks-ip:port>[,max_connections] i.e. \"tor,127.0.0.1:9050,100\""};
+    const command_line::arg_descriptor<std::vector<std::string> > arg_proxy = {"proxy", "<network-type>,<socks-ip:port>[,max_connections][,disable_noise] i.e. \"tor,127.0.0.1:9050,100,disable_noise\""};
     const command_line::arg_descriptor<std::vector<std::string> > arg_anonymous_inbound = {"anonymous-inbound", "<hidden-service-address>,<[bind-ip:]port>[,max_connections] i.e. \"x.onion,127.0.0.1:18083,100\""};
     const command_line::arg_descriptor<bool> arg_p2p_hide_my_port   =    {"hide-my-port", "Do not announce yourself as peerlist candidate", false, true};
     const command_line::arg_descriptor<bool> arg_no_sync = {"no-sync", "Don't synchronize the blockchain with other peers", false};
@@ -183,7 +183,17 @@ namespace nodetool
             const boost::string_ref proxy{next->begin(), next->size()};
 
             ++next;
-            if (!next.eof())
+            for (unsigned count = 0; !next.eof(); ++count, ++next)
+            {
+                if (2 <= count)
+                {
+                    MERROR("Too many ',' characters given to --" << arg_proxy.name);
+                    return boost::none;
+                }
+
+                if (boost::string_ref{next->begin(), next->size()} == "disable_noise")
+                    proxies.back().noise = false;
+                else
                 {
                     proxies.back().max_connections = get_max_connections(*next);
                     if (proxies.back().max_connections == 0)
@@ -192,6 +202,7 @@ namespace nodetool
                         return boost::none;
                     }
                 }
+            }
 
             switch (epee::net_utils::zone_from_string(zone))
             {

src/p2p/net_node.h

@@ -43,6 +43,7 @@
 #include <vector>
 
 #include "cryptonote_config.h"
+#include "cryptonote_protocol/levin_notify.h"
 #include "warnings.h"
 #include "net/abstract_tcp_server2.h"
 #include "net/levin_protocol_handler.h"
@@ -66,12 +67,14 @@ namespace nodetool
     proxy()
       : max_connections(-1),
         address(),
-        zone(epee::net_utils::zone::invalid)
+        zone(epee::net_utils::zone::invalid),
+        noise(true)
     {}
 
     std::int64_t max_connections;
     boost::asio::ip::tcp::endpoint address;
     epee::net_utils::zone zone;
+    bool noise;
   };
 
   struct anonymous_inbound
@@ -154,6 +157,7 @@ namespace nodetool
           m_bind_ipv6_address(),
           m_port(),
           m_port_ipv6(),
+          m_notifier(),
           m_our_address(),
           m_peerlist(),
           m_config{},
@@ -172,6 +176,7 @@ namespace nodetool
           m_bind_ipv6_address(),
           m_port(),
           m_port_ipv6(),
+          m_notifier(),
           m_our_address(),
           m_peerlist(),
           m_config{},
@@ -189,6 +194,7 @@ namespace nodetool
       std::string m_bind_ipv6_address;
       std::string m_port;
       std::string m_port_ipv6;
+      cryptonote::levin::notify m_notifier;
       epee::net_utils::network_address m_our_address; // in anonymity networks
       peerlist_manager m_peerlist;
       config m_config;
@@ -255,7 +261,6 @@ namespace nodetool
     size_t get_public_gray_peers_count();
     void get_public_peerlist(std::vector<peerlist_entry>& gray, std::vector<peerlist_entry>& white);
     void get_peerlist(std::vector<peerlist_entry>& gray, std::vector<peerlist_entry>& white);
-    size_t get_zone_count() const { return m_network_zones.size(); }
 
     void change_max_out_public_peers(size_t count);
     uint32_t get_max_out_public_peers() const;
@@ -330,6 +335,7 @@ namespace nodetool
     virtual void callback(p2p_connection_context& context);
     //----------------- i_p2p_endpoint -------------------------------------------------------------
     virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections);
+    virtual epee::net_utils::zone send_txs(std::vector<cryptonote::blobdata> txs, const epee::net_utils::zone origin, const boost::uuids::uuid& source, const bool pad_txs);
     virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context);
     virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context);
     virtual bool drop_connection(const epee::net_utils::connection_context_base& context);

src/p2p/net_node.inl

@@ -383,6 +383,9 @@ namespace nodetool
     m_offline = command_line::get_arg(vm, cryptonote::arg_offline);
     m_use_ipv6 = command_line::get_arg(vm, arg_p2p_use_ipv6);
     m_require_ipv4 = command_line::get_arg(vm, arg_p2p_require_ipv4);
+    public_zone.m_notifier = cryptonote::levin::notify{
+      public_zone.m_net_server.get_io_service(), public_zone.m_net_server.get_config_shared(), nullptr
+    };
 
     if (command_line::has_arg(vm, arg_p2p_add_peer))
     {
@@ -462,6 +465,7 @@ namespace nodetool
       return false;
 
 
+    epee::byte_slice noise = nullptr;
     auto proxies = get_proxies(vm);
     if (!proxies)
       return false;
@@ -479,6 +483,20 @@ namespace nodetool
 
       if (!set_max_out_peers(zone, proxy.max_connections))
         return false;
+
+      epee::byte_slice this_noise = nullptr;
+      if (proxy.noise)
+      {
+        static_assert(sizeof(epee::levin::bucket_head2) < CRYPTONOTE_NOISE_BYTES, "noise bytes too small");
+        if (noise.empty())
+          noise = epee::levin::make_noise_notify(CRYPTONOTE_NOISE_BYTES);
+
+        this_noise = noise.clone();
+      }
+
+      zone.m_notifier = cryptonote::levin::notify{
+        zone.m_net_server.get_io_service(), zone.m_net_server.get_config_shared(), std::move(this_noise)
+      };
     }
 
     for (const auto& zone : m_network_zones)
@@ -494,6 +512,7 @@ namespace nodetool
     if (!inbounds)
       return false;
 
+    const std::size_t tx_relay_zones = m_network_zones.size();
     for (auto& inbound : *inbounds)
     {
       network_zone& zone = add_zone(inbound.our_address.get_zone());
@@ -504,6 +523,12 @@ namespace nodetool
         return false;
       }
 
+      if (zone.m_connect == nullptr && tx_relay_zones <= 1)
+      {
+        MERROR("Listed --" << arg_anonymous_inbound.name << " without listing any --" << arg_proxy.name << ". The latter is necessary for sending origin txes over anonymity networks");
+        return false;
+      }
+
       zone.m_bind_ip = std::move(inbound.local_ip);
       zone.m_port = std::move(inbound.local_port);
       zone.m_net_server.set_default_remote(std::move(inbound.default_remote));
@@ -1266,6 +1291,7 @@ namespace nodetool
     ape.first_seen = first_seen_stamp ? first_seen_stamp : time(nullptr);
 
     zone.m_peerlist.append_with_peer_anchor(ape);
+    zone.m_notifier.new_out_connection();
 
     LOG_DEBUG_CC(*con, "CONNECTION HANDSHAKED OK.");
     return true;
@@ -2000,6 +2026,61 @@ namespace nodetool
   }
   //-----------------------------------------------------------------------------------
   template<class t_payload_net_handler>
+  epee::net_utils::zone node_server<t_payload_net_handler>::send_txs(std::vector<cryptonote::blobdata> txs, const epee::net_utils::zone origin, const boost::uuids::uuid& source, const bool pad_txs)
+  {
+    namespace enet = epee::net_utils;
+
+    const auto send = [&txs, &source, pad_txs] (std::pair<const enet::zone, network_zone>& network)
+    {
+      if (network.second.m_notifier.send_txs(std::move(txs), source, (pad_txs || network.first != enet::zone::public_)))
+        return network.first;
+      return enet::zone::invalid;
+    };
+
+    if (m_network_zones.empty())
+      return enet::zone::invalid;
+
+    if (origin != enet::zone::invalid)
+      return send(*m_network_zones.begin()); // send all txs received via p2p over public network
+
+    if (m_network_zones.size() <= 2)
+      return send(*m_network_zones.rbegin()); // see static asserts below; sends over anonymity network iff enabled
+
+    /* These checks are to ensure that i2p is highest priority if multiple
+       zones are selected. Make sure to update logic if the values cannot be
+       in the same relative order. `m_network_zones` must be sorted map too. */
+    static_assert(std::is_same<std::underlying_type<enet::zone>::type, std::uint8_t>{}, "expected uint8_t zone");
+    static_assert(unsigned(enet::zone::invalid) == 0, "invalid expected to be 0");
+    static_assert(unsigned(enet::zone::public_) == 1, "public_ expected to be 1");
+    static_assert(unsigned(enet::zone::i2p) == 2, "i2p expected to be 2");
+    static_assert(unsigned(enet::zone::tor) == 3, "tor expected to be 3");
+
+    // check for anonymity networks with noise and connections
+    for (auto network = ++m_network_zones.begin(); network != m_network_zones.end(); ++network)
+    {
+      if (enet::zone::tor < network->first)
+        break; // unknown network
+
+      const auto status = network->second.m_notifier.get_status();
+      if (status.has_noise && status.connections_filled)
+        return send(*network);
+    }
+
+    // use the anonymity network with outbound support
+    for (auto network = ++m_network_zones.begin(); network != m_network_zones.end(); ++network)
+    {
+      if (enet::zone::tor < network->first)
+        break; // unknown network
+
+      if (network->second.m_connect)
+        return send(*network);
+    }
+
+    // configuration should not allow this scenario
+    return enet::zone::invalid;
+  }
+  //-----------------------------------------------------------------------------------
+  template<class t_payload_net_handler>
   void node_server<t_payload_net_handler>::callback(p2p_connection_context& context)
   {
     m_payload_handler.on_callback(context);

src/p2p/net_node_common.h

@@ -33,6 +33,8 @@
 #include <boost/uuid/uuid.hpp>
 #include <utility>
 #include <vector>
+#include "cryptonote_basic/blobdatatype.h"
+#include "net/enums.h"
 #include "net/net_utils_base.h"
 #include "p2p_protocol_defs.h"
 
@@ -46,12 +48,12 @@ namespace nodetool
   struct i_p2p_endpoint
   {
     virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> connections)=0;
+    virtual epee::net_utils::zone send_txs(std::vector<cryptonote::blobdata> txs, const epee::net_utils::zone origin, const boost::uuids::uuid& source, const bool pad_txs)=0;
     virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context)=0;
     virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context)=0;
     virtual bool drop_connection(const epee::net_utils::connection_context_base& context)=0;
     virtual void request_callback(const epee::net_utils::connection_context_base& context)=0;
     virtual uint64_t get_public_connections_count()=0;
-    virtual size_t get_zone_count() const=0;
     virtual void for_each_connection(std::function<bool(t_connection_context&, peerid_type, uint32_t)> f)=0;
     virtual bool for_connection(const boost::uuids::uuid&, std::function<bool(t_connection_context&, peerid_type, uint32_t)> f)=0;
     virtual bool block_host(const epee::net_utils::network_address &address, time_t seconds = 0)=0;
@@ -71,6 +73,10 @@ namespace nodetool
     {
       return false;
     }
+    virtual epee::net_utils::zone send_txs(std::vector<cryptonote::blobdata> txs, const epee::net_utils::zone origin, const boost::uuids::uuid& source, const bool pad_txs)
+    {
+      return epee::net_utils::zone::invalid;
+    }
     virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context)
     {
       return false;
@@ -96,11 +102,6 @@ namespace nodetool
       return false;
     }
 
-    virtual size_t get_zone_count() const
-    {
-      return 0;
-    }
-
     virtual uint64_t get_public_connections_count()    
     {
       return false;

tests/fuzz/levin.cpp

@@ -149,11 +149,11 @@ namespace
     }
 
     // Implement epee::net_utils::i_service_endpoint interface
-    virtual bool do_send(const void* ptr, size_t cb)
+    virtual bool do_send(epee::byte_slice message)
     {
       m_send_counter.inc();
       boost::unique_lock<boost::mutex> lock(m_mutex);
-      m_last_send_data.append(reinterpret_cast<const char*>(ptr), cb);
+      m_last_send_data.append(reinterpret_cast<const char*>(message.data()), message.size());
       return m_send_return;
     }
 

tests/unit_tests/CMakeLists.txt

@@ -56,6 +56,7 @@ set(unit_tests_sources
   hmac_keccak.cpp
   http.cpp
   keccak.cpp
+  levin.cpp
   logging.cpp
   long_term_block_weight.cpp
   lmdb.cpp

tests/unit_tests/epee_levin_protocol_handler_async.cpp

@@ -140,12 +140,12 @@ namespace
     }
 
     // Implement epee::net_utils::i_service_endpoint interface
-    virtual bool do_send(const void* ptr, size_t cb)
+    virtual bool do_send(epee::byte_slice message)
     {
       //std::cout << "test_connection::do_send()" << std::endl;
       m_send_counter.inc();
       boost::unique_lock<boost::mutex> lock(m_mutex);
-      m_last_send_data.append(reinterpret_cast<const char*>(ptr), cb);
+      m_last_send_data.append(reinterpret_cast<const char*>(message.data()), message.size());
       return m_send_return;
     }
 
@@ -367,8 +367,8 @@ TEST_F(positive_test_connection_to_levin_protocol_handler_calls, handler_process
   // Parse send data
   std::string send_data = conn->last_send_data();
   epee::levin::bucket_head2 resp_head;
-  resp_head = *reinterpret_cast<const epee::levin::bucket_head2*>(send_data.data());
   ASSERT_LT(sizeof(resp_head), send_data.size());
+  std::memcpy(std::addressof(resp_head), send_data.data(), sizeof(resp_head));
   std::string out_data = send_data.substr(sizeof(resp_head));
 
   // Check sent response
@@ -425,6 +425,95 @@ TEST_F(positive_test_connection_to_levin_protocol_handler_calls, handler_process
   ASSERT_EQ(3, m_commands_handler.callback_counter());
 }
 
+TEST_F(positive_test_connection_to_levin_protocol_handler_calls, handler_processes_handle_read_as_dummy)
+{
+  // Setup
+  const int expected_command = 4673261;
+  const std::string in_data(256, 'e');
+
+  const epee::byte_slice noise = epee::levin::make_noise_notify(1024);
+  const epee::byte_slice notify = epee::levin::make_notify(expected_command, epee::strspan<std::uint8_t>(in_data));
+
+  test_connection_ptr conn = create_connection();
+
+  // Test
+  ASSERT_TRUE(conn->m_protocol_handler.handle_recv(noise.data(), noise.size()));
+
+  // Check connection and levin_commands_handler states
+  ASSERT_EQ(0u, m_commands_handler.notify_counter());
+  ASSERT_EQ(0u, m_commands_handler.invoke_counter());
+  ASSERT_EQ(-1, m_commands_handler.last_command());
+  ASSERT_TRUE(m_commands_handler.last_in_buf().empty());
+  ASSERT_EQ(0u, conn->send_counter());
+  ASSERT_TRUE(conn->last_send_data().empty());
+
+
+  ASSERT_TRUE(conn->m_protocol_handler.handle_recv(notify.data(), notify.size()));
+
+  // Check connection and levin_commands_handler states
+  ASSERT_EQ(1u, m_commands_handler.notify_counter());
+  ASSERT_EQ(0u, m_commands_handler.invoke_counter());
+  ASSERT_EQ(expected_command, m_commands_handler.last_command());
+  ASSERT_EQ(in_data, m_commands_handler.last_in_buf());
+  ASSERT_EQ(0u, conn->send_counter());
+  ASSERT_TRUE(conn->last_send_data().empty());
+}
+
+TEST_F(positive_test_connection_to_levin_protocol_handler_calls, handler_processes_handle_read_as_fragment)
+{
+  // Setup
+  const int expected_command = 4673261;
+  const int expected_fragmented_command = 46732;
+  const std::string in_data(256, 'e');
+  std::string in_fragmented_data(1024 * 4, 'c');
+
+  const epee::byte_slice noise = epee::levin::make_noise_notify(1024);
+  const epee::byte_slice notify = epee::levin::make_notify(expected_command, epee::strspan<std::uint8_t>(in_data));
+  epee::byte_slice fragmented = epee::levin::make_fragmented_notify(noise, expected_fragmented_command, epee::strspan<std::uint8_t>(in_fragmented_data));
+
+  EXPECT_EQ(5u, fragmented.size() / 1024);
+  EXPECT_EQ(0u, fragmented.size() % 1024);
+
+  test_connection_ptr conn = create_connection();
+
+  while (!fragmented.empty())
+  {
+    if ((fragmented.size() / 1024) % 2 == 1)
+    {
+      ASSERT_TRUE(conn->m_protocol_handler.handle_recv(notify.data(), notify.size()));
+    }
+
+    ASSERT_EQ(3u - (fragmented.size() / 2048), m_commands_handler.notify_counter());
+    ASSERT_EQ(0u, m_commands_handler.invoke_counter());
+    ASSERT_EQ(expected_command, m_commands_handler.last_command());
+    ASSERT_EQ(in_data, m_commands_handler.last_in_buf());
+    ASSERT_EQ(0u, conn->send_counter());
+    ASSERT_TRUE(conn->last_send_data().empty());
+
+    epee::byte_slice next = fragmented.take_slice(1024);
+    ASSERT_TRUE(conn->m_protocol_handler.handle_recv(next.data(), next.size()));
+  }
+
+  in_fragmented_data.resize(((1024 - sizeof(epee::levin::bucket_head2)) * 5) - sizeof(epee::levin::bucket_head2)); // add padding zeroes
+  ASSERT_EQ(4u, m_commands_handler.notify_counter());
+  ASSERT_EQ(0u, m_commands_handler.invoke_counter());
+  ASSERT_EQ(expected_fragmented_command, m_commands_handler.last_command());
+  ASSERT_EQ(in_fragmented_data, m_commands_handler.last_in_buf());
+  ASSERT_EQ(0u, conn->send_counter());
+  ASSERT_TRUE(conn->last_send_data().empty());
+
+
+  ASSERT_TRUE(conn->m_protocol_handler.handle_recv(notify.data(), notify.size()));
+
+  ASSERT_EQ(5u, m_commands_handler.notify_counter());
+  ASSERT_EQ(0u, m_commands_handler.invoke_counter());
+  ASSERT_EQ(expected_command, m_commands_handler.last_command());
+  ASSERT_EQ(in_data, m_commands_handler.last_in_buf());
+  ASSERT_EQ(0u, conn->send_counter());
+  ASSERT_TRUE(conn->last_send_data().empty());
+}
+
+
 TEST_F(test_levin_protocol_handler__hanle_recv_with_invalid_data, handles_big_packet_1)
 {
   std::string buf("yyyyyy");
@@ -534,3 +623,19 @@ TEST_F(test_levin_protocol_handler__hanle_recv_with_invalid_data, handles_unexpe
 
   ASSERT_FALSE(m_conn->m_protocol_handler.handle_recv(m_buf.data(), m_buf.size()));
 }
+
+TEST_F(test_levin_protocol_handler__hanle_recv_with_invalid_data, handles_short_fragment)
+{
+  m_req_head.m_cb = 1;
+  m_req_head.m_flags = LEVIN_PACKET_BEGIN;
+  m_req_head.m_command = 0;
+  m_in_data.resize(1);
+  prepare_buf();
+
+  ASSERT_TRUE(m_conn->m_protocol_handler.handle_recv(m_buf.data(), m_buf.size()));
+
+  m_req_head.m_flags = LEVIN_PACKET_END;
+  prepare_buf();
+
+  ASSERT_FALSE(m_conn->m_protocol_handler.handle_recv(m_buf.data(), m_buf.size()));
+}

tests/unit_tests/epee_utils.cpp

@@ -44,6 +44,7 @@
 
 #include "boost/archive/portable_binary_iarchive.hpp"
 #include "boost/archive/portable_binary_oarchive.hpp"
+#include "byte_slice.h"
 #include "hex.h"
 #include "net/net_utils_base.h"
 #include "net/local_ip.h"
@@ -375,6 +376,438 @@ TEST(Span, ToMutSpan)
   EXPECT_EQ((std::vector<unsigned>{1, 2, 3, 4}), mut);
 }
 
+TEST(ByteSlice, Construction)
+{
+  EXPECT_TRUE(std::is_default_constructible<epee::byte_slice>());
+  EXPECT_TRUE(std::is_move_constructible<epee::byte_slice>());
+  EXPECT_FALSE(std::is_copy_constructible<epee::byte_slice>());
+  EXPECT_TRUE(std::is_move_assignable<epee::byte_slice>());
+  EXPECT_FALSE(std::is_copy_assignable<epee::byte_slice>());
+}
+
+TEST(ByteSlice, NoExcept)
+{
+  EXPECT_TRUE(std::is_nothrow_default_constructible<epee::byte_slice>());
+  EXPECT_TRUE(std::is_nothrow_move_constructible<epee::byte_slice>());
+  EXPECT_TRUE(std::is_nothrow_move_assignable<epee::byte_slice>());
+
+  epee::byte_slice lvalue{};
+  const epee::byte_slice clvalue{};
+
+  EXPECT_TRUE(noexcept(lvalue.clone()));
+  EXPECT_TRUE(noexcept(clvalue.clone()));
+
+  EXPECT_TRUE(noexcept(lvalue.begin()));
+  EXPECT_TRUE(noexcept(clvalue.begin()));
+  EXPECT_TRUE(noexcept(lvalue.end()));
+  EXPECT_TRUE(noexcept(clvalue.end()));
+
+  EXPECT_TRUE(noexcept(lvalue.cbegin()));
+  EXPECT_TRUE(noexcept(clvalue.cbegin()));
+  EXPECT_TRUE(noexcept(lvalue.cend()));
+  EXPECT_TRUE(noexcept(clvalue.cend()));
+
+  EXPECT_TRUE(noexcept(lvalue.empty()));
+  EXPECT_TRUE(noexcept(clvalue.empty()));
+
+  EXPECT_TRUE(noexcept(lvalue.data()));
+  EXPECT_TRUE(noexcept(clvalue.data()));
+  EXPECT_TRUE(noexcept(lvalue.size()));
+  EXPECT_TRUE(noexcept(clvalue.size()));
+
+  EXPECT_TRUE(noexcept(lvalue.remove_prefix(0)));
+  EXPECT_TRUE(noexcept(lvalue.take_slice(0)));
+}
+
+TEST(ByteSlice, Empty)
+{
+  epee::byte_slice slice{};
+
+  EXPECT_EQ(slice.begin(), slice.end());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_TRUE(slice.empty());
+  EXPECT_EQ(0u, slice.size());
+  EXPECT_EQ(slice.begin(), slice.data());
+
+  EXPECT_EQ(0u, slice.get_slice(0, 0).size());
+  EXPECT_THROW(slice.get_slice(0, 1), std::out_of_range);
+  EXPECT_EQ(0u, slice.remove_prefix(1));
+  EXPECT_EQ(0u, slice.take_slice(1).size());
+}
+
+TEST(ByteSlice, CopySpans)
+{
+  const epee::span<const std::uint8_t> part1 = epee::as_byte_span("this is part1");
+  const epee::span<const std::uint8_t> part2 = epee::as_byte_span("then part2");
+  const epee::span<const std::uint8_t> part3 = epee::as_byte_span("finally part3");
+
+  const epee::byte_slice slice{part1, part2, part3};
+
+  EXPECT_NE(nullptr, slice.begin());
+  EXPECT_NE(nullptr, slice.end());
+  EXPECT_NE(slice.begin(), slice.end());
+  EXPECT_NE(slice.cbegin(), slice.cend());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+  ASSERT_EQ(slice.size(), std::size_t(slice.end() - slice.begin()));
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_EQ(slice.begin(), slice.data());
+  ASSERT_EQ(part1.size() + part2.size() + part3.size(), slice.size());
+  EXPECT_TRUE(
+    boost::range::equal(
+      part1, boost::make_iterator_range(slice.begin(), slice.begin() + part1.size())
+    )
+  );
+  EXPECT_TRUE(
+    boost::range::equal(
+      part2, boost::make_iterator_range(slice.begin() + part1.size(), slice.end() - part3.size())
+    )
+  );
+  EXPECT_TRUE(
+    boost::range::equal(
+      part3, boost::make_iterator_range(slice.end() - part3.size(), slice.end())
+    )
+  );
+}
+
+TEST(ByteSlice, AdaptString)
+{
+  static constexpr const char base_string[] = "this is an example message";
+  std::string adapted = base_string;
+
+  const epee::span<const uint8_t> original = epee::to_byte_span(epee::to_span(adapted));
+  const epee::byte_slice slice{std::move(adapted)};
+
+  EXPECT_EQ(original.begin(), slice.begin());
+  EXPECT_EQ(original.cbegin(), slice.cbegin());
+  EXPECT_EQ(original.end(), slice.end());
+  EXPECT_EQ(original.cend(), slice.cend());
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_EQ(original.data(), slice.data());
+  EXPECT_EQ(original.size(), slice.size());
+  EXPECT_TRUE(boost::range::equal(boost::string_ref{base_string}, slice));
+}
+
+TEST(ByteSlice, EmptyAdaptString)
+{
+  epee::byte_slice slice{std::string{}};
+
+  EXPECT_EQ(slice.begin(), slice.end());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_TRUE(slice.empty());
+  EXPECT_EQ(0u, slice.size());
+  EXPECT_EQ(slice.begin(), slice.data());
+
+  EXPECT_EQ(0u, slice.get_slice(0, 0).size());
+  EXPECT_THROW(slice.get_slice(0, 1), std::out_of_range);
+  EXPECT_EQ(0u, slice.remove_prefix(1));
+  EXPECT_EQ(0u, slice.take_slice(1).size());
+}
+
+TEST(ByteSlice, AdaptVector)
+{
+  static constexpr const char base_string[] = "this is an example message";
+  std::vector<std::uint8_t> adapted(sizeof(base_string));
+
+  ASSERT_EQ(sizeof(base_string), adapted.size());
+  std::memcpy(adapted.data(), base_string, sizeof(base_string));
+
+  const epee::span<const uint8_t> original = epee::to_span(adapted);
+  const epee::byte_slice slice{std::move(adapted)};
+
+  EXPECT_EQ(sizeof(base_string), original.size());
+
+  EXPECT_EQ(original.begin(), slice.begin());
+  EXPECT_EQ(original.cbegin(), slice.cbegin());
+  EXPECT_EQ(original.end(), slice.end());
+  EXPECT_EQ(original.cend(), slice.cend());
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_EQ(original.data(), slice.data());
+  EXPECT_EQ(original.size(), slice.size());
+  EXPECT_TRUE(boost::range::equal(base_string, slice));
+}
+
+TEST(ByteSlice, EmptyAdaptVector)
+{
+  epee::byte_slice slice{std::vector<std::uint8_t>{}};
+
+  EXPECT_EQ(slice.begin(), slice.end());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_TRUE(slice.empty());
+  EXPECT_EQ(0u, slice.size());
+  EXPECT_EQ(slice.begin(), slice.data());
+
+  EXPECT_EQ(0u, slice.get_slice(0, 0).size());
+  EXPECT_THROW(slice.get_slice(0, 1), std::out_of_range);
+  EXPECT_EQ(0u, slice.remove_prefix(1));
+  EXPECT_EQ(0u, slice.take_slice(1).size());
+}
+
+TEST(ByteSlice, Move)
+{
+  static constexpr const char base_string[] = "another example message";
+
+  epee::byte_slice slice{epee::as_byte_span(base_string)};
+  EXPECT_TRUE(boost::range::equal(base_string, slice));
+
+  const epee::span<const std::uint8_t> original = epee::to_span(slice);
+  epee::byte_slice moved{std::move(slice)};
+  EXPECT_TRUE(boost::range::equal(base_string, moved));
+
+  EXPECT_EQ(slice.begin(), slice.end());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_EQ(original.begin(), moved.begin());
+  EXPECT_EQ(moved.begin(), moved.cbegin());
+  EXPECT_EQ(original.end(), moved.end());
+  EXPECT_EQ(moved.end(), moved.cend());
+
+  EXPECT_TRUE(slice.empty());
+  EXPECT_EQ(slice.begin(), slice.data());
+  EXPECT_EQ(0u, slice.size());
+
+  EXPECT_FALSE(moved.empty());
+  EXPECT_EQ(moved.begin(), moved.data());
+  EXPECT_EQ(original.size(), moved.size());
+
+  slice = std::move(moved);
+  EXPECT_TRUE(boost::range::equal(base_string, slice));
+
+  EXPECT_EQ(original.begin(), slice.begin());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(original.end(), slice.end());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_EQ(slice.begin(), slice.data());
+  EXPECT_EQ(original.size(), slice.size());
+
+  EXPECT_TRUE(moved.empty());
+  EXPECT_EQ(moved.begin(), moved.data());
+  EXPECT_EQ(0u, moved.size());
+}
+
+TEST(ByteSlice, Clone)
+{
+  static constexpr const char base_string[] = "another example message";
+
+  const epee::byte_slice slice{epee::as_byte_span(base_string)};
+  EXPECT_TRUE(boost::range::equal(base_string, slice));
+
+  const epee::byte_slice clone{slice.clone()};
+  EXPECT_TRUE(boost::range::equal(base_string, clone));
+
+  EXPECT_EQ(slice.begin(), clone.begin());
+  EXPECT_EQ(slice.cbegin(), clone.cbegin());
+  EXPECT_EQ(slice.end(), clone.end());
+  EXPECT_EQ(slice.cend(), clone.cend());
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_FALSE(clone.empty());
+  EXPECT_EQ(slice.cbegin(), slice.data());
+  EXPECT_EQ(slice.data(), clone.data());
+  EXPECT_EQ(sizeof(base_string), slice.size());
+  EXPECT_EQ(slice.size(), clone.size());
+}
+
+TEST(ByteSlice, RemovePrefix)
+{
+  static constexpr const char base_string[] = "another example message";
+  static constexpr std::size_t remove_size = sizeof("another");
+  static constexpr std::size_t remaining = sizeof(base_string) - remove_size;
+
+  epee::byte_slice slice{epee::as_byte_span(base_string)};
+  EXPECT_TRUE(boost::range::equal(base_string, slice));
+
+  const epee::span<const std::uint8_t> original = epee::to_span(slice);
+  EXPECT_EQ(remove_size, slice.remove_prefix(remove_size));
+
+  EXPECT_EQ(original.begin() + remove_size, slice.begin());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(original.end(), slice.end());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_EQ(slice.cbegin(), slice.data());
+  EXPECT_EQ(remaining, slice.size());
+
+  // touch original pointers to check "free" status
+  EXPECT_TRUE(boost::range::equal(base_string, original));
+
+  EXPECT_EQ(remaining, slice.remove_prefix(remaining + 1));
+
+  EXPECT_EQ(slice.begin(), slice.end());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_TRUE(slice.empty());
+  EXPECT_EQ(slice.cbegin(), slice.data());
+  EXPECT_EQ(0, slice.size());
+}
+
+TEST(ByteSlice, TakeSlice)
+{
+  static constexpr const char base_string[] = "another example message";
+  static constexpr std::size_t remove_size = sizeof("another");
+  static constexpr std::size_t remaining = sizeof(base_string) - remove_size;
+
+  epee::byte_slice slice{epee::as_byte_span(base_string)};
+  EXPECT_TRUE(boost::range::equal(base_string, slice));
+
+  const epee::span<const std::uint8_t> original = epee::to_span(slice);
+  const epee::byte_slice slice2 = slice.take_slice(remove_size);
+
+  EXPECT_EQ(original.begin() + remove_size, slice.begin());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(original.end(), slice.end());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_EQ(original.begin(), slice2.begin());
+  EXPECT_EQ(slice2.begin(), slice2.cbegin());
+  EXPECT_EQ(original.begin() + remove_size, slice2.end());
+  EXPECT_EQ(slice2.end(), slice2.cend());
+
+  EXPECT_FALSE(slice.empty());
+  EXPECT_EQ(slice.cbegin(), slice.data());
+  EXPECT_EQ(remaining, slice.size());
+
+  EXPECT_FALSE(slice2.empty());
+  EXPECT_EQ(slice2.cbegin(), slice2.data());
+  EXPECT_EQ(remove_size, slice2.size());
+
+  // touch original pointers to check "free" status
+  EXPECT_TRUE(boost::range::equal(base_string, original));
+
+  const epee::byte_slice slice3 = slice.take_slice(remaining + 1);
+
+  EXPECT_EQ(slice.begin(), slice.end());
+  EXPECT_EQ(slice.begin(), slice.cbegin());
+  EXPECT_EQ(slice.end(), slice.cend());
+
+  EXPECT_EQ(original.begin(), slice2.begin());
+  EXPECT_EQ(slice2.begin(), slice2.cbegin());
+  EXPECT_EQ(original.begin() + remove_size, slice2.end());
+  EXPECT_EQ(slice2.end(), slice2.cend());
+
+  EXPECT_EQ(slice2.end(), slice3.begin());
+  EXPECT_EQ(slice3.begin(), slice3.cbegin());
+  EXPECT_EQ(original.end(), slice3.end());
+  EXPECT_EQ(slice3.end(), slice3.cend());
+
+  EXPECT_TRUE(slice.empty());
+  EXPECT_EQ(slice.cbegin(), slice.data());
+  EXPECT_EQ(0, slice.size());
+
+  EXPECT_FALSE(slice2.empty());
+  EXPECT_EQ(slice2.cbegin(), slice2.data());
+  EXPECT_EQ(remove_size, slice2.size());
+
+  EXPECT_FALSE(slice3.empty());
+  EXPECT_EQ(slice3.cbegin(), slice3.data());
+  EXPECT_EQ(remaining, slice3.size());
+
+  // touch original pointers to check "free" status
+  slice = nullptr;
+  EXPECT_TRUE(boost::range::equal(base_string, original));
+}
+
+TEST(ByteSlice, GetSlice)
+{
+  static constexpr const char base_string[] = "another example message";
+  static constexpr std::size_t get_size = sizeof("another");
+  static constexpr std::size_t get2_size = sizeof(base_string) - get_size;
+
+  epee::span<const std::uint8_t> original{};
+  epee::byte_slice slice2{};
+  epee::byte_slice slice3{};
+
+  // make sure get_slice increments ref count
+  {
+    const epee::byte_slice slice{epee::as_byte_span(base_string)};
+    EXPECT_TRUE(boost::range::equal(base_string, slice));
+
+    original = epee::to_span(slice);
+    slice2 = slice.get_slice(0, get_size);
+
+    EXPECT_EQ(original.begin(), slice.begin());
+    EXPECT_EQ(slice.begin(), slice.cbegin());
+    EXPECT_EQ(original.end(), slice.end());
+    EXPECT_EQ(slice.end(), slice.cend());
+
+    EXPECT_EQ(original.begin(), slice2.begin());
+    EXPECT_EQ(slice2.begin(), slice2.cbegin());
+    EXPECT_EQ(original.begin() + get_size, slice2.end());
+    EXPECT_EQ(slice2.end(), slice2.cend());
+
+    EXPECT_FALSE(slice.empty());
+    EXPECT_EQ(slice.cbegin(), slice.data());
+    EXPECT_EQ(original.size(), slice.size());
+
+    EXPECT_FALSE(slice2.empty());
+    EXPECT_EQ(slice2.cbegin(), slice2.data());
+    EXPECT_EQ(get_size, slice2.size());
+
+    // touch original pointers to check "free" status
+    EXPECT_TRUE(boost::range::equal(base_string, original));
+
+    slice3 = slice.get_slice(get_size, sizeof(base_string));
+
+    EXPECT_EQ(original.begin(), slice.begin());
+    EXPECT_EQ(slice.begin(), slice.cbegin());
+    EXPECT_EQ(original.end(), slice.end());
+    EXPECT_EQ(slice.end(), slice.cend());
+
+    EXPECT_EQ(original.begin(), slice2.begin());
+    EXPECT_EQ(slice2.begin(), slice2.cbegin());
+    EXPECT_EQ(original.begin() + get_size, slice2.end());
+    EXPECT_EQ(slice2.end(), slice2.cend());
+
+    EXPECT_EQ(slice2.end(), slice3.begin());
+    EXPECT_EQ(slice3.begin(), slice3.cbegin());
+    EXPECT_EQ(original.end(), slice3.end());
+    EXPECT_EQ(slice3.end(), slice3.cend());
+
+    EXPECT_FALSE(slice.empty());
+    EXPECT_EQ(slice.cbegin(), slice.data());
+    EXPECT_EQ(original.size(), slice.size());
+
+    EXPECT_FALSE(slice2.empty());
+    EXPECT_EQ(slice2.cbegin(), slice2.data());
+    EXPECT_EQ(get_size, slice2.size());
+
+    EXPECT_FALSE(slice3.empty());
+    EXPECT_EQ(slice3.cbegin(), slice3.data());
+    EXPECT_EQ(get2_size, slice3.size());
+
+    EXPECT_THROW(slice.get_slice(1, 0), std::out_of_range);
+    EXPECT_THROW(slice.get_slice(0, sizeof(base_string) + 1), std::out_of_range);
+    EXPECT_THROW(slice.get_slice(sizeof(base_string) + 1, sizeof(base_string) + 1), std::out_of_range);
+    EXPECT_TRUE(slice.get_slice(sizeof(base_string), sizeof(base_string)).empty());
+
+    EXPECT_EQ(original.begin(), slice.begin());
+    EXPECT_EQ(slice.begin(), slice.cbegin());
+    EXPECT_EQ(original.end(), slice.end());
+    EXPECT_EQ(slice.end(), slice.cend());
+
+    EXPECT_FALSE(slice.empty());
+    EXPECT_EQ(slice.cbegin(), slice.data());
+    EXPECT_EQ(original.size(), slice.size());
+  }
+
+  // touch original pointers to check "free" status
+  EXPECT_TRUE(boost::range::equal(base_string, original));
+}
+
 TEST(ToHex, String)
 {
   EXPECT_TRUE(epee::to_hex::string(nullptr).empty());

tests/unit_tests/levin.cpp

@@ -0,0 +1,586 @@
+// Copyright (c) 2019, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <algorithm>
+#include <boost/uuid/nil_generator.hpp>
+#include <boost/uuid/random_generator.hpp>
+#include <boost/uuid/uuid.hpp>
+#include <cstring>
+#include <gtest/gtest.h>
+#include <limits>
+#include <set>
+
+#include "byte_slice.h"
+#include "crypto/crypto.h"
+#include "cryptonote_basic/connection_context.h"
+#include "cryptonote_protocol/cryptonote_protocol_defs.h"
+#include "cryptonote_protocol/levin_notify.h"
+#include "int-util.h"
+#include "p2p/net_node.h"
+#include "net/dandelionpp.h"
+#include "net/levin_base.h"
+#include "span.h"
+
+namespace
+{
+    class test_endpoint final : public epee::net_utils::i_service_endpoint
+    {
+        boost::asio::io_service& io_service_;
+        std::size_t ref_count_;
+
+        virtual bool do_send(epee::byte_slice message) override final
+        {
+            send_queue_.push_back(std::move(message));
+            return true;
+        }
+
+        virtual bool close() override final
+        {
+            return true;
+        }
+
+        virtual bool send_done() override final
+        {
+            throw std::logic_error{"send_done not implemented"};
+        }
+
+        virtual bool call_run_once_service_io() override final
+        {
+            return io_service_.run_one();
+        }
+
+        virtual bool request_callback() override final
+        {
+            throw std::logic_error{"request_callback not implemented"};
+        }
+
+        virtual boost::asio::io_service& get_io_service() override final
+        {
+            return io_service_;
+        }
+
+        virtual bool add_ref() override final
+        {
+            ++ref_count_;
+            return true;
+        }
+
+        virtual bool release() override final
+        {
+            --ref_count_;
+            return true;
+        }
+
+    public:
+        test_endpoint(boost::asio::io_service& io_service)
+          : epee::net_utils::i_service_endpoint(),
+	          io_service_(io_service),
+            ref_count_(0),
+            send_queue_()
+        {}
+
+        virtual ~test_endpoint() noexcept(false) override final
+        {
+            EXPECT_EQ(0u, ref_count_);
+        }
+
+        std::deque<epee::byte_slice> send_queue_;
+    };
+
+    class test_connection
+    {
+        test_endpoint endpoint_;
+        cryptonote::levin::detail::p2p_context context_;
+        epee::levin::async_protocol_handler<cryptonote::levin::detail::p2p_context> handler_;
+
+    public:
+        test_connection(boost::asio::io_service& io_service, cryptonote::levin::connections& connections, boost::uuids::random_generator& random_generator)
+          : context_(),
+            endpoint_(io_service),
+            handler_(std::addressof(endpoint_), connections, context_)
+        {
+            const_cast<boost::uuids::uuid&>(context_.m_connection_id) = random_generator();
+            handler_.after_init_connection();
+        }
+
+        //\return Number of messages processed
+        std::size_t process_send_queue()
+        {
+            std::size_t count = 0;
+            for ( ; !endpoint_.send_queue_.empty(); ++count, endpoint_.send_queue_.pop_front())
+            {
+                // invalid messages shoudn't be possible in this test;
+                EXPECT_TRUE(handler_.handle_recv(endpoint_.send_queue_.front().data(), endpoint_.send_queue_.front().size()));
+            }
+            return count;
+        }
+
+        const boost::uuids::uuid& get_id() const noexcept
+        {
+            return context_.m_connection_id;
+        }
+    };
+
+    struct received_message
+    {
+        boost::uuids::uuid connection;
+        int command;
+        std::string payload;
+    };
+
+    class test_receiver : public epee::levin::levin_commands_handler<cryptonote::levin::detail::p2p_context>
+    {
+        std::deque<received_message> invoked_;
+        std::deque<received_message> notified_;
+
+        template<typename T>
+        static std::pair<boost::uuids::uuid, typename T::request> get_message(std::deque<received_message>& queue)
+        {
+            if (queue.empty())
+                throw std::logic_error{"Queue has no received messges"};
+
+            if (queue.front().command != T::ID)
+                throw std::logic_error{"Unexpected ID at front of message queue"};
+
+            epee::serialization::portable_storage storage{};
+            if(!storage.load_from_binary(epee::strspan<std::uint8_t>(queue.front().payload)))
+                throw std::logic_error{"Unable to parse epee binary format"};
+
+            typename T::request request{};
+            if (!request.load(storage))
+                throw std::logic_error{"Unable to load into expected request"};
+
+            boost::uuids::uuid connection = queue.front().connection;
+            queue.pop_front();
+            return {connection, std::move(request)};
+        }
+
+        virtual int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, cryptonote::levin::detail::p2p_context& context) override final
+        {
+            buff_out.clear();
+            invoked_.push_back(
+                {context.m_connection_id, command, std::string{reinterpret_cast<const char*>(in_buff.data()), in_buff.size()}}
+            );
+            return 1;
+        }
+
+        virtual int notify(int command, const epee::span<const uint8_t> in_buff, cryptonote::levin::detail::p2p_context& context) override final
+        {
+            notified_.push_back(
+                {context.m_connection_id, command, std::string{reinterpret_cast<const char*>(in_buff.data()), in_buff.size()}}
+            );
+            return 1;
+        }
+
+        virtual void callback(cryptonote::levin::detail::p2p_context& context) override final
+        {}
+
+        virtual void on_connection_new(cryptonote::levin::detail::p2p_context&) override final
+        {}
+
+        virtual void on_connection_close(cryptonote::levin::detail::p2p_context&) override final
+        {}
+
+    public:
+        test_receiver()
+          : epee::levin::levin_commands_handler<cryptonote::levin::detail::p2p_context>(),
+            invoked_(),
+            notified_()
+        {}
+
+        virtual ~test_receiver() noexcept override final{}
+
+        std::size_t invoked_size() const noexcept
+        {
+            return invoked_.size();
+        }
+
+        std::size_t notified_size() const noexcept
+        {
+            return notified_.size();
+        }
+
+        template<typename T>
+        std::pair<boost::uuids::uuid, typename T::request> get_invoked()
+        {
+            return get_message<T>(invoked_);
+        }
+
+        template<typename T>
+        std::pair<boost::uuids::uuid, typename T::request> get_notification()
+        {
+            return get_message<T>(notified_);
+        }
+    };
+
+    class levin_notify : public ::testing::Test
+    {
+        const std::shared_ptr<cryptonote::levin::connections> connections_;
+        std::set<boost::uuids::uuid> connection_ids_;
+
+    public:
+        levin_notify()
+          : ::testing::Test(),
+            connections_(std::make_shared<cryptonote::levin::connections>()),
+            connection_ids_(),
+            random_generator_(),
+            io_service_(),
+            receiver_(),
+            contexts_()
+        {
+            connections_->set_handler(std::addressof(receiver_), nullptr);
+        }
+
+        virtual void TearDown() override final
+        {
+            EXPECT_EQ(0u, receiver_.invoked_size());
+            EXPECT_EQ(0u, receiver_.notified_size());
+        }
+
+        void add_connection()
+        {
+            contexts_.emplace_back(io_service_, *connections_, random_generator_);
+            EXPECT_TRUE(connection_ids_.emplace(contexts_.back().get_id()).second);
+            EXPECT_EQ(connection_ids_.size(), connections_->get_connections_count());
+        }
+
+        cryptonote::levin::notify make_notifier(const std::size_t noise_size)
+        {
+            epee::byte_slice noise = nullptr;
+            if (noise_size)
+                noise = epee::levin::make_noise_notify(noise_size);
+            return cryptonote::levin::notify{io_service_, connections_, std::move(noise)};
+        }
+
+        boost::uuids::random_generator random_generator_;
+        boost::asio::io_service io_service_;
+        test_receiver receiver_;
+        std::deque<test_connection> contexts_;
+    };
+}
+
+TEST(make_header, no_expect_return)
+{
+    static constexpr const std::size_t max_length = std::numeric_limits<std::size_t>::max();
+
+    const epee::levin::bucket_head2 header1 = epee::levin::make_header(1024, max_length, 5601, false);
+    EXPECT_EQ(SWAP64LE(LEVIN_SIGNATURE), header1.m_signature);
+    EXPECT_FALSE(header1.m_have_to_return_data);
+    EXPECT_EQ(SWAP64LE(max_length), header1.m_cb);
+    EXPECT_EQ(SWAP32LE(1024), header1.m_command);
+    EXPECT_EQ(SWAP32LE(LEVIN_PROTOCOL_VER_1), header1.m_protocol_version);
+    EXPECT_EQ(SWAP32LE(5601), header1.m_flags);
+}
+
+TEST(make_header, expect_return)
+{
+    const epee::levin::bucket_head2 header1 = epee::levin::make_header(65535, 0, 0, true);
+    EXPECT_EQ(SWAP64LE(LEVIN_SIGNATURE), header1.m_signature);
+    EXPECT_TRUE(header1.m_have_to_return_data);
+    EXPECT_EQ(0u, header1.m_cb);
+    EXPECT_EQ(SWAP32LE(65535), header1.m_command);
+    EXPECT_EQ(SWAP32LE(LEVIN_PROTOCOL_VER_1), header1.m_protocol_version);
+    EXPECT_EQ(0u, header1.m_flags);
+}
+
+TEST(make_notify, empty_payload)
+{
+    const epee::byte_slice message = epee::levin::make_notify(443, nullptr);
+    const epee::levin::bucket_head2 header =
+        epee::levin::make_header(443, 0, LEVIN_PACKET_REQUEST, false);
+    ASSERT_EQ(sizeof(header), message.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), message.data(), sizeof(header)) == 0);
+}
+
+TEST(make_notify, with_payload)
+{
+    std::string bytes(100, 'a');
+    std::generate(bytes.begin(), bytes.end(), crypto::random_device{});
+
+    const epee::byte_slice message = epee::levin::make_notify(443, epee::strspan<std::uint8_t>(bytes));
+    const epee::levin::bucket_head2 header =
+        epee::levin::make_header(443, bytes.size(), LEVIN_PACKET_REQUEST, false);
+
+    ASSERT_EQ(sizeof(header) + bytes.size(), message.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), message.data(), sizeof(header)) == 0);
+    EXPECT_TRUE(std::memcmp(bytes.data(), message.data() + sizeof(header), bytes.size()) == 0);
+}
+
+TEST(make_noise, invalid)
+{
+    EXPECT_TRUE(epee::levin::make_noise_notify(sizeof(epee::levin::bucket_head2) - 1).empty());
+}
+
+TEST(make_noise, valid)
+{
+    static constexpr const std::uint32_t flags =
+        LEVIN_PACKET_BEGIN | LEVIN_PACKET_END;
+
+    const epee::byte_slice noise = epee::levin::make_noise_notify(1024);
+    const epee::levin::bucket_head2 header =
+        epee::levin::make_header(0, 1024 - sizeof(epee::levin::bucket_head2), flags, false);
+
+    ASSERT_EQ(1024, noise.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), noise.data(), sizeof(header)) == 0);
+    EXPECT_EQ(1024 - sizeof(header), std::count(noise.cbegin() + sizeof(header), noise.cend(), 0));
+}
+
+TEST(make_fragment, invalid)
+{
+    EXPECT_TRUE(epee::levin::make_fragmented_notify(nullptr, 0, nullptr).empty());
+}
+
+TEST(make_fragment, single)
+{
+    const epee::byte_slice noise = epee::levin::make_noise_notify(1024);
+    const epee::byte_slice fragment = epee::levin::make_fragmented_notify(noise, 11, nullptr);
+    const epee::levin::bucket_head2 header =
+        epee::levin::make_header(11, 1024 - sizeof(epee::levin::bucket_head2), LEVIN_PACKET_REQUEST, false);
+
+    EXPECT_EQ(1024, noise.size());
+    ASSERT_EQ(1024, fragment.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), fragment.data(), sizeof(header)) == 0);
+    EXPECT_EQ(1024 - sizeof(header), std::count(noise.cbegin() + sizeof(header), noise.cend(), 0));
+}
+
+TEST(make_fragment, multiple)
+{
+    std::string bytes(1024 * 3 - 150, 'a');
+    std::generate(bytes.begin(), bytes.end(), crypto::random_device{});
+
+    const epee::byte_slice noise = epee::levin::make_noise_notify(1024);
+    epee::byte_slice fragment = epee::levin::make_fragmented_notify(noise, 114, epee::strspan<std::uint8_t>(bytes));
+
+    epee::levin::bucket_head2 header =
+        epee::levin::make_header(0, 1024 - sizeof(epee::levin::bucket_head2), LEVIN_PACKET_BEGIN, false);
+
+    ASSERT_LE(sizeof(header), fragment.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), fragment.data(), sizeof(header)) == 0);
+
+    fragment.take_slice(sizeof(header));
+    header.m_flags = LEVIN_PACKET_REQUEST;
+    header.m_cb = bytes.size();
+    header.m_command = 114;
+
+    ASSERT_LE(sizeof(header), fragment.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), fragment.data(), sizeof(header)) == 0);
+
+    fragment.take_slice(sizeof(header));
+
+    ASSERT_LE(bytes.size(), fragment.size());
+    EXPECT_TRUE(std::memcmp(bytes.data(), fragment.data(), 1024 - sizeof(header) * 2) == 0);
+
+    bytes.erase(0, 1024 - sizeof(header) * 2);
+    fragment.take_slice(1024 - sizeof(header) * 2);
+    header.m_flags = 0;
+    header.m_cb = 1024 - sizeof(header);
+    header.m_command = 0;
+
+    ASSERT_LE(sizeof(header), fragment.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), fragment.data(), sizeof(header)) == 0);
+
+    fragment.take_slice(sizeof(header));
+
+    ASSERT_LE(bytes.size(), fragment.size());
+    EXPECT_TRUE(std::memcmp(bytes.data(), fragment.data(), 1024 - sizeof(header)) == 0);
+
+    bytes.erase(0, 1024 - sizeof(header));
+    fragment.take_slice(1024 - sizeof(header));
+    header.m_flags = LEVIN_PACKET_END;
+
+    ASSERT_LE(sizeof(header), fragment.size());
+    EXPECT_TRUE(std::memcmp(std::addressof(header), fragment.data(), sizeof(header)) == 0);
+
+    fragment.take_slice(sizeof(header));
+    EXPECT_TRUE(std::memcmp(bytes.data(), fragment.data(), bytes.size()) == 0);
+
+    fragment.take_slice(bytes.size());
+
+    EXPECT_EQ(18, std::count(fragment.cbegin(), fragment.cend(), 0));
+}
+
+TEST_F(levin_notify, defaulted)
+{
+    cryptonote::levin::notify notifier{};
+    {
+        const auto status = notifier.get_status();
+        EXPECT_FALSE(status.has_noise);
+        EXPECT_FALSE(status.connections_filled);
+    }
+    EXPECT_FALSE(notifier.send_txs({}, random_generator_(), false));
+}
+
+TEST_F(levin_notify, flood)
+{
+    cryptonote::levin::notify notifier = make_notifier(0);
+
+    for (unsigned count = 0; count < 10; ++count)
+        add_connection();
+
+    {
+        const auto status = notifier.get_status();
+        EXPECT_FALSE(status.has_noise);
+        EXPECT_FALSE(status.connections_filled);
+    }
+    notifier.new_out_connection();
+    io_service_.poll();
+    {
+        const auto status = notifier.get_status();
+        EXPECT_FALSE(status.has_noise);
+        EXPECT_FALSE(status.connections_filled); // not tracked
+    }
+
+    std::vector<cryptonote::blobdata> txs(2);
+    txs[0].resize(100, 'e');
+    txs[1].resize(200, 'f');
+
+    ASSERT_EQ(10u, contexts_.size());
+    {
+        auto context = contexts_.begin();
+        EXPECT_TRUE(notifier.send_txs(txs, context->get_id(), false));
+
+        io_service_.reset();
+        ASSERT_LT(0u, io_service_.poll());
+        EXPECT_EQ(0u, context->process_send_queue());
+        for (++context; context != contexts_.end(); ++context)
+            EXPECT_EQ(1u, context->process_send_queue());
+
+        ASSERT_EQ(9u, receiver_.notified_size());
+        for (unsigned count = 0; count < 9; ++count)
+        {
+            auto notification = receiver_.get_notification<cryptonote::NOTIFY_NEW_TRANSACTIONS>().second;
+            EXPECT_EQ(txs, notification.txs);
+            EXPECT_TRUE(notification._.empty());
+        }
+    }
+
+    ASSERT_EQ(10u, contexts_.size());
+    {
+        auto context = contexts_.begin();
+        EXPECT_TRUE(notifier.send_txs(txs, context->get_id(), true));
+
+        io_service_.reset();
+        ASSERT_LT(0u, io_service_.poll());
+        EXPECT_EQ(0u, context->process_send_queue());
+        for (++context; context != contexts_.end(); ++context)
+            EXPECT_EQ(1u, context->process_send_queue());
+
+        ASSERT_EQ(9u, receiver_.notified_size());
+        for (unsigned count = 0; count < 9; ++count)
+        {
+            auto notification = receiver_.get_notification<cryptonote::NOTIFY_NEW_TRANSACTIONS>().second;
+            EXPECT_EQ(txs, notification.txs);
+            EXPECT_FALSE(notification._.empty());
+        }
+    }
+}
+
+TEST_F(levin_notify, noise)
+{
+    for (unsigned count = 0; count < 10; ++count)
+        add_connection();
+
+    std::vector<cryptonote::blobdata> txs(1);
+    txs[0].resize(1900, 'h');
+
+    const boost::uuids::uuid incoming_id = random_generator_();
+    cryptonote::levin::notify notifier = make_notifier(2048);
+
+    {
+        const auto status = notifier.get_status();
+        EXPECT_TRUE(status.has_noise);
+        EXPECT_FALSE(status.connections_filled);
+    }
+    ASSERT_LT(0u, io_service_.poll());
+    {
+        const auto status = notifier.get_status();
+        EXPECT_TRUE(status.has_noise);
+        EXPECT_TRUE(status.connections_filled);
+    }
+
+    notifier.run_stems();
+    io_service_.reset();
+    ASSERT_LT(0u, io_service_.poll());
+    {
+        std::size_t sent = 0;
+        for (auto& context : contexts_)
+            sent += context.process_send_queue();
+
+        EXPECT_EQ(2u, sent);
+        EXPECT_EQ(0u, receiver_.notified_size());
+    }
+
+    EXPECT_TRUE(notifier.send_txs(txs, incoming_id, false));
+    notifier.run_stems();
+    io_service_.reset();
+    ASSERT_LT(0u, io_service_.poll());
+    {
+        std::size_t sent = 0;
+        for (auto& context : contexts_)
+            sent += context.process_send_queue();
+
+        ASSERT_EQ(2u, sent);
+        while (sent--)
+        {
+            auto notification = receiver_.get_notification<cryptonote::NOTIFY_NEW_TRANSACTIONS>().second;
+            EXPECT_EQ(txs, notification.txs);
+            EXPECT_TRUE(notification._.empty());
+        }
+    }
+
+    txs[0].resize(3000, 'r');
+    EXPECT_TRUE(notifier.send_txs(txs, incoming_id, true));
+    notifier.run_stems();
+    io_service_.reset();
+    ASSERT_LT(0u, io_service_.poll());
+    {
+        std::size_t sent = 0;
+        for (auto& context : contexts_)
+            sent += context.process_send_queue();
+
+        EXPECT_EQ(2u, sent);
+        EXPECT_EQ(0u, receiver_.notified_size());
+    }
+
+    notifier.run_stems();
+    io_service_.reset();
+    ASSERT_LT(0u, io_service_.poll());
+    {
+        std::size_t sent = 0;
+        for (auto& context : contexts_)
+            sent += context.process_send_queue();
+
+        ASSERT_EQ(2u, sent);
+        while (sent--)
+        {
+            auto notification = receiver_.get_notification<cryptonote::NOTIFY_NEW_TRANSACTIONS>().second;
+            EXPECT_EQ(txs, notification.txs);
+            EXPECT_TRUE(notification._.empty());
+        }
+    }
+}

tests/unit_tests/net.cpp

@@ -26,6 +26,7 @@
 // STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
 // THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+#include <algorithm>
 #include <atomic>
 #include <boost/archive/portable_binary_oarchive.hpp>
 #include <boost/archive/portable_binary_iarchive.hpp>
@@ -36,13 +37,22 @@
 #include <boost/asio/steady_timer.hpp>
 #include <boost/asio/write.hpp>
 #include <boost/endian/conversion.hpp>
+#include <boost/range/adaptor/sliced.hpp>
+#include <boost/range/combine.hpp>
 #include <boost/system/error_code.hpp>
 #include <boost/thread/thread.hpp>
+#include <boost/uuid/nil_generator.hpp>
+#include <boost/uuid/random_generator.hpp>
+#include <boost/uuid/uuid.hpp>
+#include <cstdint>
 #include <cstring>
 #include <functional>
 #include <gtest/gtest.h>
+#include <map>
 #include <memory>
+#include <type_traits>
 
+#include "net/dandelionpp.h"
 #include "net/error.h"
 #include "net/net_utils_base.h"
 #include "net/socks.h"
@@ -857,3 +867,389 @@ TEST(socks_connector, timeout)
     EXPECT_THROW(sock.get().is_open(), boost::system::system_error);
 }
 
+TEST(dandelionpp_map, traits)
+{
+    EXPECT_TRUE(std::is_default_constructible<net::dandelionpp::connection_map>());
+    EXPECT_TRUE(std::is_move_constructible<net::dandelionpp::connection_map>());
+    EXPECT_TRUE(std::is_move_assignable<net::dandelionpp::connection_map>());
+    EXPECT_FALSE(std::is_copy_constructible<net::dandelionpp::connection_map>());
+    EXPECT_FALSE(std::is_copy_assignable<net::dandelionpp::connection_map>());
+}
+
+TEST(dandelionpp_map, empty)
+{
+    const net::dandelionpp::connection_map mapper{};
+
+    EXPECT_EQ(mapper.begin(), mapper.end());
+    EXPECT_EQ(0u, mapper.size());
+
+    const net::dandelionpp::connection_map cloned = mapper.clone();
+    EXPECT_EQ(cloned.begin(), cloned.end());
+    EXPECT_EQ(0u, cloned.size());
+}
+
+TEST(dandelionpp_map, zero_stems)
+{
+    std::vector<boost::uuids::uuid> connections{6};
+    std::generate(connections.begin(), connections.end(), boost::uuids::random_generator{});
+
+    net::dandelionpp::connection_map mapper{connections, 0};
+    EXPECT_EQ(mapper.begin(), mapper.end());
+    EXPECT_EQ(0u, mapper.size());
+
+    for (const boost::uuids::uuid& connection : connections)
+        EXPECT_TRUE(mapper.get_stem(connection).is_nil());
+
+    EXPECT_FALSE(mapper.update(connections));
+    EXPECT_EQ(mapper.begin(), mapper.end());
+    EXPECT_EQ(0u, mapper.size());
+
+    for (const boost::uuids::uuid& connection : connections)
+        EXPECT_TRUE(mapper.get_stem(connection).is_nil());
+
+    const net::dandelionpp::connection_map cloned = mapper.clone();
+    EXPECT_EQ(cloned.end(), cloned.begin());
+    EXPECT_EQ(0u, cloned.size());
+}
+
+TEST(dandelionpp_map, dropped_connection)
+{
+    std::vector<boost::uuids::uuid> connections{6};
+    std::generate(connections.begin(), connections.end(), boost::uuids::random_generator{});
+    std::sort(connections.begin(), connections.end());
+
+    // select 3 of 6 outgoing connections
+    net::dandelionpp::connection_map mapper{connections, 3};
+    EXPECT_EQ(3u, mapper.size());
+    EXPECT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    {
+        const net::dandelionpp::connection_map cloned = mapper.clone();
+        EXPECT_EQ(3u, cloned.size());
+        ASSERT_EQ(mapper.end() - mapper.begin(), cloned.end() - cloned.begin());
+        for (auto elem : boost::combine(mapper, cloned))
+            EXPECT_EQ(boost::get<0>(elem), boost::get<1>(elem));
+    }
+    EXPECT_FALSE(mapper.update(connections));
+    EXPECT_EQ(3u, mapper.size());
+    ASSERT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    std::map<boost::uuids::uuid, boost::uuids::uuid> mapping;
+    std::vector<boost::uuids::uuid> in_connections{9};
+    std::generate(in_connections.begin(), in_connections.end(), boost::uuids::random_generator{});
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        std::multimap<boost::uuids::uuid, boost::uuids::uuid> inverse_mapping;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            EXPECT_TRUE(mapping.emplace(connection, out).second);
+            inverse_mapping.emplace(out, connection);
+            used[out]++;
+        }
+
+        EXPECT_EQ(3u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(3u, entry.second);
+
+        for (const boost::uuids::uuid& connection : in_connections)
+            EXPECT_EQ(mapping[connection], mapper.get_stem(connection));
+
+        // drop 1 connection, and select replacement from 1 of unused 3.
+        const boost::uuids::uuid lost_connection = *(++mapper.begin());
+        const auto elem = std::lower_bound(connections.begin(), connections.end(), lost_connection);
+        ASSERT_NE(connections.end(), elem);
+        ASSERT_EQ(lost_connection, *elem);
+        connections.erase(elem);
+
+        EXPECT_TRUE(mapper.update(connections));
+        EXPECT_EQ(3u, mapper.size());
+        ASSERT_EQ(3, mapper.end() - mapper.begin());
+
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_NE(lost_connection, connection);
+        }
+
+        const boost::uuids::uuid newly_mapped = *(++mapper.begin());
+        EXPECT_FALSE(newly_mapped.is_nil());
+        EXPECT_NE(lost_connection, newly_mapped);
+
+        for (auto elems = inverse_mapping.equal_range(lost_connection); elems.first != elems.second; ++elems.first)
+            mapping[elems.first->second] = newly_mapped;
+    }
+    {
+        const net::dandelionpp::connection_map cloned = mapper.clone();
+        EXPECT_EQ(3u, cloned.size());
+        ASSERT_EQ(mapper.end() - mapper.begin(), cloned.end() - cloned.begin());
+        for (auto elem : boost::combine(mapper, cloned))
+            EXPECT_EQ(boost::get<0>(elem), boost::get<1>(elem));
+    }
+    // mappings should remain evenly distributed amongst 2, with 3 sitting in waiting
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid& out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            EXPECT_EQ(mapping[connection], out);
+            used[out]++;
+        }
+
+        EXPECT_EQ(3u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(3u, entry.second);
+    }
+    {
+        const net::dandelionpp::connection_map cloned = mapper.clone();
+        EXPECT_EQ(3u, cloned.size());
+        ASSERT_EQ(mapper.end() - mapper.begin(), cloned.end() - cloned.begin());
+        for (auto elem : boost::combine(mapper, cloned))
+            EXPECT_EQ(boost::get<0>(elem), boost::get<1>(elem));
+    }
+}
+
+TEST(dandelionpp_map, dropped_connection_remapped)
+{
+    boost::uuids::random_generator random_uuid{};
+
+    std::vector<boost::uuids::uuid> connections{3};
+    std::generate(connections.begin(), connections.end(), random_uuid);
+    std::sort(connections.begin(), connections.end());
+
+    // select 3 of 3 outgoing connections
+    net::dandelionpp::connection_map mapper{connections, 3};
+    EXPECT_EQ(3u, mapper.size());
+    EXPECT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    EXPECT_FALSE(mapper.update(connections));
+    EXPECT_EQ(3u, mapper.size());
+    ASSERT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    std::map<boost::uuids::uuid, boost::uuids::uuid> mapping;
+    std::vector<boost::uuids::uuid> in_connections{9};
+    std::generate(in_connections.begin(), in_connections.end(), random_uuid);
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        std::multimap<boost::uuids::uuid, boost::uuids::uuid> inverse_mapping;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            EXPECT_TRUE(mapping.emplace(connection, out).second);
+            inverse_mapping.emplace(out, connection);
+            used[out]++;
+        }
+
+        EXPECT_EQ(3u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(3u, entry.second);
+
+        for (const boost::uuids::uuid& connection : in_connections)
+            EXPECT_EQ(mapping[connection], mapper.get_stem(connection));
+
+        // drop 1 connection leaving "hole"
+        const boost::uuids::uuid lost_connection = *(++mapper.begin());
+        const auto elem = std::lower_bound(connections.begin(), connections.end(), lost_connection);
+        ASSERT_NE(connections.end(), elem);
+        ASSERT_EQ(lost_connection, *elem);
+        connections.erase(elem);
+
+        EXPECT_TRUE(mapper.update(connections));
+        EXPECT_EQ(2u, mapper.size());
+        EXPECT_EQ(3, mapper.end() - mapper.begin());
+
+        for (auto elems = inverse_mapping.equal_range(lost_connection); elems.first != elems.second; ++elems.first)
+            mapping[elems.first->second] = boost::uuids::nil_uuid();
+    }
+    // remap 3 connections and map 1 new connection to 2 remaining out connections
+    in_connections.resize(10);
+    in_connections[9] = random_uuid();
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid& out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            used[out]++;
+
+            boost::uuids::uuid& expected = mapping[connection];
+            if (!expected.is_nil())
+                EXPECT_EQ(expected, out);
+            else
+                expected = out;
+        }
+
+        EXPECT_EQ(2u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(5u, entry.second);
+    }
+    // select 3 of 3 connections but do not remap existing links
+    connections.resize(3);
+    connections[2] = random_uuid();
+    EXPECT_TRUE(mapper.update(connections));
+    EXPECT_EQ(3u, mapper.size());
+    EXPECT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid& out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            used[out]++;
+
+            EXPECT_EQ(mapping[connection], out);
+        }
+
+        EXPECT_EQ(2u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(5u, entry.second);
+    }
+    // map 8 new incoming connections across 3 outgoing links
+    in_connections.resize(18);
+    std::generate(in_connections.begin() + 10, in_connections.end(), random_uuid);
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid& out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            used[out]++;
+
+            boost::uuids::uuid& expected = mapping[connection];
+            if (!expected.is_nil())
+                EXPECT_EQ(expected, out);
+            else
+                expected = out;
+        }
+
+        EXPECT_EQ(3u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(6u, entry.second);
+    }
+}
+
+TEST(dandelionpp_map, dropped_all_connections)
+{
+    boost::uuids::random_generator random_uuid{};
+
+    std::vector<boost::uuids::uuid> connections{8};
+    std::generate(connections.begin(), connections.end(), random_uuid);
+    std::sort(connections.begin(), connections.end());
+
+    // select 3 of 8 outgoing connections
+    net::dandelionpp::connection_map mapper{connections, 3};
+    EXPECT_EQ(3u, mapper.size());
+    EXPECT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    EXPECT_FALSE(mapper.update(connections));
+    EXPECT_EQ(3u, mapper.size());
+    ASSERT_EQ(3, mapper.end() - mapper.begin());
+    {
+        std::set<boost::uuids::uuid> used;
+        for (const boost::uuids::uuid& connection : mapper)
+        {
+            EXPECT_FALSE(connection.is_nil());
+            EXPECT_TRUE(used.insert(connection).second);
+            EXPECT_TRUE(std::binary_search(connections.begin(), connections.end(), connection));
+        }
+    }
+    std::vector<boost::uuids::uuid> in_connections{9};
+    std::generate(in_connections.begin(), in_connections.end(), random_uuid);
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        std::map<boost::uuids::uuid, boost::uuids::uuid> mapping;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            EXPECT_TRUE(mapping.emplace(connection, out).second);
+            used[out]++;
+        }
+
+        EXPECT_EQ(3u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(3u, entry.second);
+
+        for (const boost::uuids::uuid& connection : in_connections)
+            EXPECT_EQ(mapping[connection], mapper.get_stem(connection));
+
+        // drop all connections
+        connections.clear();
+
+        EXPECT_TRUE(mapper.update(connections));
+        EXPECT_EQ(0u, mapper.size());
+        EXPECT_EQ(3, mapper.end() - mapper.begin());
+    }
+    // remap 7 connections to nothing
+    for (const boost::uuids::uuid& connection : boost::adaptors::slice(in_connections, 0, 7))
+        EXPECT_TRUE(mapper.get_stem(connection).is_nil());
+
+    // select 3 of 30 connections, only 7 should be remapped to new indexes (but all to new uuids)
+    connections.resize(30);
+    std::generate(connections.begin(), connections.end(), random_uuid);
+    EXPECT_TRUE(mapper.update(connections));
+    {
+        std::map<boost::uuids::uuid, std::size_t> used;
+        for (const boost::uuids::uuid& connection : in_connections)
+        {
+            const boost::uuids::uuid& out = mapper.get_stem(connection);
+            EXPECT_FALSE(out.is_nil());
+            used[out]++;
+        }
+
+        EXPECT_EQ(3u, used.size());
+        for (const std::pair<boost::uuids::uuid, std::size_t>& entry : used)
+            EXPECT_EQ(3u, entry.second);
+    }
+}