The cache is discarded when a block is popped, but then gets
rebuilt when the difficulty for next block is requested.
While this is all properly locked, it does not take into account
the delay caused by a database transaction being only committed
(and thus its effects made visible to other threads) later on,
which means another thread could request difficulty between
the pop and the commit, which would end up using stale database
view to build the cache, but that cache would not be invalidated
again when the transaction gets committed, which would cause the
cache to not match the new database data.
To fix this, we now keep track of when the cache is invalidated
so we can invalidate it again upon database transaction commit
to ensure it gets calculated again with fresh data next time it
is nedeed.
On startup, it checks against the difficulty checkpoints, and if any mismatch is found, recalculates all the blocks with wrong difficulties. Additionally, once a week it recalculates difficulties of blocks after the last difficulty checkpoint.
If the peer (whether pruned or not itself) supports sending pruned blocks
to syncing nodes, the pruned version will be sent along with the hash
of the pruned data and the block weight. The original tx hashes can be
reconstructed from the pruned txes and theur prunable data hash. Those
hashes and the block weights are hashes and checked against the set of
precompiled hashes, ensuring the data we received is the original data.
It is currently not possible to use this system when not using the set
of precompiled hashes, since block weights can not otherwise be checked
for validity.
This is off by default for now, and is enabled by --sync-pruned-blocks
0605406 daemon: sort alt chains by height (moneromooo-monero)
4228ee0 daemon: add optional arguments to alt_chain_info (moneromooo-monero)
880ebfd daemon: add more chain specific info in alt_chain_info (moneromooo-monero)
ae6885f6 blockchain: incremental long term block weight cache (moneromooo-monero)
9b687c78 blockchain: simple cache for the long term block weights (moneromooo-monero)
This curbs runaway growth while still allowing substantial
spikes in block weight
Original specification from ArticMine:
here is the scaling proposal
Define: LongTermBlockWeight
Before fork:
LongTermBlockWeight = BlockWeight
At or after fork:
LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight)
Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time.
Define: LongTermEffectiveMedianBlockWeight
LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight))
Change Definition of EffectiveMedianBlockWeight
From (current definition)
EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight))
To (proposed definition)
EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight)
Notes:
1) There are no other changes to the existing penalty formula, median calculation, fees etc.
2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork.
3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.
Note: the long term block weight is stored in the database, but not in the actual block itself,
since it requires recalculating anyway for verification.
This will trigger if a reorg is seen. This may be used to do things
like stop automated withdrawals on large reorgs.
%s is replaced by the height at the split point
%h is replaced by the height of the new chain
%n is replaced by the number of new blocks after the reorg
The blockchain prunes seven eighths of prunable tx data.
This saves about two thirds of the blockchain size, while
keeping the node useful as a sync source for an eighth
of the blockchain.
No other data is currently pruned.
There are three ways to prune a blockchain:
- run monerod with --prune-blockchain
- run "prune_blockchain" in the monerod console
- run the monero-blockchain-prune utility
The first two will prune in place. Due to how LMDB works, this
will not reduce the blockchain size on disk. Instead, it will
mark parts of the file as free, so that future data will use
that free space, causing the file to not grow until free space
grows scarce.
The third way will create a second database, a pruned copy of
the original one. Since this is a new file, this one will be
smaller than the original one.
Once the database is pruned, it will stay pruned as it syncs.
That is, there is no need to use --prune-blockchain again, etc.
While the lookups are faster, the zeroCommit calls have to be
done again when storing the new outputs in the db, which ends
up making the whole thing slower after all, and the ways this
can be cached aren't very nice code wise, so let's forget it
since the gains aren't very large anyway.
Some of the inputs for block in a span will be from other earlier
blocks in that span. Keep track of those outputs so we don't have
to look them up again after those early blocks are added to the
blockchain.
