Commit Graph

112 Commits

Author SHA1 Message Date
Howard Chu
02abe3590d Use MDB_PREV_MULTIPLE
in get_global_output_indices
2016-02-17 16:14:50 +00:00
Howard Chu
7a4755d3a4 Fixup after lmdb master resync 2016-02-17 16:14:38 +00:00
Howard Chu
3b13a7473e Shutup about VERSION 0 2016-02-17 05:06:19 +00:00
Howard Chu
1537477c9f Use cursor in get_output_key 2016-02-17 04:05:29 +00:00
Howard Chu
f2faf8cdd9 Use MDB_APPEND mode where possible
When keys are contiguous and monotonically increasing, this gets
denser page utilization (doesn't leave padding in page splits).
Can't be used for keys that are inserted in random order (e.g. hashes)
In total this only saves around 1.5% of space compared to original
DB code. The previous patch accounted for 0.8% savings on its own;
the blocks tables just aren't that big.
2016-02-17 04:05:29 +00:00
Howard Chu
090b548c3b Use cursors in write txns
Saves a bit of seek overhead. LMDB frees them automatically
in txn_(commit|abort) so they need no cleanup.
2016-02-17 04:05:29 +00:00
Howard Chu
ed08d2152e Keep a running blocksize count
Used in batch size estimation, avoids rereading already processed
blocks during import
2016-02-17 04:05:28 +00:00
Howard Chu
0fc93345ea Win32 import batchsize tweaks
Reduce frequency of resizes: bump minimum increase from 128MB to 512MB
Use a bigger safety margin at small batch sizes
2016-02-17 04:05:28 +00:00
Riccardo Spagni
63e2d4195b
Merge pull request #657
1995923 BlockchainLMDB: Deal with DB exceptions at block level with particularity (warptangent)
c16cc20 BlockchainLMDB: Add sanity check for inconsistent state (warptangent)
9118d0a BlockchainLMDB: Call destructor on allocated txn if setup fails (warptangent)
f5581c3 BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr (warptangent)
2016-02-13 19:54:47 +02:00
warptangent
1995923559
BlockchainLMDB: Deal with DB exceptions at block level with particularity
Add another DB error exception type to distinguish failed txn setup from
general use of txn.

This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
2016-02-13 05:12:39 -08:00
warptangent
c16cc204d3
BlockchainLMDB: Add sanity check for inconsistent state
This hasn't been known to occur in block-level txn abort, but throw
exception if it does.
2016-02-13 04:17:37 -08:00
warptangent
9118d0a44c
BlockchainLMDB: Call destructor on allocated txn if setup fails 2016-02-13 04:17:37 -08:00
warptangent
f5581c3536
BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr
For consistency.
2016-02-13 04:17:36 -08:00
moneromooo-monero
8649b9f1ef
blockchain_db: pass hard fork object as a simple pointer 2016-02-08 20:56:31 +00:00
warptangent
e02577f594
Move HardFork DB update to BlockchainDB::add_block()
Ensures the database is consistent.

Also simplifes blockchain_import in that verify mode off has less to
work around.
2016-02-08 09:28:19 -08:00
warptangent
3800875406
Make HardFork object available to BlockchainDB and derived DB implementations
This will later allow the HardFork object's DB update functions to be
called when the DB transaction that persists across block add/remove is
open.
2016-02-08 09:28:17 -08:00
warptangent
fd46c96dce
BlockchainDB/LMDB: Refactor block-scope DB txn handling for add block
Move block-scope txn start and stop from BlockchainLMDB to BlockchainDB.
2016-02-08 09:28:15 -08:00
warptangent
f3a6000094
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove 2016-02-08 09:28:14 -08:00
warptangent
f47d5b0fe3
BlockchainLMDB: Allow two HardFork functions to update DB during block add
Note that this doesn't yet cause them to be called during block add.
2016-02-08 09:28:12 -08:00
warptangent
15ee0bef4b
BlockchainLMDB: extract txn macros used during block add/remove 2016-02-08 09:28:10 -08:00
warptangent
f7e337e625
fake_core: Check if hard fork subdbs need reset at start 2016-02-08 08:50:48 -08:00
warptangent
c657e772c4
blockchain_import: Add --drop-hard-fork command 2016-02-08 08:50:47 -08:00
moneromooo-monero
3a0f4d8a57
berkeleydb: fix delete/free mismatch
Despite being C++, the stats object is allocated by the underlying
C layer using malloc(3).
2016-02-03 08:49:11 +00:00
warptangent
b8cc6d46bd
BlockchainBDB: Continue when tx has no outputs
See 24b66243c2
2016-01-28 21:15:46 -08:00
warptangent
b8707466e2
BlockchainBDB: When removing, find amount output index fast by starting at end
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" db. This is most noticeable for
HDDs.

See ffcf6bdb95
2016-01-28 21:15:44 -08:00
warptangent
1a5c3fa729
BlockchainBDB: Remove tx outputs in reverse order
Data should be removed in the reverse order it was added.

This matches the order of removal in
blockchain_storage::pop_transaction_from_global_index.

See f11def012f
2016-01-28 21:15:42 -08:00
me0wmix
28f95eb001 OpenBSD support for Monero. 2016-01-21 11:18:26 -07:00
warptangent
24b66243c2
db_lmdb: Continue when tx has no outputs 2016-01-19 16:55:31 -08:00
Howard Chu
30f92f5630 Fix hf when import with verify off
Delete the hf tables, so the next open will rescan and regenerate
2016-01-15 17:26:19 +00:00
Riccardo Spagni
de03926850
updated copyright year 2015-12-31 08:39:56 +02:00
Riccardo Spagni
810a11267c
fixed copyrights with bad year references 2015-12-31 08:37:27 +02:00
Riccardo Spagni
fd36eea6dd
Merge pull request #572
b39aae7 Tweak 45800a25e9 (hyc)
4a5a5ff blockchain: always stop the ioservice before returning (moneromooo-monero)
78b65cf db_lmdb: safety close db at exit (moneromooo-monero)
45800a2 db_lmdb: fix a strdup/delete[] mistmatch (moneromooo-monero)
2015-12-30 09:38:41 +02:00
hyc
b39aae7aa5 Tweak 45800a25e9
trivial cleanup
2015-12-29 00:11:35 +00:00
moneromooo-monero
78b65cf7e8
db_lmdb: safety close db at exit
Noticed by hyc

Also set m_open to false when closed
2015-12-28 19:23:02 +00:00
moneromooo-monero
45800a25e9
db_lmdb: fix a strdup/delete[] mistmatch 2015-12-28 19:22:37 +00:00
warptangent
ee9d71e9f9
BlockchainDB: skip fixup check if read-only database 2015-12-26 14:30:20 -08:00
Riccardo Spagni
95ceb715dc
Merge pull request #565
79beed2 tests: fix various tests by using parameters better suited to monero (moneromooo-monero)
d0a8362 tests: fix some double spending tests (moneromooo-monero)
2358d0d tests: use 255 as a "too high" block version (moneromooo-monero)
f33a88c blockchain: fix a few block addition bugs (moneromooo-monero)
a9ff11c blockchain: fix an off by one error in unlocked time check (moneromooo-monero)
f294be3 blockchain: reinstate double spending checks in check_tx_inputs (moneromooo-monero)
737b6d6 blockchain: make some flag twiddling code closer to the original (moneromooo-monero)
81cb0fc blockchain: fix bitflipping test with quantized block rewards (moneromooo-monero)
22ddf09 blockchain: add missing m_tx_pool.on_blockchain_dec (moneromooo-monero)
d837c0c blockchain: fix switch to alternative blockchain for more than one block (moneromooo-monero)
5cec076 blockchain: add a missing validity check to rollback_blockchain_switching (moneromooo-monero)
3cabdb5 core: catch exceptions from get_output_key (moneromooo-monero)
5eef645 db: throw when given a non txout_to_key output to add (moneromooo-monero)
2015-12-26 10:23:17 +02:00
moneromooo-monero
5eef64578b
db: throw when given a non txout_to_key output to add
The check was explicit in the original version, so it seems
safer to make it explicit here, especially as it is now done
implicitely in a different place, away from the original check.
2015-12-25 21:56:37 +00:00
warptangent
d4c2fae2fb
BlockchainDB: Remove txs in reverse order
Data should be removed in the reverse order it was added. Not doing so
breaks assumptions and can cause problems in other DB implementations.

This matches the order of tx removal in
blockchain_storage::purge_block_data_from_blockchain.
2015-12-24 15:09:09 -08:00
warptangent
ffcf6bdb95
BlockchainLMDB: When removing, find amount output index fast by starting at end
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" subdb. This is most noticeable for
HDDs.

As before, the dup list is still walked if necessary (but in reverse),
and the global output index still confirmed to be the one looked for.
But under proper use, the result will be found at the end of the dup
list, so we start there.

Removing an amount output index is always done in the context of popping
a block, so the global output index being looked for should be the last
one in that amount key's dup list. Even if the txs themselves aren't
removed in reverse order (supposed to be according to original
implementation), the specified amount output index will still be near
the end, because the txs are in the same block.

TEST:

Pop blocks with blockchain_import.

Blocks should be successfully removed with no errors shown.

bitmonerod should be able to start syncing from the reduced blockchain
height.
2015-12-24 13:32:35 -08:00
warptangent
f11def012f
BlockchainLMDB: Remove tx outputs in reverse order
Data should be removed in the reverse order it was added.

This matches the order of removal in
blockchain_storage::pop_transaction_from_global_index.
2015-12-24 13:30:12 -08:00
moneromooo-monero
3f2970fadb
Add missing semicolons after log statements 2015-12-19 14:47:37 +00:00
warptangent
725acc7f17
Replace tabs with two spaces for consistency with rest of codebase
Remove trailing whitespace in same files.
2015-12-15 06:22:06 -08:00
moneromooo-monero
d462c62343
db_lmdb: always use a batch tx if possible
Also wrap that code in two prefix/suffix macros so they are
easier to use and not forget. Also add consts to returned data
casts where applicable.
2015-12-14 18:47:13 +00:00
Riccardo Spagni
0eb3b6645c
Merge pull request #532
79ff6a5 Add LMDB error string to database transaction exception message (warptangent)
2015-12-13 22:16:05 +02:00
Riccardo Spagni
bdf738bc7f
Merge pull request #531
cbded43 core_tests: fix ring_signature_1 tests (moneromooo-monero)
c3d208f core_tests: bump default test fee to 0.02 monero (moneromooo-monero)
10da0a0 add a --fakechain argument for tests (moneromooo-monero)
eee44e6 unit_tests: fix block reward test using post hard fork settings (moneromooo-monero)
595893f blockchain: log block (not chain) height in "BLOCK SUCCESFULLY ADDED" (moneromooo-monero)
2369968 blockchain: fix off by one in get_blocks (moneromooo-monero)
8af913a db_lmdb: implement BlockchainLMDB::reset (moneromooo-monero)
4833f4f db_bdb: implement BlockchainBDB::reset (moneromooo-monero)
18bf06e tx_pool: fix "minumim" typo in message (moneromooo-monero)
44f1267 tests: fix a typo in test name (moneromooo-monero)
1494557 db_lmdb: create all needed directories, not just the leaf one (moneromooo-monero)
015b68a db_bdb: create all needed directories, not just the leaf one (moneromooo-monero)
f141869 tests: remove data-dir argument registration (moneromooo-monero)
2015-12-13 22:14:56 +02:00
warptangent
79ff6a5a56
Add LMDB error string to database transaction exception message
When throwing an exception from being unable to begin an LMDB
transaction, include the reason.

It's often been due to a write transaction attempted within a write
transaction (batch mode), but there can be other reasons such as write
transaction attempted while database was opened read only, or
environment's map needs to be resized.
2015-12-13 09:09:46 -08:00
moneromooo-monero
8af913a3d1
db_lmdb: implement BlockchainLMDB::reset
It is needed by the core tests
2015-12-13 11:14:06 +00:00
moneromooo-monero
4833f4f96f
db_bdb: implement BlockchainBDB::reset
It is needed by the core tests
2015-12-13 11:14:00 +00:00
moneromooo-monero
1494557633
db_lmdb: create all needed directories, not just the leaf one 2015-12-13 11:09:43 +00:00