*  Reduce solidifier worker pool capacity

*  Use GOMAXPROCS

*  Change worker count to use GOMAXPROCS

*  Set DefaultRequestWorkerCount to GOMAXPROCS

*  Fix linter warnings

*  Update hive.go

* refactor plugins

* fix fmt

* fix compile err

* fix errors

* expose singletons via sync.once

* re-disable remotelog and webauth plugins

* re-enable autopeering plugin

* Apply suggestions from code review

* Apply suggestions from code review

* add descriptive comments for getters

Co-authored-by: Luca Moser <moser.luca@gmail.com>

* improve gossip logging

* fix naming

*  Grafana datasource and dashboard provisioning

 - Datasource (Prometheus) definition for docker network
 - Local and global dashboard stored as JSON in repo

Warning: If you make change to the datasources or dashboards in
Grafana UI and save them (to grafana.db), these definitions will
be overwritten upon start.
Export dashboards as JSON and update their files!

*  Update grafana dashboard

* add pow and nonce to the message

* add pow plugin

* fix pow tests

* fix linter warnings

* fix typo

* fix panic when messagelayer is disabled

* improve logging

* fix typo

* make the pow check a byte filter

* decrease the test PoW difficulty even further

* expose the pow filter and use it in the pow plugin

* Apply suggestions from code review

* make length of nonce a const

Co-authored-by: Luca Moser <moser.luca@gmail.com>

*  Fix grpc server race condition

*  Fix graceful shutdown

*  Add nil check

* 

 Refactor FPC gRPC server

* stop worker on server issues

Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* removes periodic badgerdb gc, adds dirty/health flag to db, do badgerdb gc at startup/shutdown

* removes error shadowing

* instruct the user to delete the db folder if the db can't be opened

* addresses review comments

*  Add timestamp to MongoDB

*  Update code due to code review changes.

Adds new analysis-dashboard, improves analysis server, adds proper metrics collection and Prometheus exporter (#502)

*  Clean up

*  Add TestCalculateBranchOfTransaction

*  Upgrade fpc dashboard

*  Fix docker-compose

*  Add TestMoveTransactionToBranch

*  Add TestFork

*  Add TestBookPayload

* Add test for checkPayloadSolidity

*  Add TestSetTransactionPreferred

* Add more tests

*  Fix Tangle test

* Feat: started implementing lucas test cases

* Feat: fixed some issued + further tests

* Feat: started adding invalid txs check

* Feat: added removal logic for invalid transactions

* Refactor: removed Println

*  Add test for 2nd Reattachment

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

*  Add aggregated branches test cases

* Feat: added a method to generate AggregatedBranchIDs

*  Use GenerateAggregatedBranchID in test

* Feat: refactored delete logic

* Fix: fixed broken test

* Feat: added final test cases for invalid txs / payloads

* ja nei was gaht dän da ab

*  Make FPCHeartbeat smaller

*  Split vote context to fit into an FPC update

*  WIP

* Value tangle concurrency tests (#451)

* Add simple concurrency test

* Add reverse and concurrent transaction and value object solidification tests and fix bug when value object was visited more than once

* Add some documentation to make tests easily understandable

*  Add FPC analysis data persistence

*  Add Prometheus

*  Update FPC test

*  Remove linter warning

*  Remove linter warnings

*  Fix event closure

*  Add prometheus config

* WIP propagation tests but fixed already couple of bugs

* Fix: fixed some bugs

* Feat: added propagation to inclusion states to tx and its outputs

* Feat: finished the propagation down to the tx and its outputs

*  WIP

*  update docker-network entry node

*  Add .gitignore

*  Update Prometheus

*  Fix MongoDB ctx bug

* WIP propagation tests and fix bugs

*  Delete finalized conflicts

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* Feat: fixed some issues and introduced a Debugger

* Refactor: added a few comments

* Split massive test file into slightly more digestible chunks

* Initial commit for metrics collections package

* Clean up propagation tests

* Feat: fixed bugs

* Feat: enabled missing tests

* Add some documentation and missing checks for aggregated branches

*  WIP

* Clean up tangle tests

* adds snapshot type

* Fix: finalized wasn't propagated when a branch was rejected

* Skeleton for implementation

* Measure isSynced

* UI Improvements

* Make it compile

* Measure TPS in value tangle

* implements ReadFrom and WriteTo for Snapshot

* read in snapshot file if snapshot path is defined

* renames snapshot test file

*  WIP metrics

* Measure Tips in value tangle

*  Add DBSize metric

* WIP debugging concurrency bug of death

* Measure tips in message tangle

*  Add AvgNeighborConnectionLifeTime metric

*  Bump hive.go

*  Add autopeering distance metric

* Feat: added more reliable fails in test case

* Fix: fixes a race condition in solidification

*  Add gossip network traffic metric

* Clean up test

* adds assets volume to integration test containers

* fixes some asserts

* adds non-working conflict integration test

* check transaction availability in partition

* renames integration test

* Measure MPS per payload type

* Measure (cumulative) total msg count and per payload

* Measure FPC number of currently active conflicts

* lower amount of peers

* Package updates

*  Link valuetransfer+FPC+dashboard

*  Update packr

*  Fix logo

* first passing version of consensus integration test

* remove debug printlns

* do all integration tests again

* increases avg. network delay fcob rule, removes debug printlns

* go mod tidy by Marie Kondō

* renames incl. state. conflict to conflicting

* go fmt tangle.go

* go fmt tangle_test, goimports dapp.go

* goimports again because the dog is sad

* run consensus integration test on the CI

* use explicit pumba version 0.7.2

* pray to the CI gods for the test to pass

* fix panic when tangle.Fork() is called

*  Add double spend test

*  Update hive.go

* readd all tests again

*  Add AnalysisOutboundBytes metric

*  Update go.mod

*  Start setting prometheus metrics

* reset integration framework paras

*  Update docker-compose

*  Fix typo

*  WIP

*  Update docker-compose

*  Bump golang version

*  Fix snapshot file print

*  Change client url for testing

*  Remove pflag double import

* OpinionEvent struct for calling Finalized and Failed vote events

* Measure finalized/failed conflicts + average rounds of finalization

* refactor: Remove FPC page from dashboard

* refactor: Remove Drng link from dashboard

*  Add autopeering network traffic metric

*  Bump up golang version

*  WIP

* Measure voting queries (received, unreplied) and number of opinions

*  Add sendPayload API

*  Avoid sending empty rounds

*  Add a bunch of metrics to prometheus

* Feat: outputs inherit status of transaction

*  Add Grafana integration

* Refactor: fixed erroneous rename

*  WIP

* Fix: fixed missing marshaling of output bools

* Fixes after merge

*  Fix some bugs

*  Prometheus FPC data collection

*  Small fixes

* Fix: fixed decision pending

*  Prometheus Tangle metrics data collection

*  Refactor FPC metric events

*  Bump up hive.go

*  Add metric heartbeat packet

*  Add clients metrics collection via the analysis server

*  WIP

* Prometheus clients info

* Fix metrics config + enable prometheus collection on entry_node

* use new protocol

* wip: redial on lost connection

* assure conn is not nil

* close on write

* Fix bugs + enable spammer on peer_master

* Fix analysis fpc livefeed bug

* connector cleanup

*  Autopeering NeighborCount + Network Diameter metrics

* do not log the dial error since the logger might not available

* removes test snapshot plugin

*  Improve analysis-server

*  Change log to debug level

* graph pkg from autopeering-sim

* Remove debug line + hive.go update

*  Change hive.go version

* get rid of test snapshot plugin

* fixes wrong use of Println

* removes random tool

*  Move conn initialization

* removes duplicated value entry in GH CI workflows

*  Update hive.go

*  Clean go.mod

*  Fix graph pkg linter warnings

*  improve analysis plugin

*  Fix metrics plugin linter warning

* xxx

* wip

* fixes integration test

*  Add FPC global metrics to prometheus

*  Fix double register

*  Fix client pkg linter warning

*  Fix webapi linter warnings

*  Run you fools!

*  Fix testutil import

* Adjust inbox worker pool capacity to default

* Adjust inbox worker pool capacity to default (#505)

*  Address PR review comments

*  Fix watch dog warnings

*  Let's try with a new bone for the dog now..

* upgrade hive.go to master

*  Fix local dashboard

*  Improves mongoDB reliabilty

*  Fix linter warnings

*  Fix test

*  Make dbSize a prometheus Gauge

* 

 Remove unecessary initialization

* introduce worker pool for storing finalized vote ctxs

* Update packr

* type alias FPCRecords to []FPCRecord

Co-authored-by: jonastheis <mail@jonastheis.de>
Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: jkrvivian <jkrvivian@gmail.com>
Co-authored-by: Luca Moser <moser.luca@gmail.com>
Co-authored-by: Levente Pap <levente.pap@iota.org>
Co-authored-by: Martyn Janes <martyn@obany.com>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* Feat: initial commit

* Feat: added setPreferred to TransactionMetadata

* Feat: added a Conflicting() method to the transactionMetadata

* Fix: fixed logic bug

* Feat: refactored fcob

* Refactor: refactored additional code

* Fix: fixed a bug in ForeachConsumers

* Refactor: cleaned up code

* Feat: implemented FCOB consensus into the valuetransfer dapp

* Refactor: refactored FCOB

* Docs: added some additional comments

* Docs: fixed comments

* add branch manager conflict test

* cleans failing test

* Refactor: commit before branch change

* Fix: fixed bug in AggregateBranches

* assert aggr. branch IDs

* expands branch conflict detection test

* add visualisation of branch graph of test

* Feat: added PayloadLiked Event

* Refactor: fixed some missing comments + added liked to marshal

* Feat: reworked the preferred and liked propagation

* Refactor: cleaned up some logic

* Refactor: simplified code

* Refactor: cleaned up more stuff :P

* Refactor: refactor

* Feat: moved test + refactored fcob

* adds more tests

* fixes liked state not getting updated correctly of conflict members

* adds additional liked/preferred propagation test

* Fix: fixed missing preferred propagation to aggregated branches

* Fix: fixed a few bugs in liked propagation

* adapt to new hive.go version

* upgrade hive.go

* Feat: started implementing a wallet

* Feat: extended wallet files

* remove weird test

* use mem db for tests

* more tests

* use store backed sequence

* add option to use in-memory database

* address review comments

* First tests for individual components of AttachPayloadSync

* Fix: fixed missing events in branchmanaer

* Feat: propagate changes from branch to transaction

* Add tests for checkTransactionOutputs

* Feat: started implementing confirmed propagation

* Fix: fixed unreachable code

* Add more tests

* Refactor: refactored some code according to wolfgangs review

* Refactor: cleaned up the code according to DRY

* Refactor: refactored according to wollac

* Refactor: refactored according to wollac

* Refactor: refactored according to wollac

* Refactor: refactored the code to make it more readable

* Refactor: added some doc comments + cleaned up some more code

*  adds orderedMap unit tests

*  Fix linter warnings

* test: Add queue unit tests

* Add more tests

*  Adjust imports order

* WIP more tests

*  Add TestBookTransaction

*  Update TestBookTransaction

* Add more tests

*  WIP tests

*  Add TestCalculateBranchOfTransaction

*  Add TestMoveTransactionToBranch

*  Add TestFork

*  Add TestBookPayload

* Add test for checkPayloadSolidity

*  Add TestSetTransactionPreferred

* Add more tests

*  Fix Tangle test

* Feat: started implementing lucas test cases

* Feat: fixed some issued + further tests

* Feat: started adding invalid txs check

* Feat: added removal logic for invalid transactions

* Refactor: removed Println

*  Add test for 2nd Reattachment

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

*  Add aggregated branches test cases

* Feat: added a method to generate AggregatedBranchIDs

*  Use GenerateAggregatedBranchID in test

* Feat: refactored delete logic

* Fix: fixed broken test

* Feat: added final test cases for invalid txs / payloads

* 

 WIP

* Value tangle concurrency tests (#451)

* Add simple concurrency test

* Add reverse and concurrent transaction and value object solidification tests and fix bug when value object was visited more than once

* Add some documentation to make tests easily understandable

* WIP propagation tests but fixed already couple of bugs

* Fix: fixed some bugs

* Feat: added propagation to inclusion states to tx and its outputs

* Feat: finished the propagation down to the tx and its outputs

* WIP propagation tests and fix bugs

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* Feat: fixed some issues and introduced a Debugger

* Refactor: added a few comments

* Split massive test file into slightly more digestible chunks

* Clean up propagation tests

* Feat: fixed bugs

* Feat: enabled missing tests

* Add some documentation and missing checks for aggregated branches

* Clean up tangle tests

* adds snapshot type

* Fix: finalized wasn't propagated when a branch was rejected

* implements ReadFrom and WriteTo for Snapshot

* read in snapshot file if snapshot path is defined

* renames snapshot test file

* WIP debugging concurrency bug of death

* Feat: added more reliable fails in test case

* Fix: fixes a race condition in solidification

* Clean up test

* adds assets volume to integration test containers

* fixes some asserts

* adds non-working conflict integration test

* check transaction availability in partition

* renames integration test

* lower amount of peers

* first passing version of consensus integration test

* remove debug printlns

* do all integration tests again

* increases avg. network delay fcob rule, removes debug printlns

* go mod tidy by Marie Kondō

* renames incl. state. conflict to conflicting

* go fmt tangle.go

* go fmt tangle_test, goimports dapp.go

* goimports again because the dog is sad

* run consensus integration test on the CI

* use explicit pumba version 0.7.2

* pray to the CI gods for the test to pass

* fix panic when tangle.Fork() is called

* readd all tests again

* reset integration framework paras

* removes test snapshot plugin

* get rid of test snapshot plugin

* fixes wrong use of Println

* removes random tool

* removes duplicated value entry in GH CI workflows

* xxx

* wip

* fixes integration test

Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>
Co-authored-by: jonastheis <mail@jonastheis.de>
Co-authored-by: capossele <angelocapossele@gmail.com>
Co-authored-by: jkrvivian <jkrvivian@gmail.com>

*  Add sendPayload API

*  WIP

* 

 Fix webapi linter warnings

* Fix after cherry-pick

* Fix linter

Co-authored-by: capossele <angelocapossele@gmail.com>

* upgrade hive.go

* upgrade testify

* fix minor errors

* test that the correct events are triggered

* add test for MissingPayloadReceived event

* remove unused PayloadUnsolidifiable event

* fix TODOs

* Fix: fixed wrong testcase

* Fix: fixed wrong pending parameter in tests

* Feat: added missing PayloadInvalid Event

* upgrade testify

* Feat: propagate payload invalid events

* Feat: added error variables

* Refactor: added missing comment for error

* Refactor: final refactor

Co-authored-by: Hans Moog <hm@mkjc.net>

* test that the correct events are triggered

* assure that all events are mocked

* cleanup event wrapper

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* refactor: refactor balance list update method

* feat: Add inclusion states to weapis

* refactor: Rename Conflict to Rejected

* refactor: Remove unused parameter

* refactor: Simplify colored token test and increase waiting time

* fix: fix updateBalanceList

* fix: enable testsnapshots plugin for each peer

* refactor: Rename package name of each tests

* refactor: Rename functions and add more comments

* refactor: Add maxRetry to limit to avoid infinte loop and reduce else statement

* refactor: Refactor functions and remove redundant codes

* refactor: Rename shadowing variables

* test: Add value test in runTest.sh

* fixes some func comments

* increase startup time to make up for slow CI

Co-authored-by: Luca Moser <moser.luca@gmail.com>

* Introduce worker pool to decouple receiving of messages and processing them

* Adjust shutdown/close order

* copy data before submitting it to the packet worker pool

Co-authored-by: Luca Moser <moser.luca@gmail.com>

Co-authored-by: Hans Moog <hm@mkjc.net>

* Feat: outputs inherit status of transaction

* Refactor: fixed erroneous rename

* Fix: fixed missing marshaling of output bools

* Fix: fixed decision pending

* Feat: fixed missing event

* Update dapps/valuetransfers/packages/tangle/tangle.go

Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* Refactor: made some setter private

* Refactor: made more setters private

Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* add pow package

* Update packages/pow/pow.go

Co-authored-by: Luca Moser <moser.luca@gmail.com>

Co-authored-by: Luca Moser <moser.luca@gmail.com>

* Feat: Added Signature checks for value transfers to MessageParser

* Refactor: renamed to SignatureFilter

* Update dapps/valuetransfers/packages/tangle/signature_filter.go

Co-authored-by: Angelo Capossele <angelocapossele@gmail.com>

* Update dapps/valuetransfers/packages/tangle/signature_filter.go

Co-authored-by: Angelo Capossele <angelocapossele@gmail.com>

* Refactor: refactored code according to guard programming style

* Refactor: refactored some code

* Refactor: refactored code + adjusted go import settings

* Refactor: changed goimport of value dapp

* Refactor: refactored some more code

* Feat: more refactor :D

* Refactor: simplified constructor

* Refactor: reordered properties of struct

* Refactor: test

* Fix: fixed bug in payload parser (fallback to generic)

* Refactor: adjusted comment

* Fix: fixed conditions for third test case

* Refactor: refactored comment

Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: Angelo Capossele <angelocapossele@gmail.com>

* Create base for little dapp to measure actual network delay

* Adjust remote log to support additional messages

* Send network delay objects via remote log functionality

* Create wrapper for remote logger connection for easier use

* Add check for issuer public key

* Disable debug log in Logstash

* Disable debug log in Logstash

* Bind host time into containers

* feat: Add inclusion states to weapis

* refactor: Rename Conflict to Rejected

* Feat: initial commit

* Feat: added setPreferred to TransactionMetadata

* Feat: added a Conflicting() method to the transactionMetadata

* Fix: fixed logic bug

* Feat: refactored fcob

* Refactor: refactored additional code

* Fix: fixed a bug in ForeachConsumers

* Refactor: cleaned up code

* Feat: implemented FCOB consensus into the valuetransfer dapp

* Refactor: refactored FCOB

* Docs: added some additional comments

* Docs: fixed comments

* add branch manager conflict test

* cleans failing test

* Refactor: commit before branch change

* Fix: fixed bug in AggregateBranches

* assert aggr. branch IDs

* expands branch conflict detection test

* add visualisation of branch graph of test

* Feat: added PayloadLiked Event

* Refactor: fixed some missing comments + added liked to marshal

* Feat: reworked the preferred and liked propagation

* Refactor: cleaned up some logic

* Refactor: simplified code

* Refactor: cleaned up more stuff :P

* Refactor: refactor

* Feat: moved test + refactored fcob

* adds more tests

* fixes liked state not getting updated correctly of conflict members

* adds additional liked/preferred propagation test

* Fix: fixed missing preferred propagation to aggregated branches

* Fix: fixed a few bugs in liked propagation

* adapt to new hive.go version

* upgrade hive.go

* Feat: started implementing a wallet

* Feat: extended wallet files

* remove weird test

* use mem db for tests

* more tests

* use store backed sequence

* add option to use in-memory database

* address review comments

* First tests for individual components of AttachPayloadSync

* Fix: fixed missing events in branchmanaer

* Feat: propagate changes from branch to transaction

* Add tests for checkTransactionOutputs

* Feat: started implementing confirmed propagation

* Fix: fixed unreachable code

* Add more tests

* Refactor: refactored some code according to wolfgangs review

* Refactor: cleaned up the code according to DRY

* Refactor: refactored according to wollac

* Refactor: refactored according to wollac

* Refactor: refactored according to wollac

* Refactor: refactored the code to make it more readable

* Refactor: added some doc comments + cleaned up some more code

*  adds orderedMap unit tests

*  Fix linter warnings

* test: Add queue unit tests

* Add more tests

*  Adjust imports order

* WIP more tests

*  Add TestBookTransaction

*  Update TestBookTransaction

* Add more tests

*  WIP tests

*  Add TestCalculateBranchOfTransaction

*  Add TestMoveTransactionToBranch

*  Add TestFork

*  Add TestBookPayload

* Add test for checkPayloadSolidity

*  Add TestSetTransactionPreferred

* Add more tests

*  Fix Tangle test

* Feat: started implementing lucas test cases

* Feat: fixed some issued + further tests

* Feat: started adding invalid txs check

* Feat: added removal logic for invalid transactions

* Refactor: removed Println

*  Add test for 2nd Reattachment

*  Add aggregated branches test cases

* Feat: added a method to generate AggregatedBranchIDs

*  Use GenerateAggregatedBranchID in test

* Feat: refactored delete logic

* Fix: fixed broken test

* Feat: added final test cases for invalid txs / payloads

* 

 WIP

* Value tangle concurrency tests (#451)

* Add simple concurrency test

* Add reverse and concurrent transaction and value object solidification tests and fix bug when value object was visited more than once

* Add some documentation to make tests easily understandable

* WIP propagation tests but fixed already couple of bugs

* Fix: fixed some bugs

* Feat: added propagation to inclusion states to tx and its outputs

* Feat: finished the propagation down to the tx and its outputs

* WIP propagation tests and fix bugs

* Feat: fixed some issues and introduced a Debugger

* Refactor: added a few comments

* Split massive test file into slightly more digestible chunks

* Clean up propagation tests

* Feat: fixed bugs

* Feat: enabled missing tests

* Add some documentation and missing checks for aggregated branches

* Clean up tangle tests

* Fix: finalized wasn't propagated when a branch was rejected

* WIP debugging concurrency bug of death

* Feat: added more reliable fails in test case

* Fix: fixes a race condition in solidification

* Clean up test

Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: Luca Moser <moser.luca@gmail.com>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>
Co-authored-by: capossele <angelocapossele@gmail.com>
Co-authored-by: jkrvivian <jkrvivian@gmail.com>

*  Fix enabled/disabled plugins spelling

*  Fix publicKey camelCase

* sort enabled/disabled plugins list in info api

* use node.IsSkipped