Adds new analysis-dashboard, improves analysis server, adds proper metrics collection and Prometheus exporter (#502)

* ♻️ Clean up

* ✅ Add TestCalculateBranchOfTransaction

* 🚀 Upgrade fpc dashboard

* 🐳 Fix docker-compose

* ✅ Add TestMoveTransactionToBranch

* ✅ Add TestFork

* ✅ Add TestBookPayload

* Add test for checkPayloadSolidity

* ✅ Add TestSetTransactionPreferred

* Add more tests

* ✅ Fix Tangle test

* Feat: started implementing lucas test cases

* Feat: fixed some issued + further tests

* Feat: started adding invalid txs check

* Feat: added removal logic for invalid transactions

* Refactor: removed Println

* ✅ Add test for 2nd Reattachment

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

* ✅ Add aggregated branches test cases

* Feat: added a method to generate AggregatedBranchIDs

* 🎨 Use GenerateAggregatedBranchID in test

* Feat: refactored delete logic

* Fix: fixed broken test

* Feat: added final test cases for invalid txs / payloads

* ja nei was gaht dän da ab

* 🚧 Make FPCHeartbeat smaller

* 🚧 Split vote context to fit into an FPC update

* 🚧 WIP

* Value tangle concurrency tests (#451)

* Add simple concurrency test

* Add reverse and concurrent transaction and value object solidification tests and fix bug when value object was visited more than once

* Add some documentation to make tests easily understandable

* 🗃️ Add FPC analysis data persistence

* ✨ Add Prometheus

* ✅ Update FPC test

* 🚨 Remove linter warning

* 🚨 Remove linter warnings

* ♻️ Fix event closure

* 🔧 Add prometheus config

* WIP propagation tests but fixed already couple of bugs

* Fix: fixed some bugs

* Feat: added propagation to inclusion states to tx and its outputs

* Feat: finished the propagation down to the tx and its outputs

* 🚧 WIP

* 🐳 update docker-network entry node

* 🙈 Add .gitignore

* 📈 Update Prometheus

* 🐛 Fix MongoDB ctx bug

* WIP propagation tests and fix bugs

* 🔥 Delete finalized conflicts

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* Feat: fixed some issues and introduced a Debugger

* Refactor: added a few comments

* Split massive test file into slightly more digestible chunks

* Initial commit for metrics collections package

* Clean up propagation tests

* Feat: fixed bugs

* Feat: enabled missing tests

* Add some documentation and missing checks for aggregated branches

* 🚧 WIP

* Clean up tangle tests

* adds snapshot type

* Fix: finalized wasn't propagated when a branch was rejected

* Skeleton for implementation

* Measure isSynced

* UI Improvements

* Make it compile

* Measure TPS in value tangle

* implements ReadFrom and WriteTo for Snapshot

* read in snapshot file if snapshot path is defined

* renames snapshot test file

* 🚧 WIP metrics

* Measure Tips in value tangle

* 📈 Add DBSize metric

* WIP debugging concurrency bug of death

* Measure tips in message tangle

* 📈 Add AvgNeighborConnectionLifeTime metric

* ⬆️ Bump hive.go

* 📈 Add autopeering distance metric

* Feat: added more reliable fails in test case

* Fix: fixes a race condition in solidification

* 📈 Add gossip network traffic metric

* Clean up test

* adds assets volume to integration test containers

* fixes some asserts

* adds non-working conflict integration test

* check transaction availability in partition

* renames integration test

* Measure MPS per payload type

* Measure (cumulative) total msg count and per payload

* Measure FPC number of currently active conflicts

* lower amount of peers

* Package updates

* 🎉 Link valuetransfer+FPC+dashboard

* 📦 Update packr

* 🐛 Fix logo

* first passing version of consensus integration test

* remove debug printlns

* do all integration tests again

* increases avg. network delay fcob rule, removes debug printlns

* go mod tidy by Marie Kondō

* renames incl. state. conflict to conflicting

* go fmt tangle.go

* go fmt tangle_test, goimports dapp.go

* goimports again because the dog is sad

* run consensus integration test on the CI

* use explicit pumba version 0.7.2

* pray to the CI gods for the test to pass

* fix panic when tangle.Fork() is called

* ✨ Add double spend test

* ⬆️ Update hive.go

* readd all tests again

* 📈 Add AnalysisOutboundBytes metric

* ⬆️ Update go.mod

* 📈 Start setting prometheus metrics

* reset integration framework paras

* 🐳 Update docker-compose

* 🐳 Fix typo

* 🚧 WIP

* 🐳 Update docker-compose

* ⬆️ Bump golang version

* 🐛 Fix snapshot file print

* 🔧 Change client url for testing

* 🐛 Remove pflag double import

* OpinionEvent struct for calling Finalized and Failed vote events

* Measure finalized/failed conflicts + average rounds of finalization

* refactor: Remove FPC page from dashboard

* refactor: Remove Drng link from dashboard

* 📈 Add autopeering network traffic metric

* 🐳 Bump up golang version

* 🚧 WIP

* Measure voting queries (received, unreplied) and number of opinions

* ✨ Add sendPayload API

* ⚡ Avoid sending empty rounds

* 📈 Add a bunch of metrics to prometheus

* Feat: outputs inherit status of transaction

* 🚧 Add Grafana integration

* Refactor: fixed erroneous rename

* 🚧 WIP

* Fix: fixed missing marshaling of output bools

* Fixes after merge

* 🐛 Fix some bugs

* 📈 Prometheus FPC data collection

* 🎨 Small fixes

* Fix: fixed decision pending

* 📈 Prometheus Tangle metrics data collection

* ♻️ Refactor FPC metric events

* ⬆️ Bump up hive.go

* ✨ Add metric heartbeat packet

* 📈 Add clients metrics collection via the analysis server

* 🚧 WIP

* Prometheus clients info

* Fix metrics config + enable prometheus collection on entry_node

* use new protocol

* wip: redial on lost connection

* assure conn is not nil

* close on write

* Fix bugs + enable spammer on peer_master

* Fix analysis fpc livefeed bug

* connector cleanup

* 📈 Autopeering NeighborCount + Network Diameter metrics

* do not log the dial error since the logger might not available

* removes test snapshot plugin

* ⚡ Improve analysis-server

* 🔊 Change log to debug level

* graph pkg from autopeering-sim

* Remove debug line + hive.go update

* ⬆️ Change hive.go version

* get rid of test snapshot plugin

* fixes wrong use of Println

* removes random tool

* 🐛 Move conn initialization

* removes duplicated value entry in GH CI workflows

* ⬆️ Update hive.go

* ♻️ Clean go.mod

* 🚨 Fix graph pkg linter warnings

* ♻️ improve analysis plugin

* 🚨 Fix metrics plugin linter warning

* xxx

* wip

* fixes integration test

* 📈 Add FPC global metrics to prometheus

* 🐛 Fix double register

* 🚨 Fix client pkg linter warning

* 🚨 Fix webapi linter warnings

* 🚑 Run you fools!

* ✅ Fix testutil import

* Adjust inbox worker pool capacity to default

* Adjust inbox worker pool capacity to default (#505)

* 👌 Address PR review comments

* 🐶 Fix watch dog warnings

* 🦴 Let's try with a new bone for the dog now..

* upgrade hive.go to master

* 🐛 Fix local dashboard

* ⚡ Improves mongoDB reliabilty

* 🚨 Fix linter warnings

* ✅ Fix test

* 📈 Make dbSize a prometheus Gauge

* ♻️

 Remove unecessary initialization

* introduce worker pool for storing finalized vote ctxs

* Update packr

* type alias FPCRecords to []FPCRecord

Co-authored-by: jonastheis <mail@jonastheis.de>
Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: jkrvivian <jkrvivian@gmail.com>
Co-authored-by: Luca Moser <moser.luca@gmail.com>
Co-authored-by: Levente Pap <levente.pap@iota.org>
Co-authored-by: Martyn Janes <martyn@obany.com>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* Feat: initial commit

* Feat: added setPreferred to TransactionMetadata

* Feat: added a Conflicting() method to the transactionMetadata

* Fix: fixed logic bug

* Feat: refactored fcob

* Refactor: refactored additional code

* Fix: fixed a bug in ForeachConsumers

* Refactor: cleaned up code

* Feat: implemented FCOB consensus into the valuetransfer dapp

* Refactor: refactored FCOB

* Docs: added some additional comments

* Docs: fixed comments

* add branch manager conflict test

* cleans failing test

* Refactor: commit before branch change

* Fix: fixed bug in AggregateBranches

* assert aggr. branch IDs

* expands branch conflict detection test

* add visualisation of branch graph of test

* Feat: added PayloadLiked Event

* Refactor: fixed some missing comments + added liked to marshal

* Feat: reworked the preferred and liked propagation

* Refactor: cleaned up some logic

* Refactor: simplified code

* Refactor: cleaned up more stuff :P

* Refactor: refactor

* Feat: moved test + refactored fcob

* adds more tests

* fixes liked state not getting updated correctly of conflict members

* adds additional liked/preferred propagation test

* Fix: fixed missing preferred propagation to aggregated branches

* Fix: fixed a few bugs in liked propagation

* adapt to new hive.go version

* upgrade hive.go

* Feat: started implementing a wallet

* Feat: extended wallet files

* remove weird test

* use mem db for tests

* more tests

* use store backed sequence

* add option to use in-memory database

* address review comments

* First tests for individual components of AttachPayloadSync

* Fix: fixed missing events in branchmanaer

* Feat: propagate changes from branch to transaction

* Add tests for checkTransactionOutputs

* Feat: started implementing confirmed propagation

* Fix: fixed unreachable code

* Add more tests

* Refactor: refactored some code according to wolfgangs review

* Refactor: cleaned up the code according to DRY

* Refactor: refactored according to wollac

* Refactor: refactored according to wollac

* Refactor: refactored according to wollac

* Refactor: refactored the code to make it more readable

* Refactor: added some doc comments + cleaned up some more code

* ✅ adds orderedMap unit tests

* 🚨 Fix linter warnings

* test: Add queue unit tests

* Add more tests

* 💄 Adjust imports order

* WIP more tests

* ✅ Add TestBookTransaction

* ✅ Update TestBookTransaction

* Add more tests

* 🚧 WIP tests

* ✅ Add TestCalculateBranchOfTransaction

* ✅ Add TestMoveTransactionToBranch

* ✅ Add TestFork

* ✅ Add TestBookPayload

* Add test for checkPayloadSolidity

* ✅ Add TestSetTransactionPreferred

* Add more tests

* ✅ Fix Tangle test

* Feat: started implementing lucas test cases

* Feat: fixed some issued + further tests

* Feat: started adding invalid txs check

* Feat: added removal logic for invalid transactions

* Refactor: removed Println

* ✅ Add test for 2nd Reattachment

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

* ✅ Add aggregated branches test cases

* Feat: added a method to generate AggregatedBranchIDs

* 🎨 Use GenerateAggregatedBranchID in test

* Feat: refactored delete logic

* Fix: fixed broken test

* Feat: added final test cases for invalid txs / payloads

* 🚧

 WIP

* Value tangle concurrency tests (#451)

* Add simple concurrency test

* Add reverse and concurrent transaction and value object solidification tests and fix bug when value object was visited more than once

* Add some documentation to make tests easily understandable

* WIP propagation tests but fixed already couple of bugs

* Fix: fixed some bugs

* Feat: added propagation to inclusion states to tx and its outputs

* Feat: finished the propagation down to the tx and its outputs

* WIP propagation tests and fix bugs

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* Feat: fixed some issues and introduced a Debugger

* Refactor: added a few comments

* Split massive test file into slightly more digestible chunks

* Clean up propagation tests

* Feat: fixed bugs

* Feat: enabled missing tests

* Add some documentation and missing checks for aggregated branches

* Clean up tangle tests

* adds snapshot type

* Fix: finalized wasn't propagated when a branch was rejected

* implements ReadFrom and WriteTo for Snapshot

* read in snapshot file if snapshot path is defined

* renames snapshot test file

* WIP debugging concurrency bug of death

* Feat: added more reliable fails in test case

* Fix: fixes a race condition in solidification

* Clean up test

* adds assets volume to integration test containers

* fixes some asserts

* adds non-working conflict integration test

* check transaction availability in partition

* renames integration test

* lower amount of peers

* first passing version of consensus integration test

* remove debug printlns

* do all integration tests again

* increases avg. network delay fcob rule, removes debug printlns

* go mod tidy by Marie Kondō

* renames incl. state. conflict to conflicting

* go fmt tangle.go

* go fmt tangle_test, goimports dapp.go

* goimports again because the dog is sad

* run consensus integration test on the CI

* use explicit pumba version 0.7.2

* pray to the CI gods for the test to pass

* fix panic when tangle.Fork() is called

* readd all tests again

* reset integration framework paras

* removes test snapshot plugin

* get rid of test snapshot plugin

* fixes wrong use of Println

* removes random tool

* removes duplicated value entry in GH CI workflows

* xxx

* wip

* fixes integration test

Co-authored-by: Hans Moog <hm@mkjc.net>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>
Co-authored-by: jonastheis <mail@jonastheis.de>
Co-authored-by: capossele <angelocapossele@gmail.com>
Co-authored-by: jkrvivian <jkrvivian@gmail.com>

* upgrade hive.go

* upgrade testify

* fix minor errors

* test that the correct events are triggered

* add test for MissingPayloadReceived event

* remove unused PayloadUnsolidifiable event

* fix TODOs

* Fix: fixed wrong testcase

* Fix: fixed wrong pending parameter in tests

* Feat: added missing PayloadInvalid Event

* upgrade testify

* Feat: propagate payload invalid events

* Feat: added error variables

* Refactor: added missing comment for error

* Refactor: final refactor

Co-authored-by: Hans Moog <hm@mkjc.net>

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* refactor: refactor balance list update method

* feat: Add inclusion states to weapis

* refactor: Rename Conflict to Rejected

* refactor: Remove unused parameter

* refactor: Simplify colored token test and increase waiting time

* fix: fix updateBalanceList

* fix: enable testsnapshots plugin for each peer

* refactor: Rename package name of each tests

* refactor: Rename functions and add more comments

* refactor: Add maxRetry to limit to avoid infinte loop and reduce else statement

* refactor: Refactor functions and remove redundant codes

* refactor: Rename shadowing variables

* test: Add value test in runTest.sh

* fixes some func comments

* increase startup time to make up for slow CI

Co-authored-by: Luca Moser <moser.luca@gmail.com>

* Introduce worker pool to decouple receiving of messages and processing them

* Adjust shutdown/close order

* copy data before submitting it to the packet worker pool

Co-authored-by: Luca Moser <moser.luca@gmail.com>

* Feat: updated to latest hive.go

* Fix: fixed bugs related to api change

* Feat: bump integration tests

Co-authored-by: Hans Moog <hm@mkjc.net>

* Adjust autopeering/neighbors endpoint to display public key in default (base58)

* Allow to specify node seed as base58 or base64 + add flag to show seed

* Interpret entry node public key as base58

* Adjust integration tests to use base58 instead of base64

* Hex to base58

* Remove flag to print seed

* Update hive.go

* fix: Fix bugs while enabling value transfer

* ➖ Remove unused plugin

* feat: Implement value web api & client library (#438)

* feat: Implement value api: Attachment

* feat: Implement value api: UnspentOutputs

* feat: Implement value api: transactionByID

* feat: Implement client lib & value api: sendTransaction

* fix: minor tweak

* fix: minor fix

* refactor: Refactor sendTransaction api

* Refactor: Fix 🐶

* refactor: Fix 🐶

* refactor: Rename api route of testSendTxn to camel case

* fix: Fix bugs in webapi

* Release object storage after retreiving it
* Handle IOTA color specifically

* refactor: Add balances in unspentOutput api

* Fix entry node disabled plugins for integration tests

* Fix grpc server nil

* 🐛 Fix FPC round initiator

* 💄 Fix backgroundWorker name

* 🚨 Adjust imports order

* Wait for autopeering when node starts

* Fix background worker name conflict with message tangle

* 🎨 Use Voter()

* 🐛

 Re-fix ValueTangle BackgroundWorker name

Co-authored-by: capossele <angelocapossele@gmail.com>
Co-authored-by: jonastheis <mail@jonastheis.de>

* Split tests in different packages so that they can run in parallel

* Set name to env name

* Add jobs for other integration tests

* Adjust waiting times

* Remove cleanup from workflow

* Add timeout to wait for peers to start

* Transition to Analysis Dashboard for visualization

 * Add FPC heartbeat changes to analysis plugin
 * Introduce Dashboard for analysis related visualization
 * Refactor Analysis Dashboard backend
 * Prepare base of frontend
 * Add skeleton of AutopeeringStore in frontend

* Update analysis dashboard frontend dependencies

* Remove old analysis webinterface

* Remove unnecessary components from analysis dashboard

* Remove analysis webinterface from pluginmgr

* Analysis Dashboard Frontend updates for autopeering

* Use base58 encoding for autopeering node ID visualization

* Add Ping WS message to analysis dashboard

* Frontend visuals update

* AutopeeringStore updates
 * Graph mouse hover highlighting
 * Fill graph with data upon rendering
 * Dynamic node spacing in graph based on number of nodes
 * Added code comments for readability

* packr config updates

* Websocket connection lost warning in frontend

* Update title of index.html

* packr update for analysis dashboard

* Update frontend README

* Update default config: webinterface -> dashboard

* Fix linter warnings

* Delete package-lock.json

* Frontend updates
 * Size of vertices enlarges during selection
 * Decrease visibility of not highlighted links
 * Fixed layout for react components with auto overflow (no jumping)

* Disable Analysis-Dashboard plugin by default

* Adjust docker container configs for analysis dashboard

Co-authored-by: Angelo Capossele <angelocapossele@gmail.com>

* adapt to new hive.go version

* upgrade hive.go

* use store backed sequence

* add option to use in-memory database

* address review comments

* Add caching volume to tester container to significantly speed up integration test development

* Increase wait time for peer to sync

* Enable the creation of a partitioned network with Pumba to block traffic between the partitions

* Add autopeering test

* Refactor heartbeat packet into message for TLV

* Refactor Analysis-Server for TLV protocol

* Adjust Heartbeat packet tests for TLV

* Remove obsolete ping packet

* Add comments

* Add message registry for analysis packet types

* Update hive.go

* Refactor analysis message registry initialization

* go mod tidy for integration tests

* format files with gofmt

* Linter warning fix

* Small fix

* Add status code check to network shutdown so that a test fails if a container stops caused by an error

* Speed up Docker network through little caching trick

* Add common integration tests

* Add message layer persistence test

* Adjust bootstrap time

* Feat: started adding conflict checls

* Feat: finished BranchConflicting function

* Fix: fixed bugs in BranchesConflicting

* Feat: InheritBranches now checks conflicts

* Feat: completed AddBranch method

* Feat: refactored InheritBranches

* Refactor: refactored AddBranch

* Feat: started implementing objectstorage in branchmanager

* Refactor: refactored Parse functions

* Feat: added KeyPartitions to branchmanager objectstorage

* Refactor: refactored some code

* Feat: added events to Branch

* Refactor: refactored event logic

* Refactor: removed unnecessary code

* Refactor: replaced events with manager based implementation

* Refactor: refactored branchmanager to update branches

* Feat: removed unused property

* Refactor: removed unused method (fork does everything)

* Refactor: commit before branch change

* Feat: added a method to update the parent branch

* Feat: added SetModified in models

* Feat: completed different logics

* Feat: added ability to uncolor coins

* Refactor: refactored some code + comments

* Refactor: renamed ancestor to descendant :P

* adds comments from meeting (#411)

* Refactor: refactored some last stuff before merge

* Feat: updated to latest hive.go

* Feat: updated hive.go in tester

Co-authored-by: Luca Moser <moser.luca@gmail.com>

* Refactor gracefulshutdown plugin

 * Remove reporting hanging workers
 * Add ShutdownWithError() func to trigger graceful shutdown
   from outside of the package with error message

* Simplify code

* Increase forcekill timeout to 60 seconds

* Add back reporting of hanging workers

* Upgrade hive.go

* go mod tidy

Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* Fix #355 Terminate GoShimmer without config file

 - `skip-config` flag to bypass check and supply values
   from cli

* Remove empty line

* Add some comments

* change termination comment

* lower case beginning of comments

* Don't mount config file into docker container by default

* Integration tests skip config file check

* Add flags for logger config

* Log at debug level for integration tests

* Update hive.go to latest version

* upgrade hive.go

Signed-off-by: Wolfgang Welz <welzwo@gmail.com>

Co-authored-by: Luca Moser <moser.luca@gmail.com>
Co-authored-by: jonastheis <mail@jonastheis.de>
Co-authored-by: Wolfgang Welz <welzwo@gmail.com>

* normalize bind addresses prints

* refactors analysis plugins

* adds tests for parsing/serializing heartbeat packets

* adds additional test case

* normalize usage of word 'ids'

* updates packr files

* normalizes handler routes for static assets for the analysis web interface

* normalize config key names

* adjusts configs for new analysis config keys

* fixes review comments

* upgrade drand version

* enable drng tests

* fix go.sum

* ✅ fix drng tests

* 🦴

 fix watchdog warnings

Co-authored-by: capossele <angelocapossele@gmail.com>

* adds bootstrap plugin

* check sync state in sync-integration test

* adds synced and bootstrapping plugin to integration test networks

* fix 🐶comments

* re-introduce go.mod into integration test dir

* adds desynchronization monitor

* adds shutdown priority for bootstrap plugin bk. worker

* give the dog some pedigree

* adds anchor point cleanup interval

* fix review dog comments

* go mod tidy powered by Marie Kondo

* adapt to hive.go autopeering events changes

* add exposed function to start autopeering neighbor selection

* start autopeering neighbor selection when gossip is started

* update go modules

* Apply suggestions from code review

Co-Authored-By: Angelo Capossele <angelocapossele@gmail.com>

* fix build errors

Co-authored-by: Angelo Capossele <angelocapossele@gmail.com>

* fix: Rename SPA to dashboard and disable portcheck of peer_replica

* fix: Remove gossip from disablePlugins of entry node

* Fix: fixed some error must be last param

* Refactor: continued refactoring according to golint

* Refactor: addresses more golint warnings

* Refactor: refactored error last

* Fix: fixed hive.go

* Refactor: refactored alot according to golang-ci-list

* Refactor: intermediary commit

* Refactor: refactored additional code according to golint-ci

* Refactor: continued to refactor according to golint

* Refactor: refactored according to golint

* Fix: fixed last golinter reports

* remove submodules

* remove graph plugin

* remove graph plugin from configs and texts

* Introduce concept of networks

* Adjust tests to use flexible Docker networks

* Adjust test execution workflow

* Separately build GoShimmer Docker image

* Generate identities to create containers from seeds

* Move Docker network functionality

* Add comments and docs

* Update readme

* Update readme

* Add synchronization test

* Adjust to merge changes

* Add additional checks including solidification check

* Adjust to naming changes

* Simplify ids seen check

* Address PR comments

* add docker based entry node

* disable analysis client plugin

* 🚨 Fixes linter warnings

* 🚧 removes newline

* 🚧 WIP

* 🚧 restore folder

* ✅ Fixes integration test