Skip to content
Snippets Groups Projects
user avatar
Angelo Capossele authored
Adds new analysis-dashboard, improves analysis server, adds proper metrics collection and Prometheus exporter (#502)

* :recycle: Clean up

* :white_check_mark: Add TestCalculateBranchOfTransaction

* :rocket: Upgrade fpc dashboard

* :whale: Fix docker-compose

* :white_check_mark: Add TestMoveTransactionToBranch

* :white_check_mark: Add TestFork

* :white_check_mark: Add TestBookPayload

* Add test for checkPayloadSolidity

* :white_check_mark: Add TestSetTransactionPreferred

* Add more tests

* :white_check_mark: Fix Tangle test

* Feat: started implementing lucas test cases

* Feat: fixed some issued + further tests

* Feat: started adding invalid txs check

* Feat: added removal logic for invalid transactions

* Refactor: removed Println

* :white_check_mark: Add test for 2nd Reattachment

* feat: Add first value transfer integration test

* fix: fix wrong plugin name

* :white_check_mark: Add aggregated branches test cases

* Feat: added a method to generate AggregatedBranchIDs

* :art: Use GenerateAggregatedBranchID in test

* Feat: refactored delete logic

* Fix: fixed broken test

* Feat: added final test cases for invalid txs / payloads

* ja nei was gaht dän da ab

* :construction: Make FPCHeartbeat smaller

* :construction: Split vote context to fit into an FPC update

* :construction: WIP

* Value tangle concurrency tests (#451)

* Add simple concurrency test

* Add reverse and concurrent transaction and value object solidification tests and fix bug when value object was visited more than once

* Add some documentation to make tests easily understandable

* :card_box: Add FPC analysis data persistence

* :sparkles: Add Prometheus

* :white_check_mark: Update FPC test

* :rotating_light: Remove linter warning

* :rotating_light: Remove linter warnings

* :recycle: Fix event closure

* :wrench: Add prometheus config

* WIP propagation tests but fixed already couple of bugs

* Fix: fixed some bugs

* Feat: added propagation to inclusion states to tx and its outputs

* Feat: finished the propagation down to the tx and its outputs

* :construction: WIP

* :whale: update docker-network entry node

* :see_no_evil: Add .gitignore

* :chart_with_upwards_trend: Update Prometheus

* :bug: Fix MongoDB ctx bug

* WIP propagation tests and fix bugs

* :fire: Delete finalized conflicts

* Add colored tokens test

* Add value tangle test to github workflow

* fix: Fix wrong function name in comments

* refactor: Make testSnapshots disabled in default and minor tweaks

* Feat: fixed some issues and introduced a Debugger

* Refactor: added a few comments

* Split massive test file into slightly more digestible chunks

* Initial commit for metrics collections package

* Clean up propagation tests

* Feat: fixed bugs

* Feat: enabled missing tests

* Add some documentation and missing checks for aggregated branches

* :construction: WIP

* Clean up tangle tests

* adds snapshot type

* Fix: finalized wasn't propagated when a branch was rejected

* Skeleton for implementation

* Measure isSynced

* UI Improvements

* Make it compile

* Measure TPS in value tangle

* implements ReadFrom and WriteTo for Snapshot

* read in snapshot file if snapshot path is defined

* renames snapshot test file

* :construction: WIP metrics

* Measure Tips in value tangle

* :chart_with_upwards_trend: Add DBSize metric

* WIP debugging concurrency bug of death

* Measure tips in message tangle

* :chart_with_upwards_trend: Add AvgNeighborConnectionLifeTime metric

* :arrow_up: Bump hive.go

* :chart_with_upwards_trend: Add autopeering distance metric

* Feat: added more reliable fails in test case

* Fix: fixes a race condition in solidification

* :chart_with_upwards_trend: Add gossip network traffic metric

* Clean up test

* adds assets volume to integration test containers

* fixes some asserts

* adds non-working conflict integration test

* check transaction availability in partition

* renames integration test

* Measure MPS per payload type

* Measure (cumulative) total msg count and per payload

* Measure FPC number of currently active conflicts

* lower amount of peers

* Package updates

* :tada: Link valuetransfer+FPC+dashboard

* :package: Update packr

* :bug: Fix logo

* first passing version of consensus integration test

* remove debug printlns

* do all integration tests again

* increases avg. network delay fcob rule, removes debug printlns

* go mod tidy by Marie Kondō

* renames incl. state. conflict to conflicting

* go fmt tangle.go

* go fmt tangle_test, goimports dapp.go

* goimports again because the dog is sad

* run consensus integration test on the CI

* use explicit pumba version 0.7.2

* pray to the CI gods for the test to pass

* fix panic when tangle.Fork() is called

* :sparkles: Add double spend test

* :arrow_up: Update hive.go

* readd all tests again

* :chart_with_upwards_trend: Add AnalysisOutboundBytes metric

* :arrow_up: Update go.mod

* :chart_with_upwards_trend: Start setting prometheus metrics

* reset integration framework paras

* :whale: Update docker-compose

* :whale: Fix typo

* :construction: WIP

* :whale: Update docker-compose

* :arrow_up: Bump golang version

* :bug: Fix snapshot file print

* :wrench: Change client url for testing

* :bug: Remove pflag double import

* OpinionEvent struct for calling Finalized and Failed vote events

* Measure finalized/failed conflicts + average rounds of finalization

* refactor: Remove FPC page from dashboard

* refactor: Remove Drng link from dashboard

* :chart_with_upwards_trend: Add autopeering network traffic metric

* :whale: Bump up golang version

* :construction: WIP

* Measure voting queries (received, unreplied) and number of opinions

* :sparkles: Add sendPayload API

* :zap: Avoid sending empty rounds

* :chart_with_upwards_trend: Add a bunch of metrics to prometheus

* Feat: outputs inherit status of transaction

* :construction: Add Grafana integration

* Refactor: fixed erroneous rename

* :construction: WIP

* Fix: fixed missing marshaling of output bools

* Fixes after merge

* :bug: Fix some bugs

* :chart_with_upwards_trend: Prometheus FPC data collection

* :art: Small fixes

* Fix: fixed decision pending

* :chart_with_upwards_trend: Prometheus Tangle metrics data collection

* :recycle: Refactor FPC metric events

* :arrow_up: Bump up hive.go

* :sparkles: Add metric heartbeat packet

* :chart_with_upwards_trend: Add clients metrics collection via the analysis server

* :construction: WIP

* Prometheus clients info

* Fix metrics config + enable prometheus collection on entry_node

* use new protocol

* wip: redial on lost connection

* assure conn is not nil

* close on write

* Fix bugs + enable spammer on peer_master

* Fix analysis fpc livefeed bug

* connector cleanup

* :chart_with_upwards_trend: Autopeering NeighborCount + Network Diameter metrics

* do not log the dial error since the logger might not available

* removes test snapshot plugin

* :zap: Improve analysis-server

* :loud_sound: Change log to debug level

* graph pkg from autopeering-sim

* Remove debug line + hive.go update

* :arrow_up: Change hive.go version

* get rid of test snapshot plugin

* fixes wrong use of Println

* removes random tool

* :bug: Move conn initialization

* removes duplicated value entry in GH CI workflows

* :arrow_up: Update hive.go

* :recycle: Clean go.mod

* :rotating_light: Fix graph pkg linter warnings

* :recycle: improve analysis plugin

* :rotating_light: Fix metrics plugin linter warning

* xxx

* wip

* fixes integration test

* :chart_with_upwards_trend: Add FPC global metrics to prometheus

* :bug: Fix double register

* :rotating_light: Fix client pkg linter warning

* :rotating_light: Fix webapi linter warnings

* :ambulance: Run you fools!

* :white_check_mark: Fix testutil import

* Adjust inbox worker pool capacity to default

* Adjust inbox worker pool capacity to default (#505)

* :ok_hand: Address PR review comments

* :dog: Fix watch dog warnings

* :bone: Let's try with a new bone for the dog now..

* upgrade hive.go to master

* :bug: Fix local dashboard

* :zap: Improves mongoDB reliabilty

* :rotating_light: Fix linter warnings

* :white_check_mark: Fix test

* :chart_with_upwards_trend: Make dbSize a prometheus Gauge

* :recycle:

 Remove unecessary initialization

* introduce worker pool for storing finalized vote ctxs

* Update packr

* type alias FPCRecords to []FPCRecord

Co-authored-by: default avatarjonastheis <mail@jonastheis.de>
Co-authored-by: default avatarHans Moog <hm@mkjc.net>
Co-authored-by: default avatarjkrvivian <jkrvivian@gmail.com>
Co-authored-by: default avatarLuca Moser <moser.luca@gmail.com>
Co-authored-by: default avatarLevente Pap <levente.pap@iota.org>
Co-authored-by: default avatarMartyn Janes <martyn@obany.com>
Co-authored-by: default avatarWolfgang Welz <welzwo@gmail.com>
f4b7f3f8
History


Prototype node software for an IOTA network without the Coordinator

Developer documentation portal

Discord StackExchange Apache 2.0 license Go version Build status Latest release

AboutDesignImplemented Coordicide modulesWork-in-progress modulesInstallationGetting startedClient-Library and HTTP API referenceSupporting the projectJoining the discussion


About

This repository is where the IOTA Foundation's Research Department runs simulations of the Coordicide modules to study and evaluate their performance.

The aim of this open repository is to give the community the opportunity to follow developments, take part in testing, and learn more about Coordicide.

Note: You can find details about future development plans in our roadmap.

Design

The code in GoShimmer is modular, where each module represents either one of the Coordicide components or a basic node function such as the gossip layer, ledger state, and API.

Coordicide blueprint

This approach allows us to develop each module in parallel and to test GoShimmer with one or more different versions.

Each module is defined in the packages directory and can be enabled, using the plugins directory.

Note: See the main.go file to see which plugins are currently supported.

Implemented Coordicide modules

The master branch is the stable version of the GoShimmer software, which includes a minimal set of modules to allow you to send and gossip zero-value transactions.

The master branch includes the following Coordicide modules:

The autopeering module is divided into two submodules:

  • Peer discovery: Responsible for operations such as discovering new peers and verifying their online status

  • Neighbor selection: Responsible for finding and managing neighbors

Autopeering design

We also have a standalone autopeering simulator in this repository.

Work-in-progress modules

Work-in-progress modules are typically kept on a different branch such as mana, and are not compatible with the master branch. Therefore, nodes that run these branches cannot join the current network because the code either is still too experimental or it includes breaking changes.

The following Coordicide modules are a work in progress:

  • Mana: The mana branch contains a first implementation of the mana module in the packages directory.

  • Cellular Consensus: The ca branch contains a first implementation of the Cellular Consensus module in the packages directory.

  • Fast Probabilistic Consensus: The fpc branch contains a first implementation of the Fast Probabilistic Consensus module in the packages directory. We also have a standalone FPC simulator in this repository.

  • Spam Protection: You can find the initial simulation source code of the rate control in this repository and the source code of the Adaptive Proof of Work simulator here.

As well as these modules, we are working on the following node functions:

Client-Library and HTTP API reference

You can use the Go client-library to interact with GoShimmer (located under github.com/iotaledger/goshimmer/client).

Alternatively, you can check out the API docs to implement your own client or inspect the available HTTP API endpoints.

For code generation, you might want to use the OAS/Swagger specification file directly.

Installation

You have two options to install and run GoShimmer:

  • Use the precompiled executable file
  • Compile the code from source

Execute the precompiled executable file

The release page includes downloadable files for Linux, macOS, and Windows.

To run the node, all you need to do is download and execute one of these files, depending on your operating system.

# Linux and macOS
./goshimmer
# Windows
goshimmer.exe

Compile the code from source

If you want to build your own executable file, you need to follow these steps.

Prerequisites

To complete this guide, you need to have at least version 1.14 of Go installed on your device.

To check if you have Go installed, run the following command:

go version

If Go is installed, you should see the version that's installed.


  1. Clone the repository

    git clone https://github.com/iotaledger/goshimmer.git
  2. Change into the goshimmer directory

  3. Copy and adjust config.default.json

    # Linux and macOS
    cp config.default.json config.json
    # Windows
    copy config.default.json config.json
  4. Use one of the following commands to build your executable file, depending on your operating system

    # Linux and macOS
    go build -o goshimmer
    # Windows
    go build -o  goshimmer.exe

    Note: If you're using Windows PowerShell, enclose goshimmer.exe in single quotation marks. For example: go build -o 'goshimmer.exe'.

Getting started

When you first run GoShimmer, the node starts running and tries to connects to neighbors, using the autopeering module.

To run other modules such as the spammer, you can configure GoShimmer to enable them through plugins.

Note: For a list of all the available configuration parameters, you can run the following command:

# Linux and macOS
./goshimmer -help
# Windows
goshimmer.exe -help

You can configure GoShimmer in the following ways:

  • Use a configuration file called config.json
  • Use command-line options

The repository includes a config.json file, which the executable file will find and use when you execute it.

To use the command line, execute the file with one of the following commands, depending on your operating system

# Linux and macOS
./goshimmer --node.enablePlugins "spammer"
# Windows
goshimmer.exe --node.enablePlugins "spammer"

Here, we use the command-line flags to enable the spammer plugin. This plugin allows you to send spam transactions to your node.

Dashboard

GoShimmer provides access to a SPA dashboard showing TPS, memory chart, neighbors and a Tangle explorer.

You can change its configuration (e.g, bind address, port) under the section dashboard of the config.json file, for example by changing the bind address to 0.0.0.0:8081 to enable the access from remote and/or by enabling the authentication.

To access the dashboard, you can use your browser (the default address is http://127.0.0.1:8081).

dashboard

Supporting the project

If you want to contribute to the code, consider posting a bug report, feature request or a pull request.

When creating a pull request, we recommend that you do the following:

  1. Clone the repository
  2. Create a new branch for your fix or feature. For example, git checkout -b fix/my-fix or git checkout -b feat/my-feature.
  3. Run the go fmt command to make sure your code is well formatted
  4. Document any exported packages
  5. Target your pull request to be merged with dev

Joining the discussion

If you want to get involved in the community, need help getting started, have any issues related to the repository or just want to discuss blockchain, distributed ledgers, and IoT with other people, feel free to join our Discord.