SirMordred

SirMordred is the tool used to coordinate the execution of the GrimoireLab platform, via two main configuration files, the setup.cfg and projects.json, which are summarized below,

Setup.cfg

The setup file holds the configuration to arrange all process underlying GrimoireLab. It is composed of sections which allow to define the general settings such as which phases to activate (e.g., collection, enrichment) and where to store the logs, as well as the location and credentials for SortingHat and the ElasticSearch instances where the raw and enriched data is stored. Furthermore, it also includes backend sections to set up the parameters used by Perceval to access the software development tools (e.g., GitHub tokens, gerrit username) and fetch their data.

Dashboards can be automatically uploaded via the setup.cfg if the phase panels is enabled. The Data Status and Overview dashboards will contain widgets that summarize the information of the data sources declared in the setup.cfg. Note that the widgets are not updated when adding new data sources, thus you need to manually delete the dashboards Data Status and Overview, and restart mordred again (making sure that the option panels is enabled).

[es_collection]

• password (str: None): Password for connection to Elasticsearch
• url (str: http://172.17.0.1:9200): Elasticsearch URL (Required)
• user (str: None): User for connection to Elasticsearch

[es_enrichment]

• autorefresh (bool: True): Execute the autorefresh of identities
• autorefresh_interval (int: 2): Time interval (days) to autorefresh identities
• password (str: None): Password for connection to Elasticsearch
• url (str: http://172.17.0.1:9200): Elasticsearch URL (Required)
• user (str: None): User for connection to Elasticsearch

[general]

• bulk_size (int: 1000): Number of items to write in Elasticsearch using bulk operations
• debug (bool: True): Debug mode (logging mainly) (Required)
• logs_dir (str: logs): Directory with the logs of sirmordred (Required)
• min_update_delay (int: 60): Short delay between tasks (collect, enrich ...)
• scroll_size (int: 100): Number of items to read from Elasticsearch when scrolling
• short_name (str: Short name): Short name of the project (Required)
• update (bool: False): Execute the tasks in loop (Required)
• aliases_file (str: ./aliases.json): JSON file to define aliases for raw and enriched indexes
• menu_file (str: ./menu.yaml): YAML file to define the menus to be shown in Kibiter
• global_data_sources (list: bugzilla, bugzillarest, confluence, discourse, gerrit, jenkins, jira): List of data sources collected globally, they are declared in the section 'unknown' of the projects.json
• retention_time (int: None): the maximum number of minutes wrt the current date to retain the data

[panels]

• community (bool: True): Include community section in dashboard
• kibiter_default_index (str: git): Default index pattern for Kibiter
• kibiter_time_from (str: now-90d): Default time interval for Kibiter
• kibiter_url (str): Kibiter URL (Required)
• kibiter_version (str: None): Kibiter version
• kafka (bool: False): Include KIP section in dashboard
• github-comments (bool: False): Enable GitHub comments menu. Note that if enabled, the github2:issue and github2:pulls sections in the setup.cfg and projects.json should be declared
• github-repos (bool: False): Enable GitHub repo stats menu. Note that if enabled, the github:repo section in the setup.cfg and projects.json should be declared
• gitlab-issues (bool: False): Enable GitLab issues menu. Note that if enabled, the gitlab:issue section in the setup.cfg and projects.json should be declared
• gitlab-merges (bool: False): Enable GitLab merge requests menu. Note that if enabled, the gitlab:merge sections in the setup.cfg and projects.json should be declared
• mattermost (bool: False): Enable Mattermost menu
• code-license (bool: False): Enable code license menu. Note that if enabled, colic sections in the setup.cfg and projects.json should be declared
• code-complexity (bool: False): Enable code complexity menu. Note that if enabled, cocom sections in the setup.cfg and projects.json should be declared
• strict (bool: True): Enable strict panels loading
• contact (str: None): Support repository URL

[phases]

• collection (bool: True): Activate collection of items (Required)
• enrichment (bool: True): Activate enrichment of items (Required)
• identities (bool: True): Do the identities tasks (Required)
• panels (bool: True): Load panels, create alias and other tasks related (Required)

[projects]

• projects_file (str: projects.json): Projects file path with repositories to be collected grouped by projects
• projects_url (str: None): Projects file URL, the projects_file is required to store the file locally

[sortinghat]

• affiliate (bool: True): Affiliate identities to organizations (Required)
• autogender (bool: False): Add gender to the profiles (executes autogender)
• autoprofile (list: ['customer', 'git', 'github']): Order in which to get the identities information for filling the profile (Required)
• database (str: sortinghat_db): Name of the Sortinghat database (Required)
• host (str: mariadb): Host with the Sortinghat database (Required)
• identities_api_token (str: None): API token for remote operation with GitHub and Gitlab
• identities_export_url (str: None): URL in which to export the identities in Sortinghat
• identities_file (list: []): File path with the identities to be loaded in Sortinghat
• identities_format (str: sortinghat): Format of the identities data to be loaded
• load_orgs (bool: False):
• matching (list: ['email']): Algorithm for matching identities in Sortinghat (Required)
• orgs_file (str: None): File path with the organizations to be loaded in Sortinghat
• password (str: ): Password to access the Sortinghat database (Required)
• reset_on_load (bool: False): Unmerge and remove affiliations for all identities on load
• sleep_for (int: 3600): Delay between task identities executions (Required)
• strict_mapping (bool: True): rigorous check of values in identities matching (i.e, well formed email addresses)
• unaffiliated_group (str: Unknown): Name of the organization for unaffiliated identities (Required)
• user (str: root): User to access the Sortinghat database (Required)

[backend-name:tag] (tag is optional)

• collect (bool: True): enable/disable collection phase
• raw_index (str: None): Index name in which to store the raw items (Required)
• enriched_index (str: None): Index name in which to store the enriched items (Required)
• studies (list: []): List of studies to be executed
• backend-param-1: ..
• backend-param-2: ..
• backend-param-n: ..

The template of a backend section is shown above. Further information about Perceval backends parameters are available at:

• Params details: http://perceval.readthedocs.io/en/latest/perceval.backends.core.html
• Examples: https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfg

Note that some backend sections allow to specify specific enrichment options, which are listed below.

[jenkins]

• node_regex: regular expression for extracting node name from builtOn field. This regular expression must contain at least one group. First group will be used to extract node name. More groups are allowed but not used to extract anything else.

[studies-name:tag] (tag is optional)

• study-param-1: ..
• study-param-2: ..
• study-param-n: ..

A template of a study section is shown above. A complete list of studies parameters is available at:

• https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfg

Projects.json

The projects.json aims at describing the repositories grouped by project that will be shown on the dashboards.

The project file enables the users to list the instances of the software development tools to analyse, such as local and remote Git repositories, the URLs of GitHub and GitLab issue trackers and the name of Slack channels. Furthermore, it also allows the users to organize these instances into nested groups, which structure is reflected in the visualization artifacts (i.e., documents and dashboards). Groups can be useful to represent projects within a single company, sub-projects within a large project such as Linux and Eclipse, or the organizations within a collaborative project.

1. First level: project names
2. Second level: data sources and metadata
3. Third level: data source URLs

There are some filters, labels, and a special section:

• Filter --filter-no-collection=true: This filter is used to show old enriched data within the dashboards from repositories that don't exist anymore in upstream.
• Filter --filter-raw and the section unknown: The data sources will only collected at the section unknown but this allow to add the same source in different sections to enrich using the filter --filter-raw.
• Label --labels=[example]: The data source will have the label of example which can be used to create visualisations for specific sets of data
• Section unknown: If the data source is only under this section it will be enriched as project main.

{
    "Chaoss": {
        "gerrit": [
            "gerrit.chaoss.org --filter-raw=data.projects:CHAOSS"
        ]
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github": [
            "https:/github.com/chaoss/grimoirelab-perceval --filter-no-collection=true",
            "https:/github.com/chaoss/grimoirelab-sirmordred  --labels=[example]"
        ]
    },
    "GrimoireLab": {
        "gerrit": [
            "gerrit.chaoss.org --filter-raw=data.projects:GrimoireLab"
        ],
        "meta": {
            "title": "GrimoireLab",
            "type": "Dev",
            "program" : "Bitergia",
            "state": "Operating"
        },
    }
    "unknown": {
        "gerrit": [
            "gerrit.chaoss.org"
        ],
        "confluence": [
            "https://wiki.chaoss.org"
        ]
}

In the projects.json above:

• The data included in the repo gerrit.chaoss.org will be collected entirely since the repo is listed in the unknown section. However only the project GrimoireLab will be enriched as declared in the GrimoireLab section.
• In the section Chaoss in the data source github the repository grimoirelab-perceval is not collected for raw index but it will enriched in the enriched index.
• In the section GrimoireLab the metadata will showed in the enriched index as extra fields.
• In the section unknown the data source confluence will be enriched as the project main.

Supported data sources

These are the data sources GrimoireLab supports:

askbot

Questions and answers from Askbot site

• projects.json

{
    "Chaoss": {
        "askbot": [
            "https://ask.puppet.com"
        ]
    }
}

• setup.cfg

[askbot]
raw_index = askbot_raw
enriched_index = askbot_enrcihed

bugzilla

Bugs from Bugzilla

• projects.json

{
    "Chaoss": {
        "bugzilla": [
            "https://bugs.eclipse.org/bugs/"
        ]
    }
}

• setup.cfg

[bugzilla]
raw_index = bugzilla_raw
enriched_index = bugzilla_enriched
backend-user = yyyy (optional)
backend-password = xxxx (optional)
no-archive = true (suggested)

bugzillarest

Bugs from Bugzilla server (>=5.0) using its REST API

• projects.json

{
    "Chaoss": {
        "bugzillarest": [
            "https://bugzilla.mozilla.org"
        ]
    }
}

• setup.cfg

[bugzillarest]
raw_index = bugzillarest_raw
enriched_index = bugzillarest_enriched
backend-user = yyyy (optional)
backend-password = xxxx (optional)
no-archive = true (suggested)

cocom

Code complexity integration. Some graal dependencies like cloc might be required, https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables

• projects.json

{
    "Chaoss":{
        "cocom": [
            "https://github.com/chaoss/grimoirelab-toolkit"
        ]
    }
}

• setup.cfg

[cocom]
raw_index = cocom_chaoss
enriched_index = cocom_chaoss_enrich
category = code_complexity_lizard_file
studies = [enrich_cocom_analysis]
branches = master
worktree-path = /tmp/cocom/

colic

Code license backend.

• projects.json

{
    "Chaoss":{
        "colic": [
            "https://github.com/chaoss/grimoirelab-toolkit"
        ]
    }
}

• setup.cfg

[colic]
raw_index = colic_chaoss
enriched_index = colic_chaoss_enrich
category = code_license_nomos
studies = [enrich_colic_analysis]
exec-path = /usr/share/fossology/nomos/agent/nomossa
branches = master
worktree-path = /tmp/colic

confluence

contents from Confluence

• projects.json

{
    "Chaoss": {
        "confluence": [
            "https://wiki.open-o.org/"
        ]
    }
}

• setup.cfg

[confluence]
raw_index = confluence_raw
enriched_index = confluence_enriched
no-archive = true (suggested)

crates

packages from Crates.io

• projects.json

{
    "Chaoss": {
        "crates": [
            ""
        ]
    }
}

• setup.cfg

[crates]
raw_index = crates_raw
enriched_index = crates_enriched

discourse

Topics from Discourse

• projects.json

{
    "Chaoss": {
        "discourse": [
            "https://foro.mozilla-hispano.org/"
        ]
    }
}

• setup.cfg

[discourse]
raw_index = discourse_raw
enriched_index = discourse_enriched
no-archive = true (suggested)

dockerhub

Repositories info from DockerHub

• projects.json

{
    "Chaoss": {
        "dockerhub": [
            "bitergia kibiter"
        ]
    }
}

• setup.cfg

[dockerhub]
raw_index = dockerhub_raw
enriched_index = dockerhub_enriched
no-archive = true (suggested)

dockerdeps

Dependencies extracted from Docker files. Requires https://github.com/crossminer/crossJadolint

• projects.json

{
    "Chaoss": {
        "dockerdeps": [
            "https://github.com/chaoss/grimoirelab"
        ]
    }
}

• setup.cfg

[dockerdeps]
raw_index = dockerdeps_raw
enriched_index = dockerdeps_enrich
category = code_dependencies_jadolint
exec-path = <jadolint-local-path>/jadolint.jar
in-paths = [Dockerfile, Dockerfile-full, Dockerfile-secured, Dockerfile-factory, Dockerfile-installed]

dockersmells

Smells extracted from Docker files. Requires https://github.com/crossminer/crossJadolint

• projects.json

{
    "Chaoss": {
        "dockersmells": [
            "https://github.com/chaoss/grimoirelab"
        ]
    }
}

• setup.cfg

[dockersmells]
raw_index = dockersmells_raw
enriched_index = dockersmells_enrich
category = code_quality_jadolint
exec-path = <jadolint-local-path>/jadolint.jar
in-paths = [Dockerfile, Dockerfile-full, Dockerfile-secured, Dockerfile-factory, Dockerfile-installed]

functest

Tests from functest

• projects.json

{
    "Chaoss": {
        "functest": [
            "http://testresults.opnfv.org/test/"
        ]
    }
}

• setup.cfg

[functest]
raw_index = functest_raw
enriched_index = functest_enriched
no-archive = true (suggested)

gerrit

Reviews from Gerrit

You have to add your public key in the gerrit server.

• projects.json

{
    "Chaoss": {
        "gerrit": [
            "review.opendev.org"
        ]
    }
}

• setup.cfg

[gerrit]
raw_index = gerrit_raw
enriched_index = gerrit_enriched
user = xxxx
no-archive = true (suggested)
blacklist-ids = [] (optional)
max-reviews = 500 (optional)
studies = [enrich_demography:gerrit, enrich_onion:gerrit] (optional)

[enrich_demography:gerrit] (optional)

[enrich_onion:gerrit] (optional)
in_index = gerrit_enriched
out_index = gerrit-onion_enriched

git

Commits from Git

• projects.json

{
    "Chaoss": {
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

• setup.cfg

[git]
raw_index = git_raw
enriched_index = git_enriched
latest-items = true (suggested)
studies = [enrich_demography:git, enrich_git_branches:git, enrich_areas_of_code:git, enrich_onion:git, enrich_extra_data:git] (optional)

[enrich_demography:git] (optional)

[enrich_git_branches:git] (optional)

[enrich_areas_of_code:git] (optional)
in_index = git_raw
out_index = git-aoc_enriched

[enrich_onion:git] (optional)
in_index = git_enriched
out_index = git-onion_enriched

[enrich_extra_data:git]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_forecast_activity]
out_index = git_study_forecast

github

Issues and PRs from GitHub

issue

• projects.json

{
    "Chaoss": {
        "github:issue": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

• setup.cfg

[github:issue]
raw_index = github_raw
enriched_index = github_enriched
api-token = xxxx
category = issue
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_onion:github, 
           enrich_geolocation:user, 
           enrich_geolocation:assignee, 
           enrich_extra_data:github,
           enrich_backlog_analysis] (optional)

[enrich_onion:github] (optional)
in_index_iss = github_issues_onion-src
in_index_prs = github_prs_onion-src
out_index_iss = github-issues-onion_enriched
out_index_prs = github-prs-onion_enriched

[enrich_geolocation:user] (optional)
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee] (optional)
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_backlog_analysis]
out_index = github_enrich_backlog
interval_days = 7
reduced_labels = [bug,enhancement]
map_label = [others, bugs, enhancements]

pull request

• projects.json

{
    "Chaoss": {
        "github:pull": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

• setup.cfg

[github:pull]
raw_index = github-pull_raw
enriched_index = github-pull_enriched
api-token = xxxx
category = pull_request
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github] (optional)

[enrich_geolocation:user]
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee]
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

repo

The number of forks, starts, and subscribers in the repository.

• projects.json

{
    "Chaoss": {
        "github:repo": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

• setup.cfg

[github:repo]
raw_index = github-repo_raw
enriched_index = github-repo_enriched
api-token = xxxx
category = repository
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_extra_data:github]

[enrich_extra_data:github]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

github2

Comments from GitHub

The corresponding dashboards can be automatically uploaded by setting github-comments to true in the panels section within the setup.cfg

issue

• projects.json

{
    "Chaoss": {
        "github2:issue": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

• setup.cfg

[github2:issue]
api-token = xxxx
raw_index = github2-issues_raw
enriched_index = github2-issues_enriched
sleep-for-rate = true
category = issue
no-archive = true (suggested)
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github2, enrich_feelings] (optional)

[enrich_geolocation:user] (optional)
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee] (optional)
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github2]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_feelings]
attributes = [title, body]
nlp_rest_url = http://localhost:2901

pull request

• projects.json

{
    "Chaoss": {
        "github2:pull": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

• setup.cfg

[github2:pull]
api-token = xxxx
raw_index = github2-pull_raw
enriched_index = github2-pull_enriched
sleep-for-rate = true
category = pull_request
no-archive = true (suggested)
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:git, enrich_feelings] (optional)

[enrich_geolocation:user] (optional)
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee] (optional)
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github2]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_feelings]
attributes = [title, body]
nlp_rest_url = http://localhost:2901

gitlab

Issues and MRs from GitLab

GitLab issues and merge requests need to be configured in two different sections. The corresponding dashboards can be automatically uploaded by setting gitlab-issue and gitlab-merge to true in the panels section within the setup.cfg

If a given GitLab repository is under more than 1 level, all the slashes / starting from the second level have to be replaced by %2F. For instance, for a repository with a structure similar to this one https://gitlab.com/Molly/lab/first.

issue

• projects.json

{
    "Chaoss": {
        "gitlab:issues": [
            "https://gitlab.com/Molly/first",
            "https://gitlab.com/Molly/lab%2Fsecond"
        ]
    }
}

• setup.cfg

[panels]
gitlab-issues = true

[gitlab:issue]
category = issue
raw_index = gitlab-issues_raw
enriched_index = gitlab-issues_enriched
api-token = xxxx
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_onion:gitlab-issue] (optional)

[enrich_onion:gitlab-issue] (optional)
in_index = gitlab-issues_enriched
out_index = gitlab-issues-onion_enriched
data_source = gitlab-issues

merge request

• projects.json

{
    "Chaoss": {
        "gitlab:merge": [
            "https://gitlab.com/Molly/first",
            "https://gitlab.com/Molly/lab%2Fsecond"
        ],
    }
}

• setup.cfg

[panels]
gitlab-merges = true

[gitlab:merge]
category = merge_request
raw_index = gitlab-mrs_raw
enriched_index = gitlab-mrs_enriched
api-token = xxxx
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_onion:gitlab-merge] (optional)

[enrich_onion:gitlab-merge] (optional)
in_index = gitlab-mrs_enriched
out_index = gitlab-mrs-onion_enriched
data_source = gitlab-merges

google_hits

Number of hits for a set of keywords from Google

• projects.json

{
    "Chaoss": {
        "google_hits": [
            "bitergia grimoirelab"
        ]
    }
}

• setup.cfg

[google_hits]
raw_index = google_hits_raw
enriched_index =google_hits_enriched

groupsio

Messages from Groupsio

To know the lists you are subscribed to: https://gist.github.com/valeriocos/ad33a0b9b2d13a8336230c8c59df3c55

• projects.json

{
    "Chaoss": {
        "groupsio": [
            "group1",
            "group2"
        ]
    }
}

• setup.cfg

[groupsio]
raw_index = groupsio_raw
enriched_index = groupsio_enriched
email = yyyy
password = xxxx

hyperkitty

Messages from a HyperKitty

• projects.json

{
    "Chaoss": {
        "hyperkitty": [
            "https://lists.mailman3.org/archives/list/[email protected]"
        ]
    }
}

• setup.cfg

[hyperkitty]
raw_index = hyperkitty_raw
enriched_index = hyperkitty_enriched

jenkins

Builds from a Jenkins

• projects.json

{
    "Chaoss": {
        "jenkins": [
            "https://build.opnfv.org/ci"
        ]
    }
}

• setup.cfg

[jenkins]
raw_index = jenkins_raw
enriched_index = jenkins_enriched
no-archive = true (suggested)

jira

Issues data from JIRA issue trackers

• projects.json

{
    "Chaoss":{
        "jira": [
            "https://jira.opnfv.org"
        ]
    }
}

• setup.cfg

[jira]
raw_index = jira_raw
enriched_index = jira_enriched
no-archive = true (suggested)
backend-user = yyyy (optional)
backend-password = xxxx (optional)

kitsune

Questions and answers from KitSune

• projects.json

{
    "Chaoss": {
        "kitsune": [
            ""
        ]
    }
}

• setup.cfg

[kitsune]
raw_index = kitsune_raw
enriched_index = kitsune_enriched

mattermost

Messages from Mattermost channels

• projects.json

{
    "Chaoss": {
        "mattermost": [
            "https://chat.openshift.io 8j366ft5affy3p36987pcugaoa"
        ]
    }
}

• setup.cfg

[mattermost]
raw_index = mattermost_raw
enriched_index = mattermost_enriched
api-token = xxxx

mbox

Messages from MBox files

For mbox files, it is needed the name of the mailing list and the path where the mboxes can be found. In the example below, the name of the mailing list is set to "mirageos-devel".

• projects.json

{
    "Chaoss": {
        "mbox": [
            "mirageos-devel /home/bitergia/mbox/mirageos-devel/"
        ]
    }
}

• setup.cfg

[mbox]
raw_index = mbox_raw
enriched_index = mbox_enriched

mediawiki

Pages and revisions from MediaWiki

-projects.json

{
    "Chaoss": {
        "mediawiki": [
            "https://www.mediawiki.org/w https://www.mediawiki.org/wiki"
        ]
    }
}

• setup.cfg

[mediawiki]
raw_index = mediawiki_raw
enriched_index = mediawiki_enriched
no-archive = true (suggested)

meetup

Events from Meetup groups

For meetup groups it is only needed the identifier of the meetup group and an API token: https://chaoss.github.io/grimoirelab-tutorial/gelk/meetup.html#gathering-meetup-groups-data

• projects.json

{
    "Chaoss": {
        "meetup": [
        "Alicante-Bitergia-Users-Group",
        "South-East-Bitergia-User-Group"
        ]
    }
}

• setup.cfg

[meetup]
raw_index = meetup_raw
enriched_index = meetup_enriched
api-token = xxxx
sleep-for-rate = true
sleep-time = "300" (optional)
no-archive = true (suggested)

mozillaclub

Events from Mozillaclub

• projects.json

{
    "Chaoss": {
        "mozillaclub": [
            "https://spreadsheets.google.com/feeds/cells/1QHl2bjBhMslyFzR5XXPzMLdzzx7oeSKTbgR5PM8qp64/ohaibtm/public/values?alt=json"
        ]
    }
}

• setup.cfg

[mozillaclub]
raw_index = mozillaclub_raw
enriched_index = mozillaclub_enriched

nntp

Articles from NNTP newsgroups

The way to setup netnews is adding the server and the news channel to be monitored. In the example below, the news.myproject.org is the server name.

• projects.json

{
    "Chaoss": {
        "nntp": [
            "news.myproject.org mozilla.dev.tech.crypto.checkins",
            "news.myproject.org mozilla.dev.tech.electrolysis",
            "news.myproject.org mozilla.dev.tech.gfx",
            "news.myproject.org mozilla.dev.tech.java"
        ]
    }
}

• setup.cfg

[nntp]
raw_index = nntp_raw
enriched_index =  nntp_enriched

phabricator

Tasks from Phabricator

• projects.json

{
    "Chaoss": {
        "phabricator": [
            "https://phabricator.wikimedia.org"
        ]
    }
}

• setup.cfg

[phabricator]
raw_index = phabricator_raw
enriched_index = phabricator_enriched
api-token = xxxx
no-archive = true (suggested)

pipermail

Messages from Pipermail

• projects.json

{
    "Chaoss": {
        "pipermail": [
            "https://lists.linuxfoundation.org/pipermail/grimoirelab-discussions/"
        ]
    }
}

• setup.cfg

[pipermail]
raw_index = pipermail_raw
enriched_index = pipermail_enriched

puppetforge

Modules and their releases from Puppet's forge

• projects.json

{
    "Chaoss": {
        "puppetforge": [
            ""
        ]
    }
}

• setup.cfg

[puppetforge]
raw_index = puppetforge_raw
enriched_index = puppetforge_enriched

redmine

Issues from Redmine

• project.json

{
    "Chaoss": {
        "redmine": [
            "http://tracker.ceph.com/"
        ]
    }
}

• setup.cfg

[redmine]
raw_index = redmine_raw
enriched_index = redmine_enriched
api-token = XXXXX

remo

Events, people and activities from ReMo

• project.json

{
    "Chaoss": {
        "remo": [
            "https://reps.mozilla.org"
        ]
    }
}

• setup.cfg

[remo]
raw_index = remo_raw
enriched_index = remo_enriched

rss

Entries from RSS feeds

• project.json

{
    "Chaoss": {
        "remo": [
            "https://reps.mozilla.org"
        ]
    }
}

• setup.cfg

[rss]
raw_index = rss_raw
enriched_index = rss_enriched

slack

Messages from Slack channels

The information needed to monitor slack channels is the channel id.

• projects.json

{
    "Chaoss": {
        "slack": [
            "A195YQBLL",
            "A495YQBM2"
        ]
    }
}

• setup.cfg

[slack]
raw_index = slack_raw
enriched_index = slack_enriched
api-token = xxxx
no-archive = true (suggested)

stackexchange

Questions, answers and comments from StackExchange

• projects.json

{
    "Chaoss": {
        "stackexchange": [
            "http://stackoverflow.com/questions/tagged/chef",
            "http://stackoverflow.com/questions/tagged/chefcookbook",
            "http://stackoverflow.com/questions/tagged/ohai",
            "http://stackoverflow.com/questions/tagged/test-kitchen",
            "http://stackoverflow.com/questions/tagged/knife"
        ]
    }
}

• setup.cfg

[stackexchange]
raw_index = stackexchange_raw
enriched_index = stackexchange_enriched
api-token = xxxx
no-archive = true (suggested)

supybot

Messages from Supybot log files

For supybot files, it is needed the name of the IRC channel and the path where the logs can be found. In the example below, the name of the channel is set to "irc://irc.freenode.net/atomic".

• projects.json

{
    "Chaoss": {
        "supybot": [
            "irc://irc.freenode.net/atomic /home/bitergia/irc/percevalbot/logs/ChannelLogger/freenode/#atomic"
        ]
    }
}

• setup.cfg

[supybot]
raw_index = supybot_raw
enriched_index = supybot_enriched

telegram

Messages from Telegram

You need to have an API token: https://github.com/chaoss/grimoirelab-perceval#telegram

• projects.json

{
    "Chaoss": {
        "telegram": [
            "Mozilla_analytics"
        ]
    }
}

• setup.cfg

[telegram]
raw_index = telegram_raw
enriched_index = telegram_enriched
api-token = XXXXX

twitter

Messages from Twitter

You need to provide a search query and an API token (which requires to create an app). The script at https://gist.github.com/valeriocos/7d4d28f72f53fbce49f1512ba77ef5f6 helps obtaining a token.

• projects.json

{
    "Chaoss": {
        "twitter": [
            "bitergia"
        ]
    }
}

• setup.cfg

[twitter]
raw_index = twitter_raw
enriched_index = twitter_enriched
api-token = XXXX

Troubleshooting

Following is a list of common problems encountered while setting up GrimoireLab

---

NOTE

In order to see the logs, run docker-compose up without the -d or --detach option while starting/(re)creating/building/attaching containers for a service.

---

Low Virtual Memory

• Indications: Cannot open https://localhost:9200/ in browser. Shows Secure connection Failed, PR_END_OF_FILE_ERROR, SSL_ERROR_SYSCALL in connection to localhost:9200 messages.

• Diagnosis: Check for the following log in the output of docker-compose up

  elasticsearch_1  | ERROR: [1] bootstrap checks failed
  elasticsearch_1  | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
  

• Solution: Increase the kernel max_map_count parameter of vm. Execute the following command sudo sysctl -w vm.max_map_count=262144 Now stop the container services and re-run docker-compose up. Note that this is valid only for current session. To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.

Permisssion Denied

• Indications: Can't create indices in Kibana. Nothing happens after clicking create index.

• Diagonosis: Check for the following log in the output of docker-compose up

  elasticsearch_1 |[INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=readall, roles=[readall], requestedTenant=null] [IndexType [index=.kibana, type=doc]] [Action [[indices:data/write/index]]] [RolesChecked [sg_own_index, sg_readall]]
  
  elasticsearch_1 | [c.f.s.c.PrivilegesEvaluator] No permissions for {sg_own_index=[IndexType [index=.kibana, type=doc]], sg_readall=[IndexType [index=.kibana, type=doc]]}
  
  kibiter_1 | {"type":"response","@timestamp":CURRENT_TIME,"tags":[],"pid":1,"method":"post","statusCode":403,"req":{"url":"/api/saved_objects/index-pattern?overwrite=false","method":"post","headers":{"host":"localhost:5601","user-agent":YOUR_USER_AGENT,"accept":"application/json, text/plain, /","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","content-type":"application/json;charset=utf-8","kbn-version":"6.1.4-1","content-length":"59","connection":"keep-alive"},"remoteAddress":YOUR_IP,"userAgent":YOUR_IP,"referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":403,"responseTime":25,"contentLength":9},"message":"POST /api/saved_objects/index-pattern?overwrite=false 403 25ms - 9.0B"} 
  

  or any type of 403 error.

• Solution: This message generally appears when you try to create an index pattern but you are not logged in Kibana. Try logging in to Kibana (the login button is on the bottom left corner).

Micro Mordred

Micro Mordred is a simplified version of Mordred which omits the use of its scheduler. Thus, Micro Mordred allows to run single Mordred tasks (e.g., raw collection, enrichment) per execution.

Micro Mordred is located in the /utils folder of this same repository. It can be executed via command line, its parameters are summarized below:

--debug: execute Micro Mordred in debug mode

--raw: activate raw task

--enrich: activate enrich task

--identities: activate merge identities task

--panels: activate panels task

--cfg: path of the onfiguration file
--backends: list of cfg sections where the active tasks will be executed

Examples of possible executions are shown below:

cd .../grimoirelab-sirmordred/utils/
micro.py --raw --enrich --cfg ./setup.cfg --backends git #  execute the Raw and Enrich tasks for the Git cfg section
micro.py --panels # execute the Panels task to load the Sigils panels to Kibiter

Getting started

SirModred relies on ElasticSearch, Kibiter and MySQL/MariaDB. The current versions used are:

• ElasticSearch 6.1.0
• Kibiter 6.1.4
• MySQL/MariaDB (5.7.24/10.0) There are 3 options to get started with SirMordred:

Source code

You will need to install ElasticSearch (6.1.0), Kibiter (6.1.4) and a MySQL/MariaDB database (5.7.24/10.0), and the following components:

• SirModred
• ELK
• Graal
• Perceval
• Perceval for Mozilla
• Perceval for OPNFV
• Perceval for Puppet
• Perceval for FINOS
• SortingHat
• Sigils
• Kidash
• Toolkit
• Cereslib
• Manuscripts

Source code and docker

You will have to install the GrimoireLab components listed above, and use the following docker-compose to have ElasticSearch, Kibiter and MariaDB. Note that you can omit the mariadb section in case you have MySQL/MariaDB already installed in your system.

elasticsearch:
  restart: on-failure:5
  image: bitergia/elasticsearch:6.1.0-secured
  command: elasticsearch -Enetwork.bind_host=0.0.0.0 -Ehttp.max_content_length=2000mb
  environment:
    - ES_JAVA_OPTS=-Xms2g -Xmx2g
  ports:
    - 9200:9200

kibiter:
  restart: on-failure:5
  image: bitergia/kibiter:secured-v6.1.4-5
  environment:
    - PROJECT_NAME=Development
    - NODE_OPTIONS=--max-old-space-size=1000
    - ELASTICSEARCH_URL=https://elasticsearch:9200
    - ELASTICSEARCH_USER=kibanaserver
    - ELASTICSEARCH_PASSWORD=kibanaserver
  links:
    - elasticsearch
  ports:
    - 5601:5601
    
mariadb:
  restart: on-failure:5
  image: mariadb:10.0
  expose:
    - "3306"
  ports:
    - "3306:3306"
  environment:
    - MYSQL_ROOT_PASSWORD=
    - MYSQL_ALLOW_EMPTY_PASSWORD=yes
    - MYSQL_DATABASE=test_sh
  command: --wait_timeout=2592000 --interactive_timeout=2592000 --max_connections=300
  log_driver: "json-file"
  log_opt:
      max-size: "100m"
      max-file: "3"

Save the above into a docker-compose.yml file and run

docker-compose up -d

to get ElasticSearch, Kibiter and MariaDB. Comment/remove the mariadb section in case you have MariaDB or MySQL already installed in your system.

You can read more about docker and docker-compose here

Only docker

Follow the instruction in the GrimoireLab tutorial to have SirMordred in a container

Setting up a Pycharm dev environment

This section provides details about how to set up a dev environment using PyCharm. The first step consists in forking the GitHub repos below and cloning them to a target local folder (e.g., sources).

• SirModred
• ELK
• Graal
• Perceval
• Perceval for Mozilla
• Perceval for OPNFV
• Perceval for Puppet
• Perceval for FINOS
• SortingHat
• Sigils
• Kidash
• Toolkit
• Cereslib
• Manuscripts

Each local repo should have two remotes: origin points to the forked repo, while upstream points to the original CHAOSS repo. An example is provided below.

~/sources/perceval$ git remote -v
origin	https://github.com/valeriocos/perceval (fetch)
origin	https://github.com/valeriocos/perceval (push)
upstream	https://github.com/chaoss/grimoirelab-perceval (fetch)
upstream	https://github.com/chaoss/grimoirelab-perceval (push)

In order to add a remote to a Git repository, you can use the following command:

~/sources/perceval$ git remote add upstream https://github.com/chaoss/grimoirelab-perceval

Then, you can download the PyCharm community edition, and create a project in the grimoirelab-sirmordred directory. PyCharm will automatically create a virtual env, where you should install the dependencies listed in each requirements.txt, excluding the ones concerning the grimoirelab components.

To install the dependencies, you can click on File -> Settings -> Project -> Project Interpreter, and then the + located on the top right corner (see figure below).

Later, you can add the dependencies to the grimoirelab components via File -> Settings -> Project -> Project Structure. The final results should be something similar to the image below.

Finally, you can use the docker-compose shown at Section source-code-and-docker, define a setup.cfg and projects.json, and run the following commands, which will collect and enrich the data coming from the git and cocom sections and upload the corresponding panels to Kibiter:

micro.py --raw --enrich --cfg ./setup.cfg --backends git cocom
micro.py --panels --cfg ./setup.cfg

Optionally, you can create a configuration in PyCharm to speed up the executions (Run -> Edit configuration -> +). The final results should be something similar to the image below.