Home

WMArchive wiki pages

WMArchive is CMS data-service to store FrameWork JobReport (FWJR) documents. It provides a set of tools to store FWJR into short-term (MongoDB) and long-term (HDFS) storages, to convert FWJR from JSON to AVRO data-formats and end-user tools (myspark) to run a spark job to find FWJR on HDFS. Please use side bar to search for specific topic.

Technologies

• Big-data storage backend: Apache Hadoop
• Aggregation pipeline: Apache Spark, MongoDB, Python
• REST web server: WMArchive, Python
• Frontend: JavaScript with Backbone.js, HTML, CSS, Sass
• Visualization: D3.js

Documentation

Preparation:

• Running the WMArchive Server
• Generating sample data

Usage and Implementation:

• Performance data REST endpoint
• Performance UI architecture
• Common tasks

Aggregation and Data:

• Performance data structure
• Aggregation procedure

Outlook:

• Pending improvements
• Known issues