Artifact of Mastery paper
| Main Author: | Anonymous |
|---|---|
| Format: | info software Journal |
| Bahasa: | eng |
| Terbitan: |
, 2021
|
| Online Access: |
https://zenodo.org/record/5507555 |
Daftar Isi:
- Evaluation of mergers This guide describes how to repoduce the evaluation described in our paper. We evaluate five tools: Mastery, IntelliMerge, JDime, jFSTMerge and GitMerge. Set up docker Firstly, you need to download the provided tar archive (mastery-artifact-image.tar), which is a Docker image containing all of the tools, environments and merge scenarios to reproduce the evaluation. You could download Docker from here. Once you have Docker installed, the next step is to load the image from this tar archive and run the image as a container. # load image docker load -i mastery-artifact-image.tar # run image docker run --name mastery-artifact-container -it mastery-artifact-image This will drop you into a shell inside the container. The rest of this guide will assume that you run commands from the ~ directory -- things may not work as expected otherwise. We have installed vim in the image for convenience to edit and view files. You can use Docker command cp to copy files into and out of the container. File structure bin/: jar files of semistructured and structured merging tools mastery/: source code of mastery merge-scenarios/: the dataset (i.e. all considered merge scenarios) of our evaluation scripts/: scripts to run the experiments and show the results README.md: this guidance Reproduce evaluation To run all mergers on all of our merge scenarios, you can use the following command: python3 scripts/run.py As we consider a large dataset, this execution may cost hours or days, please be patient. For speeding up, this script will run on about 3/43 / 43/4 CPUs in parallel. Some error messages may be shown when a merger fails on a merge scenario, which doesn't disturb the functionality of the script. The raw results will be located in ~/merge-scenarios-results/. We will analyze the results in metrics mentioned in our paper by ~/scripts/summarize.py. For a more customized reproducing, see also python3 scripts/run.py -h and python3 scripts/summarize.py -h. To summarize and show the results, you can use the following command: python3 scripts/summarize.py Two tables similar to the followings will be print. The first table replicates Table 2 and Figure 6. The second one replicates Figure 8 and Figure 9. config: Mastery found config: JDime found config: jFSTMerge found config: IntelliMerge found config: GitMerge found 40533 benchmarks found Tool Expected Unexpected Conflicting Failed ------------ -------------- -------------- ------------- ------------ Mastery 33602 (82.90%) 3148 (7.77%) 3782 (9.33%) 1 (0.00%) JDime 32200 (79.44%) 2406 (5.94%) 4538 (11.20%) 1389 (3.43%) jFSTMerge 30062 (74.17%) 3837 (9.47%) 6626 (16.35%) 8 (0.02%) IntelliMerge 9777 (24.12%) 24553 (60.58%) 3442 (8.49%) 2761 (6.81%) Gge 30643 (75.60%) 791 (1.95%) 9099 (22.45%) 0 (0.00%) Tool Conflict Block Time (s) ------------ ---------------- ---------- Mastery 7057 10.16 JDime 11319 25.11 jFSTMerge 12744 13.73 IntelliMerge 5655 4.3 GitMerge 17314 0.01