-
Notifications
You must be signed in to change notification settings - Fork 47
The MDSplus code managment and build system
Currently MDSplus releases are built for a large number of platforms using Jenkins Open Source Continuous Integration Server. Jenkins jobs are configured to build MDSplus for each of the supported platforms.
These jobs are triggered by various Jenkins trigger jobs including one to test pull requests on github, one to build MDSplus alpha installers and one to build MDSplus stable installer. The pull request trigger job is automatically triggered when new pull requests are entered on github or changes are made to an existing pull request if the pull request was created by a MDSplus core developer. If another person submits a pull request a core developer must examine the changes included in the pull request and if safe to build and test the core developer can trigger a pull request test by entering a comment containing the string "ok to test". Core developers of MDSplus can also trigger a test of a pull request by entering a comment for the pull request containing the string "retest this please". No pull requests can be merged into the MDSplus code base unless the changes build and test successfully on Jenkins.
Each night the release trigger jobs are executed. These jobs fetch either the alpha or stable branch of MDSplus from github, determine whether changes have been made to the branch since the last MDSplus release was created and if so, determine whether a new release and the type of version increment needed based on the comment headers of the commits added to the branch. If the commits include a "Feature: xxx" header then the minor (second number) version increment will be made. If no new features are found and a commit with a "Fix: xxx" header is found then the patch (third number) version increment will be made. If no version change is deemed necessary no new release will be generated.
Each Jenkins trigger job is configured to trigger a Jenkins build jobs for each of the supported platforms including build options such as branch and version info and whether to test the code, build an installer or publish an installer. The pull request test job will trigger jobs to compile and test the code on all the platforms and build installers to ensure the changes did not alter the installer contents unexpectedly. If all the build jobs complete successfully the pull request test will be deemed successful and the pull request can be merged by a core developer after reviewing and accepting the proposed changes unless the branch submitted by the pull request is no longer up to date with the target branch. If the pull request has fallen behind the changes will need to be rebased onto the target branch and the branch used for the pull request will need to be updated on github which will restart the testing process. The release trigger jobs determine whether a new release version is needed and then triggers all the build jobs first to build, test and construct installers like the pull request trigger does but only if it has not yet successfully processed the version being built. If all testing completes successfully in all the build jobs, the release job will then trigger a second set of build jobs to publish the new releases. The build jobs will only publish a new release if the version being built has not already been published for that platform.
The trigger jobs all invoke the trigger shell script, deploy/trigger.sh.
Many of the build jobs use Docker to enable one system to build installation kits for many different platforms. Special docker images are constructed which have all the required compilers,utilities and libraries needed to build and test MDSplus for a particular computing platform.
The build jobs invoke the build shell script, deploy/build.sh.
MDSplus is open source software, distributed under the MIT License. The source code management has moved from CVS to GIT and is hosted on git hub https://github.com/MDSplus/MDSplus and can be cloned or checked out from there.
We are using gitflow as our branch / merge / release strategy. There are many references describing this on the web. See: https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow, http://nvie.com/posts/a-successful-git-branching-model/, etc... The basic idea is that the head is the release branch. Bug fixes are done as branches from this branch and merged both to the development branch(alpha) and back to head(release). To work on a new feature, create a feature branch from the development (alpha) branch, call it anything you like and work away. The git mantra is 'commit often' (i.e. write down what you are doing in small steps). When the feature is ready, first merge development (in case development has changed) then merge it into development. I have conveniently glossed over the distinction between your local repo and the one on github, and not listed the commands to do these things, this paragraph is more of a general description. Details below.
To clone a local copy of the MDSplus git over https (will not be able to put back changes):
git clone https://github.com/MDSplus/MDSplus.git
To clone a local copy of the MDSplus git over ssh (your ssh key will need to be installed on github)
git clone [email protected]:MDSplus/MDSplus.git
Once cloned you will have a local directory under your current working directory called MDSplus, and it will be set to the alpha branch.
You should set up your name and email address so that git will know how to label your changes. From: http://git-scm.com/book/en/Getting-Started-First-Time-Git-Setup
git config --global user.name "Joshua Stillerman" git config --global user.email [email protected]
To work on the alpha branch:
$ git checkout alpha Switched to branch 'alpha'
To work on the release branch (note - may be called release not master):
$ git checkout master Switched to branch 'master'
To find out what branch you currently have (the current branch will be marked with a '*').
$ git branch alpha * master
To get started, do the email/username config above and then:
$ git clone [email protected]:MDSplus/MDSplus.git
To work on the alpha branch:
$ git checkout alpha
To merge in any new changes made to the github repository
$ git pull
To commit changes:
$ git status ### This will list what changes you made locally to anywhere in the alpha branch
To commit changes to modules you would do one of the following followed by a commit command:
$ git add mymodule.c ### Use if mymodule.c was changed or if it is new $ git rm mymodule.c ### Use if you removed mymodule.c
Then commit using:
$ git commit -m "I mucked with mymodule.c and any others I've included in this commit using add/rm"
You can also commit changes to specific file(s) without using a git add command by listing the files in the commit:
$ git commit -m "I mucked with mymodule.c" mymodule.c
Then push these changes back to the github repository
$ git push
The same technique works for doing bug fixes on the release branch:
To work on the stable branch:
$ git checkout stable $ git pull $ git status $ git add/rm ... $ git commit ... $ git push
We need to update the .gitignore file(s) in the directory tree so we won't see extraneous files listed by git status after doing a local build. (See: Gitignore info.)
These directions will have to include release strategy and how / when to push back to github.
Features are developed in feature branches from the alpha branch. The first step on working on a feature is make sure that your alpha branch is up to date. You will probably have already cloned the repo, but if not clone it:
git clone [email protected]:MDSplus/MDSplus.git
cd into your repository and switch to the alpha branch, then create a new branch called my-new-feature (or whatever you like) and switch to it:
$ git checkout alpha Already on 'alpha'
Check that there are no outstanding changes both locally and remotely.
$ git status # On branch alpha nothing to commit (working directory clean)
Note the 'fetch' command below, could /should be 'pull'.
$ git fetch origin alpha From github.com:MDSplus/MDSplus * branch alpha -> FETCH_HEAD $ git status # On branch alpha nothing to commit (working directory clean)
If there were problems above they should be dealt with 1st. Not sure how.... directions to follow.
Now create a branch to work on your new feature and check it out:
$ git branch feature-fix-Xt-userData $ git branch * alpha feature-fix-Xt-userData master my-new-feature $ git checkout feature-fix-Xt-userData Switched to branch 'feature-fix-Xt-userData' $ git branch alpha * feature-fix-Xt-userData master my-new-feature
Now make any changes you want to the code commenting as you go....
$ git status # On branch feature-fix-Xt-userData # Changed but not updated: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: CallbacksUil.c # no changes added to commit (use "git add" and/or "git commit -a")
See what is changed:
$ git diff CallbacksUil.c
diff --git a/traverser/CallbacksUil.c b/traverser/CallbacksUil.c
index 8421581..01abb80 100644
--- a/traverser/CallbacksUil.c
+++ b/traverser/CallbacksUil.c
@@ -1620,8 +1620,10 @@ static void TagsReset(Widget w, int nid)
void tag_button_proc(Widget w, int *tag)
{
Widget tag_widget = XtParent(w);
+ XtPointer temp;
int nid;
- XtVaGetValues(tag_widget, XmNuserData, &nid, NULL);
+ XtVaGetValues(tag_widget, XmNuserData, (XtArgVal)&temp, NULL);
+ nid = (intptr_t)temp;
switch (*tag) {
case TAGS_OK:
if (TagsApply(tag_widget, nid)&1) XtUnmanageChild(tag_widget);
@@ -1674,7 +1676,7 @@ void ModifyTags(Widget w, XtPointer client_data, XmListCallbackStruct *reason)
if (!XtIsManaged(tagsw)) {
if (num_selected == 1) {
int nid = get_nid(selections[0]);
- XtVaSetValues(tagsw, XmNuserData, nid, NULL);
+ XtVaSetValues(tagsw, XmNuserData, (XtArgVal)nid, NULL);
TagsReset(tagsw, nid);
XtManageChild(tagsw);
}
Add these modifications to git and commit them with a comment.
$ git add CallbacksUil.c $ git commit -m "userData arguments must always be sizeof XtPointer" [feature-fix-Xt-userData d18eb16] userData arguments must always be sizeof XtPointer 1 files changed, 4 insertions(+), 2 deletions(-)
Repeat the steps above for any other things you need to change. In this case some files in xmdsshr and mitdevices.
$ git status
# On branch feature-fix-Xt-userData
# Changed but not updated:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: b5910a.c
# modified: l8590_mem.c
#
no changes added to commit (use "git add" and/or "git commit -a")
$ git diff b5910a.c
diff --git a/mitdevices/b5910a.c b/mitdevices/b5910a.c
index ac77338..04154dd 100644
--- a/mitdevices/b5910a.c
+++ b/mitdevices/b5910a.c
@@ -467,11 +467,13 @@ static void SetPoint(Widget w)
float x;
float y;
int num = sscanf(text, " ( %g , %g ) ",&x,&y);
+ XtPointer temp;
int userData;
int channel;
int idx;
Widget dw;
- XtVaGetValues(w,XmNuserData,&userData,NULL);
+ XtVaGetValues(w,XmNuserData,&temp,NULL);
+ userData= (intptr_t)temp;
channel = userData/65536;
idx = userData % 65536;
if (channel >= 0 && channel < 4 && num == 2)
$ git diff l8590_mem.c
diff --git a/mitdevices/l8590_mem.c b/mitdevices/l8590_mem.c
index 644c77a..d8a3eef 100644
--- a/mitdevices/l8590_mem.c
+++ b/mitdevices/l8590_mem.c
@@ -187,9 +187,11 @@ static void Load(Widget w)
static char nodename[13];
static NCI_ITM itmlst[] = {{12,NciNODE_NAME,nodename,0},{0,0,0,0}};
int i;
+ XtPointer temp;
int nid;
int found = False;
- XtVaGetValues(w, XmNuserData, &nid, NULL);
+ XtVaGetValues(w, XmNuserData, &temp, NULL);
+ nid = (intptr_t)temp;
l8590_memname = TreeGetPath(nid);
XmListDeleteAllItems(w);
for (i=1;i<17;i++)
$ git add b5910a.c l8590_mem.c
$ git commit -m "userData arguments must always be sizeof XtPointer"
[feature-fix-Xt-userData d21cbdf] userData arguments must always be sizeof XtPointer
2 files changed, 6 insertions(+), 2 deletions(-)
and
$ git status # On branch feature-fix-Xt-userData # Changed but not updated: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: XmdsDisplay.c # modified: XmdsPath.c # modified: XmdsXdBox.c # no changes added to commit (use "git add" and/or "git commit -a") $ git commit -a -m "userData arguments must always be sizeof XtPointer" [feature-fix-Xt-userData d84895b] userData arguments must always be sizeof XtPointer 3 files changed, 15 insertions(+), 15 deletions(-)
Before pushing code to alpha test and verify that it is functional. Once satisfied, first check that alpha has not changed while you were working on it. Merge your changes with alpha, test again if necessary, and then push to github. $ git checkout alpha Switched to branch 'alpha'
$ git merge origin alpha Already up-to-date. Yeeah! $ git merge feature-fix-Xt-userData Updating 3f8523b..d84895b Fast-forward mitdevices/b5910a.c | 4 +++- mitdevices/l8590_mem.c | 4 +++- traverser/CallbacksUil.c | 6 ++++-- xmdsshr/XmdsDisplay.c | 8 ++++---- xmdsshr/XmdsPath.c | 8 ++++---- xmdsshr/XmdsXdBox.c | 14 +++++++------- 6 files changed, 25 insertions(+), 19 deletions(-) $ git status # On branch alpha nothing to commit (working directory clean)
Now after testing some more push back to github.
$ git push origin alpha Counting objects: 33, done. Delta compression using up to 8 threads. Compressing objects: 100% (17/17), done. Writing objects: 100% (17/17), 2.06 KiB, done. Total 17 (delta 13), reused 0 (delta 0) To [email protected]:MDSplus/MDSplus.git 083a808..d84895b alpha -> alpha
This could be simplified to just 'git push', since the origin in the remote github, and current branch is alpha.
The next time an alpha kit is built, it will include these changes.
If you would like to contribute to MDSplus as a developer there are several ways to get started. Feel free to contact one of [email protected], [email protected] or [email protected] to discuss your ideas. The mechanics of contributing are one of:
- Clone the git repository and work locally.
- Fork the repository and put your work on github for all to see.
- Contact us as above for write access to the repository.
Differences between Alpha sent over to Stable Just the README and the .gitignore
first look at git log on alpha so you have the ids of the commits you want
git checkout alpha git log
Then cherry-pick the first one:
$ git checkout stable Switched to branch 'stable' $ git cherry-pick 59f9d5d3b80877224266871f0bb0e398bb78fe2d Finished one cherry-pick. [stable eae3b1e] adding gitignore 1 files changed, 215 insertions(+), 0 deletions(-) create mode 100644 .gitignore $ git status # On branch stable # Your branch is ahead of 'origin/stable' by 1 commit. # nothing to commit (working directory clean)
Everything is happy. No need to commit after successfully cherry-picking.
Now try to cherry-pick the second one - note this is incompatible!
git commit -c 7876c0232508ffda746a9d8c77127bea686c2060
see what git status says about things:
[jas@cmodws81 mdsplus]$ git status # On branch stable # Your branch is ahead of 'origin/stable' by 1 commit. # # Unmerged paths: # (use "git reset HEAD <file>..." to unstage) # (use "git add/rm <file>..." as appropriate to mark resolution) # # both modified: rpm/Makefile.in # no changes added to commit (use "git add" and/or "git commit -a")
IT WAS THE WRONG COMMIT JUST CHERRY-PICKED put it BACK
$ git reset HEAD Makefile.in Unstaged changes after reset: M rpm/Makefile.in
For some reason this did not put it back the way it was. The file still has conflict markers in it.
Put the file back as it was:
git checkout - - rpm/Makefile.in git add rpm/Makefile.in git commit -m “this file is unchanged”
Now cherry-pick the one we really want
$ git checkout stable Switched to branch 'stable' Your branch is ahead of 'origin/stable' by 2 commits. $ git rm README rm 'README' $ git cherry-pick ee7f40d5f96883be414322cbe871fb1effcca181 fatal: Your local changes would be overwritten by cherry-pick. Please, commit your changes or stash them to proceed.
Have to commit the removal of old one 1st....
$ git commit -m "remove the old README" [stable 16abe07] remove the old README 1 files changed, 0 insertions(+), 48 deletions(-) delete mode 100644 README $ git status # On branch stable # Your branch is ahead of 'origin/stable' by 3 commits. # nothing to commit (working directory clean)
Now cherry-pick again
$ git cherry-pick ee7f40d5f96883be414322cbe871fb1effcca181 Finished one cherry-pick. [stable 07109a7] Change README.INSTALL to describe git, and rename to README so github will display it 1 files changed, 84 insertions(+), 0 deletions(-) create mode 100644 README $ git status # On branch stable # Your branch is ahead of 'origin/stable' by 4 commits. # nothing to commit (working directory clean)
Now push it back to github stable
$ git branch alpha * stable $ git push Counting objects: 16, done. Delta compression using up to 8 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 9.57 KiB, done. Total 12 (delta 5), reused 1 (delta 0) To [email protected]:MDSplus/mdsplus.git 7cd81bc..07109a7 stable -> stable
After doing some updates a push back to origin may sometimes say:
$ git push To [email protected]:MDSplus/mdsplus.git ! [rejected] alpha -> alpha (non-fast-forward) error: failed to push some refs to '[email protected]:MDSplus/mdsplus.git' To prevent you from losing history, non-fast-forward updates were rejected Merge the remote changes before pushing again. See the 'Note about fast-forwards' section of 'git push --help' for details.
To fix this you might think a git merge origin alpha would be in order, but this just says that everything is up to date:
$ git merge origin alpha Already up-to-date. Yeeah!
What is needed is a git pull:
$ git pull
remote: Counting objects: 33, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 33 (delta 22), reused 20 (delta 9)
Unpacking objects: 100% (33/33), done.
From github.com:MDSplus/mdsplus
1a022a9..23affa6 alpha -> origin/alpha
7b5941c..48de130 stable -> origin/stable
* [new tag] alpha_release-6-3-426 -> alpha_release-6-3-426
* [new tag] stable_release-6-1-65 -> stable_release-6-1-65
Merge made by recursive.
ChangeLog |92414 +++++++++++++++++++++++++++--------------------------
deploy/release.py | 8 +-
2 files changed, 46228 insertions(+), 46194 deletions(-)
Then you can check the status and push:
$ git status # On branch alpha # Your branch is ahead of 'origin/alpha' by 2 commits. # nothing to commit (working directory clean) $ git push Counting objects: 8, done. Delta compression using up to 2 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (5/5), 689 bytes, done. Total 5 (delta 3), reused 0 (delta 0) To [email protected]:MDSplus/mdsplus.git 23affa6..7098fc9 alpha -> alpha