1.4 CMS Computing Concepts: Programming, GitHub, etc

Complete:
Detailed Review status

Goals of this page

You will learn what programming language(s) you are expected to know, what software packages you should be familiar with, and how to access software from the code repository.

Note that in 2013, the CMS software repository was moved from CVS to git. CMS no longer uses CVS. The new git software repository is hosted at a website called GitHub, which provides extra features beyond git's file version control.

Contents

• Operating Systems
• Programming Languages
• Analysis Packages
• GitHub
• Git Documentation
• Git Tutorials and Info
• CMS Releases
• Review status#OpSys

Operating Systems

Familiarity with Linux is necessary. Current versions of the CMS Offline software, CMSSW, run on Scientific Linux CERN 6 (SLC6) (which is built on Scientific Linux 6 (SL6), which is in turn built on Red Hat Enterprise Linux 6 (RHEL6)). Run I (2010-2012) was based on SL*5. (The upgrading process, from versions SL*5 to SL*6, started in 2014, and the 7_0_0 release).

Programming Languages

CMSSW code is written in C++. You will need to be able to write compatible code in C++. There are many good text books available. See Walter Brown's (of FNAL) book recommendations for C++ Programmers .

Analysis Packages

You will need to know how to use ROOT. It is an object-oriented data analysis framework.

GitHub

CMS software, including CMSSW, is now maintained in a GitHub software repository. You'll need to use GitHub to check out code and configuration files to your area so that you can modify and use them. There are lots of working examples in GitHub for you to try; various workbook tutorials point you to specific directories. Since CMSSW is such a large project we use the sparse check-out features of git. Because of this it is highly recommended that you look at the CMS git tutorials to understand how this is done.

Here we give instructions on checking out code from GitHub. When you are working in the CMS environment, you can run any of the CMS code in the release without checking it out. You only need to "check out" code that you need to modify, code that has been modified by someone else, or code that depends on modified code. Then that code can be rebuilt.

A "release" is a set of software which has a version number and has been thoroughly tested to all work together. The CMS code evolves with stable, working releases numbered and advertised from time to time as a recommended set of code to use.

Git Documentation

The primary documentation for usage of git in CMS is at the following locations:

• http://cms-sw.github.io/cmssw/index.html
  • http://cms-sw.github.io/cmssw/tutorial.html
  • http://cms-sw.github.io/cmssw/faq.html
  • http://cms-sw.github.io/cmssw/advanced-usage.html

The information located there is much more detailed and complete that the information that follows.

(Warning: Early in the introduction of git into CMS, CMS created a thing called the Topic Collector which some documentation still references. The Topic Collector is no longer used for any purpose. GitHub directly offers features that replace that functionality and now we use them. If you see references or links to the Topic Collector, you should understand you are looking at obsolete documentation that needs to be updated)

Obtaining a personal GitHub account:

If you want to modify code and then have your modifications merged into the official repository of CMSSW code, you will need a personal GitHub account. You would first edit the code and commit it to a git repository on the machine where you work (maybe the machine on your desk or lxplus). Then you would push the changes to the repository in your personal GitHub account. Then you would issue a pull request to have the modifications pulled from your personal GitHub repository to the official CMSSW GitHub repository by a release manager.

A personal GitHub account has other uses. It is also a useful way to send your code modifications to other people or groups. It can also be used as a way to backup code modifications that you make and maintain version control of your own personal code.

But a personal GitHub account is only mandatory if you want to merge code into the official CMSSW code repository. You can download code, build it, and run it locally without a personal GitHub account.

First you have to obtain a GitHub account. Go to github.com and follow the instructions to open a new account if you do not already have one.

Then fork the official CMSSW repository. That creates a copy of it in your personal GitHub account. Fork by going to https://github.com/cms-sw/cmssw and click on the fork button and follow the instructions. After the forking is done you see you have a new repository cmssw, https://github.com/<your-username>/cmssw.

Setup personal information

• Login on lxplus at CERN or wherever you plan to do your code development work.
• Run the following commands:
  • git config --global user.name "First name Last name"
  • git config --global user.email <Your-Email-Address>
  • git config --global user.github <Your-Just-Created-GitHub-Account-Username>

Easy access of GitHub (optional)

• If you have no ssh key: Follow generating-ssh-keys on GitHub to create a ssh key.
• If you have already an ssh key: Login to GitHub -> settings -> SSH keys (https://github.com/settings/ssh) and add your key. I.e. copy paste the content of

yourKey.pub

• To use your key you have to execute following commands after each login:
  • eval $(ssh-agent); ssh-add /path/to/your/key
• Test your configuration: ssh -T git@github.com
  • You should see: Hi XYZ! You've successfully authenticated, but GitHub does not provide shell access.

Checkout a CMSSW package from Git

• Setup a CMSSW environment

cmsrel CMSSW_7_4_15
cd CMSSW_7_4_15/src
cmsenv

• Checkout a package, e.g. PhysicsTools/PatExamples

git cms-addpkg PhysicsTools/PatExamples

No release tags specified, using default CMSSW_7_4_15.
...
You are on branch from-CMSSW_7_4_15
Checking out packages
PhysicsTools/PatExamples

* Create your own development branch e.g. mydev

git checkout -b mydev

Switched to a new branch 'mydev'

• Check which branches are available

git branch

  CMSSW_7_4_X
  from-CMSSW_7_4_15
* mydev

• Apply modifications

echo '#modfication' >> PhysicsTools/PatExamples/BuildFile.xml #this adds a line containing #modification at the end of BuildFile.xml

• Check if modification is recognized

git status

# On branch mydev
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   PhysicsTools/PatExamples/BuildFile.xml
#
no changes added to commit (use "git add" and/or "git commit -a")

• Add and commit your changes

git add PhysicsTools/PatExamples/BuildFile.xml
git commit -m "test message"

[mydev 6812826] test message
 1 files changed, 1 insertions(+), 0 deletions(-)

Your modifications were commited to your local git repository. You can push these changes to your personal GitHub repository with the following command.

git push my-cmssw mydev

Before you do this you will have to have already created your personal GitHub account and forked the cmssw repository. (Creating and forking are one time actions. They do not need to be done with every push, see above for more information).

If obtain this message:

ERROR: Repository not found.
fatal: The remote end hung up unexpectedly

git wasn't able to find the my-cmssw repository in you GitHub account. Check if you forked the https://github.com/cms-sw/cmssw repository. If the push is successful you see:

Counting objects: 9, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 436 bytes, done.
Total 5 (delta 4), reused 0 (delta 0)
To git@github.com:fhoehle/cmssw.git
 * [new branch]      mydev -> mydev

You visit your account at GitHub and click on the branches drop-down menu you see your new branch mydev. Inspect the PhysicsTools folder and you see that PatExamples were changed recently with a message test message.

It showed you are checkout done of specific packages and how you are able to save your local changes at your private GitHub account. Git has a lot more feature which help and support you in developing and keeping track of your code. Please visit GitHub.com for more detailed information on branches, merging them, tags, ...

Git Tutorials and Info

Tutorials

• https://indico.cern.ch/conferenceDisplay.py?confId=254365
• https://indico.cern.ch/conferenceDisplay.py?confId=254369
• https://indico.cern.ch/conferenceDisplay.py?confId=286055
• https://indico.cern.ch/conferenceDisplay.py?confId=286056

CMSSW releases

To find out which CMSSW releases are available, do:

scram list CMSSW

You'll see a list with names like CMSSW_x_y_z, some with other suffixes. The names point to directories, e.g.,

CMSSW CMSSW_7_4_15 -->  /cvmfs/cms.cern.ch/slc6_amd64_gcc491/cms/cmssw/CMSSW_7_4_15

For each listed directory, subsystems are under its src directory, i.e., under /cvmfs/cms.cern.ch/slc6_amd64_gcc491/cms/cmssw/CMSSW_7_4_15/src . As an example, let's list the subsystems under this directory:

ls /cvmfs/cms.cern.ch/slc6_amd64_gcc491/cms/cmssw/CMSSW_7_4_15/src

Alignment                   DQM                  IOMC               RecoEcal            SimCalorimetry
AnalysisAlgos               DQMOffline           IOPool             RecoEgamma          SimDataFormats
AnalysisDataFormats         DQMServices          IORawData          RecoHI              SimG4CMS
CalibCalorimetry            EgammaAnalysis       JetMETAnalysis     RecoJets            SimG4Core
CalibFormats                ElectroWeakAnalysis  JetMETCorrections  RecoLocalCalo       SimGeneral
CalibMuon                   EventFilter          L1Trigger          RecoLocalMuon       SimMuon
Calibration                 FastSimDataFormats   L1TriggerConfig    RecoLocalTracker    SimRomanPot
CalibTracker                FastSimulation       L1TriggerOffline   RecoLuminosity      SimTracker
CaloOnlineTools             Fireworks            MagneticField      RecoMET             SimTransport
CommonTools                 FWCore               Mixing             RecoMuon            SUSYBSMAnalysis
CondCore                    GeneratorInterface   MuonAnalysis       RecoParticleFlow    TauAnalysis
CondFormats                 Geometry             OnlineDB           RecoPixelVertexing  TBDataFormats
CondTools                   GeometryReaders      PackageList.cmssw  RecoPixelVZero      TopQuarkAnalysis
Configuration               HeavyFlavorAnalysis  PerfTools          RecoRomanPot        TrackingTools
DataFormats                 HeavyIonsAnalysis    PhysicsTools       RecoTauTag          TrackPropagation
DetectorDescription         HiggsAnalysis        QCDAnalysis        RecoTBCalo          Utilities
DiffractiveForwardAnalysis  HLTrigger            RecoBTag           RecoTracker         Validation
Documentation               HLTriggerOffline     RecoBTau           RecoVertex
DPGAnalysis                 IgTools              RecoCaloTools      RecoVZero

Each subsystem contains several packages, for example

ls /cvmfs/cms.cern.ch/slc6_amd64_gcc491/cms/cmssw/CMSSW_7_4_15/src/Alignment/

CocoaAnalysis     CocoaUtilities                  KalmanAlignmentAlgorithm     OfflineValidation
CocoaApplication  CommonAlignment                 LaserAlignment               ReferenceTrajectories
CocoaDaq          CommonAlignmentAlgorithm        LaserAlignmentSimulation     SurveyAnalysis
CocoaDDLObjects   CommonAlignmentMonitor          LaserDQM                     TrackerAlignment
CocoaFit          CommonAlignmentParametrization  MillePedeAlignmentAlgorithm  TwoBodyDecay
CocoaModel        CommonAlignmentProducer         MuonAlignment
CocoaToDDL        HIPAlignmentAlgorithm           MuonAlignmentAlgorithms

with directories such as interface (containing the header files), src (containing the *.cc files), plugins (defining the modules to be used), python (containing the configuration files), doc (containing files needed for reference manual documenation), test (containing test setup) and data (obsolete).

%RESPONSIBLE% SudhirMalik
%REVIEW% JuanEduardoRamirez - Dec 2015