CMS Data Analysis School Pre-Exercises - Third Set

Introduction

This is the third set of CMSDAS exercises. The purpose of these exercises are for the workshop attendees to become familiar with the basic Grid tools used in CMS for user analysis. Please run and complete each of these exercises. However, unlike the previous sets of exercises, this set will take considerably longer. Having your storage space set up may take several days, Grid jobs run with some latency, and there can be problems. You should set aside about a week to complete these five exercises. The actual effort required is not the whole week but a few hours (more than the previous two sets). If, at any time problems are encountered with the exercise please e-mail CMSDASATLPC@fnal.gov with a detailed description of your problem. For CRAB questions unrelated to passing these exercises, to send feedback and ask for support in case of CRAB related problems, please consult the CRAB troubleshooting twiki. All CRAB users should subscribe to the very useful hn-cms-computing-tools@cern.ch hypernews forum.

This section assumes that you have an account on the LPC computers at FNAL. How to get one is explained here. However, those familiar with the CERN computing environment and somewhat familiar with CRAB can answer all the questions running at CERN only. Later on, you can check with your university contact for Tier 2 or Tier 3 storage area. Once you are granted the write permission to the specified site, you can use CRAB as the below exercise but store the output to your Tier 2 or Tier 3 storage area.

Questions for each exercise are in red font.

Post the answers in the online response form:

• • CMSDAS@LPC2018 pre-exercises indico page with submit form links

To perform this set of exercises, an LPC account, Grid Certificate, and CMS VO membership are required. You should already have these things, but if not, follow these instructions from the first set of exercises.

NOTE: Legend of colors for this tutorial:

GRAY background for the commands to execute  (cut&paste)

GREEN background for the output sample of the executed commands

BLUE background for the configuration files  (cut&paste)

PINK background for the code (EDAnalyzer etc.)  (cut&paste)

Exercise 12 - Verify your grid certificate is OK

This exercise depends on obtaining a grid certificate and VOMS membership, but does not depend on any previous exercises.

After you've followed all the instructions above and installed your grid certificate, you need to verify it has all the information needed. Please install your grid certificate also on cmslpc-sl6.fnal.gov

Login to cmslpc-sl6.fnal.gov and initialize your proxy:

voms-proxy-init -voms cms

Then run the following command:

voms-proxy-info -all | grep -Ei "role|subject"

subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ydoh/CN=419191/CN=Young Do Oh/CN=proxy
subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ydoh/CN=419191/CN=Young Do Oh
attribute : /cms/Role=NULL/Capability=NULL

If you do not have the first attribute line listed above, you have not completed the VO registration above and you must complete it before continuing. For CMSDAS@LPC2018 please submit your answers at the Google Form third set.

QUESTION 12 - Copy the output corresponding to the text in the green box above.

Exercise 13 - Obtain a /store/user area and setup CRAB

Obtain a /store/user area

This exercise depends on successfully completing Exercise 12. Completion of this exercise requires a users to have /store/user/YourUserName in Tier2 or Tier3 site. (ex, eos area at cmslpc). A user at cmslpc should get this automatically once they have a cmslpc account, but it will not be linked to your grid certificate unless you provided it during account application, you can put in a Fermilab ServiceNow request following directions for CMS Storage Space Request. This area would look like this for example /store/user/malik/ (eosls /store/user/malik on cmslpc-sl6.fnal.gov). To understand the proper way to access files on FNAL eos refer to Using EOS at LPC.

CRAB Introduction

In this exercise, you will learn an important tool CRAB, which is used in all the data analysis at CMS. CRAB is a utility to submit CMSSW jobs to distributed computing resources. By using CRAB you will be able to access CMS data and Monte-Carlo which are distributed to CMS aligned centres worldwide and exploit the CPU and storage resources at CMS aligned centres. You will also test your grid certificate and your cmslpc EOS storage element which will be useful during CMSDAS@LPC2018.

HELP or QUESTIONS about CRAB3: Follow the FAQ to get help with CRAB3.

The most recent CRAB3 tutorial which is always in the WorkBook under WorkBookCRABTutorial. This tutorial provides complete instructions for beginner and expert user to use CRAB3 in their studies. We strongly recommend you to learn the CRAB3 tutorial after you finish these exercises. In this exercise, you will use CRAB3 to generate a MC sample yourself and publish it to the DBS.

Setup CRAB

First, go back to the YOURWORKINGAREA/CMSSW_9_3_2/src directory as in Exercise 3 under YOURWORKINGAREA, and setup crab.

cd YOURWORKINGAREA/CMSSW_9_3_2/src 
source /cvmfs/cms.cern.ch/crab3/crab.sh #.csh for c-shells

This script always points to the latest version of CRAB3. After sourcing this script, it is possible to use CRAB from any directory. One can check that the crab command is indeed available and the version being used by executing:

which crab

/cvmfs/cms.cern.ch/crab3/slc6_amd64_gcc493/cms/crabclient/3.3.1611-comp2/bin/crab

or

crab --version

<!-- CRAB client v3.3.1611 -->
CRAB client v3.3.1611

The /store/user area is commonly used for output storage from CRAB. When you complete Exercise 13, you can follow these instructions to make sure you can read from and write to your space using CRAB command.

Login to cmslpc-sl6.fnal.gov and initialize your proxy:

voms-proxy-init -voms cms

Check if you can write to the /store/user/ area. The crab checkwrite command can be used by a user to check if he/she has write permission in a given LFN directory path (by default /store/user/<HN-username>/) in a given site. The syntax to be used is:

crab checkwrite --site= <site-name>

For example:

crab checkwrite --site=T3_US_FNALLPC

The output should look like this:

Show result... Hide result...

[aparker@cmslpc31 test]$ crab checkwrite --site=T3_US_FNALLPC
Will check write permission in the default location /store/user/<username>
Retrieving DN from proxy...
DN is: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=asparker/CN=764176/CN=Ashley Marie Parker
Retrieving username from SiteDB...
Username is: asparker
Validating LFN /store/user/asparker...
LFN /store/user/asparker is valid.
Will use `gfal-copy`, `gfal-rm` commands for checking write permissions
Will check write permission in /store/user/asparker on site T3_US_FNALLPC

Attempting to create (dummy) directory crab3checkwrite_20161207_191214 and copy (dummy) file crab3checkwrite_20161207_191214.tmp to /store/user/asparker

Executing command: env -i X509_USER_PROXY=/tmp/x509up_u49959 gfal-copy -p -v -t 180 file:///uscms/home/aparker/cmsdas2017/pyROOT/CMSSW_9_3_2_patch1/src/pyROOTforCMSDAS/test/crab3checkwrite_20161207_191214.tmp 'srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/crab3checkwrite_20161207_191214.tmp'
Please wait...

Successfully created directory crab3checkwrite_20161207_191214 and copied file crab3checkwrite_20161207_191214.tmp to /store/user/asparker

Attempting to delete file srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/crab3checkwrite_20161207_191214.tmp

Executing command: env -i X509_USER_PROXY=/tmp/x509up_u49959 gfal-rm -v -t 180 'srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/crab3checkwrite_20161207_191214.tmp'
Please wait...

Successfully deleted file srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/crab3checkwrite_20161207_191214.tmp

Attempting to delete directory srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/

Executing command: env -i X509_USER_PROXY=/tmp/x509up_u49959 gfal-rm -r -v -t 180 'srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/'
Please wait...

Successfully deleted directory srm://cmseos.fnal.gov:8443/srm/v2/server?SFN=/eos/uscms/store/user/asparker/crab3checkwrite_20161207_191214/

Checkwrite Result:
Success: Able to write in /store/user/asparker on site T3_US_FNALLPC
Exception AttributeError: AttributeError("'NoneType' object has no attribute 'rmtree'",) in <bound method TempDirectory.__del__ of <WMCore.Services.Requests.TempDirectory object at 0x7fa4631f1f90>> ignored

these instructions to do a CMS Storage Space Request

QUESTION 13 - What is name of your directory name in eos?

Exercise 14 - Generate and publish a QCD dataset with CRAB

CMSSW configuration file to generate MC events

In this section we provide an example of a CMSSW parameter-set configuration file to generate minimum bias events with the Pythia MC generator. We call it CMSDAS_MC_generation.py. Using CRAB to generate MC events requires some special settings in the CRAB configuration file, as we will show later.

We use the cmsDriver tool to generate our configuration file:

cmsDriver.py MinBias_13TeV_pythia8_TuneCUETP8M1_cfi  --conditions auto:run2_mc -n 10 --era Run2_2016 --eventcontent FEVTDEBUG --relval 100000,300 -s GEN,SIM --datatier GEN-SIM --beamspot Realistic50ns13TeVCollision --fileout file:step1.root --no_exec --python_filename CMSDAS_MC_generation.py

If successful, cmsDriver will return the following

GEN,SIM,ENDJOB
We have determined that this is simulation (if not, rerun cmsDriver.py with --data)
Step: GEN Spec: 
Loading generator fragment from Configuration.Generator.MinBias_13TeV_pythia8_TuneCUETP8M1_cfi
Step: SIM Spec: 
Step: ENDJOB Spec: 
Config file CMSDAS_MC_generation.py created

Feel free to browse the newly outputted CMSDAS_MC_generation.py.

Generating MC events locally

We want to test this Configuration file locally for a small number of events before we submit to CRAB for massive generation. To test this file, we can run

cmsRun CMSDAS_MC_generation.py

This MC generation code will then produce an EDM output file called step1.root with the content of a GEN-SIM data tier for 10 generated events.

Show Hide

[aparker@cmslpc31 src]$ cmsRun CMSDAS_MC_generation.py 
 ################################################################### 
 # WARNING: this module is deprecated.                             # 
 # Please use CondCore.CondDB.CondDB_cfi.py                        # 
 ################################################################### 
# Conditions read from  CMS_CONDITIONS  via FrontierProd 
     MSTU(12)       changed from              0 to          12345
1****************** PYINIT: initialization of PYTHIA routines *****************
==== LHAPDF6 USING PYTHIA-TYPE LHAGLUE INTERFACE ====
LHAPDF 6.1.6 loading /cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/lhapdf/6.1.6-ikhhed/share/LHAPDF/cteq6l1/cteq6l1_0000.dat
cteq6l1 PDF set, member #0, version 4; LHAPDF ID = 10042

 ==============================================================================
 I                                                                            I
 I              PYTHIA will be initialized for a p on p collider              I
 I                  at  13000.000 GeV center-of-mass energy                   I
 I                                                                            I
 ==============================================================================

 ******** PYMAXI: summary of differential cross-section maximum search ********

           ==========================================================
           I                                      I                 I
           I  ISUB  Subprocess name               I  Maximum value  I
           I                                      I                 I
           ==========================================================
           I                                      I                 I
           I   92   Single diffractive (XB)       I    7.1189D+00   I
           I   93   Single diffractive (AX)       I    7.1189D+00   I
           I   94   Double  diffractive           I    1.0176D+01   I
           I   95   Low-pT scattering             I    5.4002D+01   I
           I   96   Semihard QCD 2 -> 2           I    9.5773D+03   I
           I                                      I                 I
           ==========================================================

 ****** PYMULT: initialization of multiple interactions for MSTP(82) = 4 ******
        pT0 = 3.01 GeV gives sigma(parton-parton) = 5.33D+02 mb: accepted

 ****** PYMIGN: initialization of multiple interactions for MSTP(82) = 4 ******
        pT0 = 3.01 GeV gives sigma(parton-parton) = 2.15D+02 mb: accepted

 ********************** PYINIT: initialization completed **********************
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 at 07-Dec-2016 19:25:35.571 CST
Begin processing the 2nd record. Run 1, Event 2, LumiSection 1 at 07-Dec-2016 19:25:51.228 CST
Begin processing the 3rd record. Run 1, Event 3, LumiSection 1 at 07-Dec-2016 19:25:53.150 CST
Begin processing the 4th record. Run 1, Event 4, LumiSection 1 at 07-Dec-2016 19:26:11.600 CST
Begin processing the 5th record. Run 1, Event 5, LumiSection 1 at 07-Dec-2016 19:27:10.036 CST
Begin processing the 6th record. Run 1, Event 6, LumiSection 1 at 07-Dec-2016 19:28:06.588 CST
Begin processing the 7th record. Run 1, Event 7, LumiSection 1 at 07-Dec-2016 19:28:19.088 CST

     Advisory warning type 9 given after        6 PYEXEC calls:
     (PYPTIS:) Sorry, I got a heavy companion quark here. Not handled yet, giving up!
Begin processing the 8th record. Run 1, Event 8, LumiSection 1 at 07-Dec-2016 19:28:57.278 CST
Begin processing the 9th record. Run 1, Event 9, LumiSection 1 at 07-Dec-2016 19:28:57.283 CST
Begin processing the 10th record. Run 1, Event 10, LumiSection 1 at 07-Dec-2016 19:28:57.286 CST
1********* PYSTAT:  Statistics on Number of Events and Cross-sections *********

 ==============================================================================
 I                                  I                            I            I
 I            Subprocess            I      Number of points      I    Sigma   I
 I                                  I                            I            I
 I----------------------------------I----------------------------I    (mb)    I
 I                                  I                            I            I
 I N:o Type                         I    Generated         Tried I            I
 I                                  I                            I            I
 ==============================================================================
 I                                  I                            I            I
 I   0 All included subprocesses    I           10           333 I  7.130D+01 I
 I  11 f + f' -> f + f' (QCD)       I            1             0 I  7.715D+00 I
 I  12 f + fbar -> f' + fbar'       I            0             0 I  0.000D+00 I
 I  13 f + fbar -> g + g            I            0             0 I  0.000D+00 I
 I  28 f + g -> f + g               I            3             0 I  2.314D+01 I
 I  53 g + g -> f + fbar            I            0             0 I  0.000D+00 I
 I  68 g + g -> g + g               I            3             0 I  2.314D+01 I
 I  92 Single diffractive (XB)      I            2             2 I  7.119D+00 I
 I  93 Single diffractive (AX)      I            0             0 I  0.000D+00 I
 I  94 Double  diffractive          I            1             1 I  1.018D+01 I
 I  95 Low-pT scattering            I            0             7 I  0.000D+00 I
 I                                  I                            I            I
 ==============================================================================

 ********* Total number of errors, excluding junctions =        0 *************
 ********* Total number of errors, including junctions =        0 *************
 ********* Total number of warnings =                           1 *************
 ********* Fraction of events that fail fragmentation cuts =  0.00000 *********

1********* PYSTAT:  Statistics on Number of Events and Cross-sections *********

 ==============================================================================
 I                                  I                            I            I
 I            Subprocess            I      Number of points      I    Sigma   I
 I                                  I                            I            I
 I----------------------------------I----------------------------I    (mb)    I
 I                                  I                            I            I
 I N:o Type                         I    Generated         Tried I            I
 I                                  I                            I            I
 ==============================================================================
 I                                  I                            I            I
 I   0 All included subprocesses    I           10           333 I  7.130D+01 I
 I  11 f + f' -> f + f' (QCD)       I            1             0 I  7.715D+00 I
 I  12 f + fbar -> f' + fbar'       I            0             0 I  0.000D+00 I
 I  13 f + fbar -> g + g            I            0             0 I  0.000D+00 I
 I  28 f + g -> f + g               I            3             0 I  2.314D+01 I
 I  53 g + g -> f + fbar            I            0             0 I  0.000D+00 I
 I  68 g + g -> g + g               I            3             0 I  2.314D+01 I
 I  92 Single diffractive (XB)      I            2             2 I  7.119D+00 I
 I  93 Single diffractive (AX)      I            0             0 I  0.000D+00 I
 I  94 Double  diffractive          I            1             1 I  1.018D+01 I
 I  95 Low-pT scattering            I            0             7 I  0.000D+00 I
 I                                  I                            I            I
 ==============================================================================

 ********* Total number of errors, excluding junctions =        0 *************
 ********* Total number of errors, including junctions =        0 *************
 ********* Total number of warnings =                           1 *************
 ********* Fraction of events that fail fragmentation cuts =  0.00000 *********


------------------------------------
GenXsecAnalyzer:
------------------------------------
Before Filtrer: total cross section = 7.130e+10 +- 0.000e+00 pb
Filter efficiency (taking into account weights)= (10) / (10) = 1.000e+00 +- 0.000e+00
Filter efficiency (event-level)= (10) / (10) = 1.000e+00 +- 0.000e+00
After filter: final cross section = 7.130e+10 +- 0.000e+00 pb

=============================================

MessageLogger Summary

 type     category        sev    module        subroutine        count    total
 ---- -------------------- -- ---------------- ----------------  -----    -----
    1 GenXSecAnalyzer      -w GenXSecAnalyzer:                       5        5

 type    category    Examples: run/evt        run/evt          run/evt
 ---- -------------------- ---------------- ---------------- ----------------
    1 GenXSecAnalyzer      PostEndRun       PostEndRun       PostEndRun

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
Warning                 5                   5
Thanks for using LHAPDF 6.1.6. Please make sure to cite the paper:
  Eur.Phys.J. C75 (2015) 3, 132  (http://arxiv.org/abs/1412.7420)

QUESTION 14.1 - What is the file size of step1.root?

Generate and publish MC dataset using CRAB

CRAB is handled by a configuration file. In CRAB3, the configuration file is in Python language. Here we give an example CRAB configuration file to run the CMSDAS_MC_generation.py MC event generation code. We name it crabConfig_MC_generation.py.

Copy and paste the below code as crabConfig_MC_generation.py under YOURWORKINGAREA/CMSSW_9_3_2/src.

from WMCore.Configuration import Configuration
config = Configuration()

config.section_("General")
config.General.requestName = 'CMSDAS_MC_generation_test0'
config.General.workArea = 'crab_projects'

config.section_("JobType")
config.JobType.pluginName = 'PrivateMC'
config.JobType.psetName = 'CMSDAS_MC_generation.py'
config.JobType.allowUndistributedCMSSW = True

config.section_("Data")
config.Data.outputPrimaryDataset = 'MinBias'
config.Data.splitting = 'EventBased'
config.Data.unitsPerJob = 10
NJOBS = 10  # This is not a configuration parameter, but an auxiliary variable that we use in the next line.
config.Data.totalUnits = config.Data.unitsPerJob * NJOBS
config.Data.publication = True
config.Data.outputDatasetTag = 'CMSDAS2018_CRAB3_MC_generation_test0'

config.section_("Site")
config.Site.storageSite = 'T3_US_FNALLPC'

All available CRAB configuration parameters are defined at CRAB3Configuration.

Now let us try to submit this job via crab by

crab submit -c crabConfig_MC_generation.py

For the detail of the crab command, you can find them from CRAB3Commands. You will be requested to enter your grid certificate password. Then you should get an output similar to this:

Will use CRAB configuration file crabConfig_MC_generation.py
Importing CMSSW configuration CMSDAS_MC_generation.py
Finished importing CMSSW configuration CMSDAS_MC_generation.py
Sending the request to the server
Success: Your task has been delivered to the CRAB3 server.
Task name: 161209_211642:asparker_crab_CMSDAS_MC_generation_test0
Please use 'crab status' to check how the submission process proceeds.

Log file is /uscms/home/aparker/cmsdas2018/ThirdPreExercise/CMSSW_9_3_2/src/crab_projects/crab_CMSDAS_MC_generation_test0/crab.log

Now you might notice a directory called crab_projects is created under CMSSW_9_3_2/src/. See what is under that directory. After you submitted the job successfully, you can check the status of a task by executing the following CRAB command:

crab status [-t] <CRAB-project-directory>

In our case, we run:

crab status crab_projects/crab_CMSDAS_MC_generation_test0

The crab status command will produce an output containing the task name, the status of the task as a whole, the details of how many jobs are in which state (submitted, running, transfering, finished, cooloff, etc.) and the location of the CRAB log (crab.log) file. It will also print the URLs of two web pages that one can use to monitor the jobs. In summary, it should look something like this:

[aparker@cmslpc27 src]$ crab status crab_projects/crab_CMSDAS_MC_generation_test0/
CRAB project directory:      /uscms/home/aparker/cmsdas2018/ThirdPreExercise/CMSSW_9_3_2/src/crab_projects/crab_CMSDAS_MC_generation_test0
Task name:         161209_211642:asparker_crab_CMSDAS_MC_generation_test0
Grid scheduler:         crab3@vocms0107.cern.ch
Task status:         SUBMITTED
Dashboard monitoring URL:   http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=asparker&table=Mains&pattern=161209_211642%3Aasparker_crab_CMSDAS_MC_generation_test0

Jobs status:         idle          100.0% (10/10)

No publication information available yet
Log file is /uscms/home/aparker/cmsdas2017/ThirdPreExercise/CMSSW_9_3_2/src/crab_projects/crab_CMSDAS_MC_generation_test0/crab.log

Now you can take a break and have some fun. Come back after couple hours and check the status again.

[aparker@cmslpc31 src]$ crab status crab_projects/crab_CMSDAS_MC_generation_test0/
CRAB project directory:      /uscms/home/aparker/cmsdas2017/ThirdPreExercise/CMSSW_9_3_2/src/crab_projects/crab_CMSDAS_MC_generation_test0
Task name:         161208_020043:asparker_crab_CMSDAS_MC_generation_test0
Grid scheduler:         crab3@vocms0198.cern.ch
Task status:         COMPLETED
Dashboard monitoring URL:   http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=asparker&table=Mains&pattern=161208_020043%3Aasparker_crab_CMSDAS_MC_generation_test0

Jobs status:         finished      100.0% (10/10)

Publication status:      finished      100.0% (10/10)

Output dataset:         /MinBias/asparker-CMSDAS2017_CRAB3_MC_generation_test0-c0565b08f5deea29d848445803c2f2d6/USER
Output dataset DAS URL:      https://cmsweb.cern.ch/das/request?input=%2FMinBias%2Fasparker-CMSDAS2017_CRAB3_MC_generation_test0-c0565b08f5deea29d848445803c2f2d6%2FUSER&instance=prod%2Fphys03
Log file is /uscms/home/aparker/cmsdas2017/ThirdPreExercise/CMSSW_9_3_2/src/crab_projects/crab_CMSDAS_MC_generation_test0/crab.log

NOTE: If at cmslpc, it will write out to your eos area. You can access them from /eos/uscms/store/user/USERNAME/subdir. Take a look at that directory.

From the bottom of the output, you can see the name of the dataset and the DAS link to it. Congratulations! This is the your first CMS dataset.

QUESTION 14.2 - What is the dataset name you published?

Exercise 15 - Running on dataset with CRAB

Now we're going to apply what you've learned using CRAB to the MiniAOD exercises you've been working on in the first two sets of exercises. Make sure that you finished and still have the scripts from Exercise 7 under the YOURWORKINGAREA/CMSSW_9_3_2_patch1/src .

Set up CRAB to run your MiniAOD jobs

If you forget, go back to the YOURWORKINGAREA/CMSSW_9_3_2/src and setup crab.

source /cvmfs/cms.cern.ch/crab3/crab.sh

Create another new CRAB config file crabConfig_data_slimMiniAOD.py with the following content:

from WMCore.Configuration import Configuration
config = Configuration()

config.section_("General")
config.General.requestName = 'CMSDAS_Data_analysis_test0'
config.General.workArea = 'crab_projects'

config.section_("JobType")
config.JobType.pluginName = 'Analysis'
config.JobType.psetName = 'slimMiniAOD_data_MuEle_cfg.py'
config.JobType.allowUndistributedCMSSW = True

config.section_("Data")
config.Data.inputDataset = '/DoubleMuon/Run2016C-03Feb2017-v1/MINIAOD'
config.Data.inputDBS = 'global'
config.Data.splitting = 'LumiBased'
config.Data.unitsPerJob = 50
config.Data.lumiMask = 'https://cms-service-dqm.web.cern.ch/cms-service-dqm/CAF/certification/Collisions16/13TeV/Cert_271036-275783_13TeV_PromptReco_Collisions16_JSON.txt'
config.Data.runRange = '275776-275782'


config.section_("Site")
config.Site.storageSite = 'T3_US_FNALLPC'

Most of this file should be familiar by now, but a few things may be new. The runRange parameter is used to further limit your jobs to a range of what is in the lumiMask file. This is needed if your two input datasets overlap. That way you can control which events come from which datasets. Instructions how to do this are at https://twiki.cern.ch/twiki/bin/viewauth/CMS/PdmV2012Analysis.

Run CRAB

Now go through the same process for this config file. You submit it with

crab submit -c crabConfig_data_slimMiniAOD.py

crab status

After a while, you should see something like below:

CRAB project directory:         /u/user/ydoh/CMSSW_9_3_2/src/crab_projects/crab_CMSDAS_Data_analysis_test0
Task name:                      150806_085723:ydoh_crab_CMSDAS_Data_analysis_test0
Task status:                    SUBMITTED
Warning:                        The CRAB3 server backend detected lumis split across files in the input dataset. Will apply the necessary corrections in the splitting algorithms. You can ignore this message..
Glidemon monitoring URL:        http://glidemon.web.cern.ch/glidemon/jobs.php?taskname=150806_085723%3Aydoh_crab_CMSDAS_Data_analysis_test0
Dashboard monitoring URL:       http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=crab&refresh=0&table=Jobs&p=1&records=25&activemenu=2&status=&site=&tid=150806_085723%3Aydoh_crab_CMSDAS_Data_analysis_test0
Details:                        running        12.5% ( 4/32)
                                transferring   87.5% (28/32)

No publication information (publication has been disabled in the CRAB configuration file)
Log file is /u/user/ydoh/CMSSW_9_3_2_patch1/src/crab_projects/crab_CMSDAS_Data_analysis_test0/crab.log

Create reports of data analyzed

When each of the sets of jobs is complete, all the output is staged, etc., run crab status and crab report one final time:

crab status
crab report

crab report prints to the screen how many events were analyzed.

QUESTION 15 - How many events were analyzed?

Optional: View the reconstructed Z peak in the combined data

Use the FWLiteHistograms executable you were using in the previous exercises to aggregate the data from all the CRAB output files. The root files created in the above step have been kept at the directory below: /eos/uscms/store/user/USER/OutputDIRECTORY/DoubleMu/CRAB3_CMSDAS_Data_analysis_test0/ One can use the same command:

FWLiteHistograms inputFiles=File1.root, File2.root, File3.root, File4.root outputFile=ZPeak_data.root maxEvents=-1 outputEvery=100

Exercise 16 - Combining the data and calculating luminosity

Install the BRIL Work Suite

We will use the BRIL work suite, a commandline toolkit for CMS Beam Radiation Instrumentation and Luminosity to calculate the total luminosity of the data we ran over.

Refer to the documentation for further information on BRIL.

Login to lxplus (not the LPC!) then enter commands:

export PATH=$HOME/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.1.7/bin:$PATH
pip install --install-option="--prefix=$HOME/.local" brilws

Find the luminosity for the dataset

From crab report, the location of JSON-formatted report file is listed:

...

Analyzed luminosity sections written to /uscms_data/d3/aparker/CMSSW_9_3_2_patch1/src/SingleElectron_Run2016B-Sep23/results/processedLumis.json
...

These files contains the list of lumi sections that CRAB successfully processed for that set of jobs. You can look at them with the cat or more command if you like. Next, let's figure out how much luminosity was run on by our jobs.

The first step is to copy the processedLumis.py file from the CRAB report to YOUR ~./local/bin/ directory on lxplus. First login onto the usual machine and cd to the directory where the file processedLumis.json resides. Then:

scp processedLumis.json lxplus.cern.ch:.local/bin/

Next login to lxplus to execute the commands:

cd .local/bin/

./brilcalc lumi -b "STABLE BEAMS" --normtag /afs/cern.ch/user/l/lumipro/public/normtag_file/normtag_DATACERT.json -i processedLumis.json -u /fb

The end of the output should look similar to this:

#Summary: 
+-------+------+-------+-------+-------------------+------------------+
| nfill | nrun | nls   | ncms  | totdelivered(/fb) | totrecorded(/fb) |
+-------+------+-------+-------+-------------------+------------------+
| 9     | 37   | 17377 | 17377 | 2.761             | 2.646            |
+-------+------+-------+-------+-------------------+------------------+
#Check JSON:
#(run,ls) in json but not in results: [(275890, 721)]

So the total recorded luminosity for this CRAB job is 2.6 /fb!

QUESTION 16 - What is the reported number of inverse femtobarns analyzed? (HINT: It is not the same sample as listed above with luminosity 2.6 /fb ) For CMSDAS@LPC2018 please submit your answers at the Google Form third set.

Where to find more on CRAB

• CRAB Home
• CRAB FAQ
• WorkBookGridJobDiagnosisTemplate: Steps to identify the problems you experience with your grid analysis jobs.
• CRAB mailing list where to send feedback and ask support in case of jobs problem (please send to us your crab.cfg file and the job stderr - stdout - log otherwise we are not able to provide support)

Note also that all CMS members using the Grid must subscribe to the Grid Annoucements CMS.HyperNews forum.

Link to SWGuideCMSDataAnalysisSchoolPreExerciseFirstSet

Link to SWGuideCMSDataAnalysisSchoolPreExerciseSecondSet

Link to SWGuideCMSDataAnalysisSchoolPreExerciseFourthSet

Link to SWGuideCMSDataAnalysisSchoolPreExerciseFifthSet

Link to SWGuideCMSDataAnalysisSchoolPreExerciseSixthSet

Show  Hide 

-- Sudhir Malik - 2017-12-15

Comments