User Space at US CMS Tier-2 Centers and the FNAL LPC
Click here to get a FNAL EOS area and learn more about FNAL EOS
Overview
When running jobs on large amounts of data, many users will need to write output files of substantial size -- larger than can be accommodated by the grid output sandbox. Additionally, it can be very useful to write this output in a way that it can be read back in easily by subsequent grid jobs. CMS has a mechanism for this. Computing sites will support a /store/user area in their storage systems to host user-generated data. CRAB can be instructed to write job outputs to this area through the SRM interface. Through the magic of the grid, no matter where your job actually runs, and no matter where the output is created, the output can be automatically transferred to whatever system is hosting your own /store/user area. Then, CRAB can also publish this data into a local-scope DBS database. Subsequent CRAB jobs can then reference the local-scope DBS, find the user-created data, and send jobs to the site that is hosting it.
US CMS has provisioned the Tier-2 computing systems and the FNAL LPC cluster to provide a 1 TB quota of /store/user space to every US-based user who is actively involved in data analysis. Note that this space is not backed up in any way, so use it with care. This page contains information on how users can make use of this space.
Complete documentation on using the distributed computing system can be found in Chapter 5 of the CMS Workbook.
Assignment of users to computing sites
Which site (Storage Element) hosts your own /store/user area? Users are grouped by
institute, and then institutes are matched up with a Tier-2 site or
the LPC on the basis of geography and capacity. The LPC has about 42%
of the total available resources, and thus hosts the largest number of
users. Each of the seven Tier-2 sites hosts about 8% of the remaining
users. Leaders of the institutes are responsible for contacting the
sites to establish accounts. Note that your institute may also have a Tier3
with associated storage space, contact your institute leader for more information.
Institute assignments are as follows:
- Caltech: (Contact: Caltech T2 web page) Caltech, Livermore, UC Davis, UCLA
- Florida (Contact: Bockjoo Kim): FIU, Florida State, Florida Tech, Florida, Puerto Rico
- MIT (Contact: Max Goncharov): Boston U., Brown, Fairfield, MIT, Northeastern
- Nebraska (Contact: Ken Bloom): Colorado, Iowa, Kansas State, Kansas, Nebraska
- Purdue (Contact: Stefan Piperov): Carnegie-Mellon, Ohio State, Purdue, SUNY-Buffalo
- UCSD (Contact: T2 support team): UC Riverside, UCSB, UCSD
- Wisconsin (Contact: Ajit Mohapatra): Minnesota, Rochester, Wayne State, Wisconsin
- FNAL (Get FNAL account; Get EOS area configured): All remaining institutes
To get /store/user space at their associated site, users must provide
a) their CERN CMS user name, which will be used as the name of
their directory in /store/user and b) the distinguished name (DN) from
their grid certificate, which will be used to verify that the user is
allowed to write into that area. The DN can most easily be obtained
by issuing the voms-proxy-info command from wherever you do your grid
submissions.
Writing to your /store/user space from a CRAB job
Consult the SWGuideCrab Workbook for the most up-to-date documentation on techniques to write output from a CRAB job
You must specify config.Site.storageSite, which will depend on which center is hosting your area, and user_remote_dir which is the subdirectory of /store/user/ you want to write to.
- Caltech storage_element = T2_US_Caltech
- Florida storage_element = T2_US_Florida
- MIT storage_element = T2_US_MIT
- Nebraska storage_element = T2_US_Nebraska
- Purdue storage_element = T2_US_Purdue
- UCSD storage_element = T2_US_UCSD
- Wisconsin storage_element = T2_US_Wisconsin
- FNAL storage_element = T3_US_FNALLPC
Accessing /store/user files through CRAB
This is described fairly well in a twiki page on publishing data using CRAB.
Then, to run on our output files with a new CRAB job, you need to specify the new dataset name, and the URL for the local-scope DBS.Accessing /store/user through Xrootd
This is perhaps the easiest way to access your files in /store/user for interactive use, including reading them directly into ROOT. You don't even need to know the physical location of your file, just the path in the /store/user area. See this page in the CMS WorkBook for all the details.
Accessing /store/user by other (less convenient) means
You can copy files to your local machine via gfal-copy.
Note that you need a valid CMS grid proxy to do this operation. More about this
technique is at the data transfer page.
