WikiPrint - from Polar Technologies

Working on the curie machine

1. Online users manual

2. Job manager commands

3. Before starting a job

3.1. Specify the project name

Since January 2013, you must specify in the header from which project you will use computing time: 

#MSUB -A genxxx

3.2. QoS test

QoS (Quality of Service) is a test queue. You can have a maximum of 2 jobs in test queue, each of them is limited to 30min and 8 nodes (= 256tasks). In the job header you must add: 

#MSUB -Q test

and change the CPU time limit 

#MSUB -T 1800

4. Other job manager commands

5. Fat nodes / Thin nodes

Fat nodes for the IPSLCM5A-LR coupled model are slower than titane (130%). Thin nodes are two times faster than fat nodes for computations; they are as fast as fat nodes for post processing.

We decided to use thin nodes for computations and fat nodes for post processing. Be careful! Since November 21st 2012, you must use at least libIGCM_v2.0_rc1 to perform post processing on fat nodes.

The job header must include #MSUB -q standard to use thin nodes.

The job header must include #MSUB -q xlarge to use fat nodes.

5.1. SSD on standard node : how to use it for rebuild job

SSD usage could accelerate rebuild job. It's very usefull for medium and high resolution configuration like IPSLCM5A-MR. You have only to change header and RUN_DIR_PATH in rebuild.job. Take care you will run faster but cost will be multiplicated by a factor of 16 because standard node ie 16 cpus are dedicated. Beware of the size of the /tmp (64GB/node) : if you have configuration with very high resolution and very high output frequency, the /tmp of standard node could be too small; in this case see below. 

#MSUB -q standard # thin nodes
#MSUB -x  # exclusive node
RUN_DIR_PATH=/tmp/REBUILD_DIR_MR_$$

5.2. RAM on fat node : how to use it for rebuild job

If /tmp of standard node (64GB) is too small for you, you can use fat node. You have to know the /tmp of fat node is also of 64GB. You have to use the RAM disk which is 250GB/node. You have only to change header and RUN_DIR_PATH in rebuild.job. Take care you will run faster but cost will be multiplicated by a factor of 128 because fat node ie 128 cpus are dedicated. 

#MSUB -q xlarge # fat nodes
#MSUB -x  # exclusive node
RUN_DIR_PATH=/dev/shm/REBUILD_DIR_MR_$$

6. Tricks

7. How to use the ddt debuger for the coupled model (or any other MPMD mode)

7.1. MPI only

7.2. Hybrid MPI-OpenMP (use of mpirun -rankfile method)

8. Errors on curie when running simulations

8.1. Job error: KILLED ... WITH SIGNAL 15

slurmd[curie1006]: error: *** STEP 639264.5 KILLED AT 2012-08-01T17:00:29 WITH SIGNAL 15 ***

This error message means that the time limit is exceeded. To solve the problem type clean_month, increase the time limit (or decrease PeriodNb) and restart.

8.2. Isn't there restart files for LMDZ?

Problem:

Solution:

don't ask questions! Type clean_month and restart the simulation.

8.3. Errors when creating or transfering files

The file system $CCCWORKDIR, $CCCSTOREDIR, $SCRATCHDIR are delicate. The error messages look like: 

 Input/output error
 Cannot send after transport endpoint shutdown

Don't ask question and resubmit the job.

8.4. Job error: Segmentation fault

/var/spool/slurmd/job637061/slurm_script: line 534:   458 Segmentation fault      /bin/ksh -x ${TEMPO_SCRIPT}

If you have this kind of message don't ask question and resubmit the job.

8.5. Error when submitting jobs

This message: 

error: Batch job submission failed: Job violates accounting policy (job submit limit, user's size and/or time limits)

means that you have submitted too many jobs (wait for the jobs to end and resubmit), that your headers are not properly written, or that you did not specify on which genci project the computing time must be deducted. The ccc_mqinfo command returns the maximum number of jobs (to this day: 300 for 24h-max jobs, 8 for 72h-max jobs and 2 for test jobs (30 min and max 8 nodes)): 

ccc_mqinfo
Name    Priority  MaxCPUs  MaxNodes  MaxRun  MaxSub     MaxTime
------  --------  -------  --------  ------  ------  ----------
long          18     1024                 2       8  3-00:00:00 
normal        20                                300  1-00:00:00 
test          40                  8               2    00:30:00

8.6. Long waiting time before a job execution

The computation of the users priority is based on 3 cumulated criteria:

If your job is far down the waiting list and if you are working on different projects, use the project with the least computing time used.

This computation is not satisfying because we would prefer to encourage long simulations. We are looking for real examples of anormal waiting situations. Please take the time to give us your feedback.

8.7. Disk quota exceeded

Be careful to quotas on /scratch! Monitor them with the command ccc_quota. Destroy the temporary directories created by jobs that ended too early and that did not clear the $SCRATCHDIR/TMPDIR_IGCM and $SCRATCHDIR/RUN_DIR directories. You should have a 20 To quota on curie. 

> ccc_quota
Disk quotas for user xxxx:

             ------------------ VOLUME --------------------  ------------------- INODE --------------------
 Filesystem       usage        soft        hard       grace       files        soft        hard       grace
 ----------       -----        ----        ----       -----       -----        ----        ----       -----
    scratch       3.53T         20T         20T           -      42.61k          2M          2M           - 
      store           -           -           -           -      93.76k        100k        101k           - 
       work     232.53G          1T        1.1T           -      844.8k        1.5M        1.5M           -

9. REDO

Simulations with the IPSLCM5A coupled model are reproducible if you use the same Bands file for LMDZ. See trusting TGCC/curie on this webpage: ​http://webservices.ipsl.jussieu.fr/trusting/

10. Feedback

10.1. On November 20th 2012

The maintenance has noticed and corrected the last two problems.

10.2. In June 2012

The 100-yr simulation of piControl in June 2012 :

10.3. Error to watch in the post processing: WARNING Intra-file non-monotonicity. Record coordinate "time_counter" does not monotonically increase