Merge pull request #62 from IanHeywood/dev

Merge dev branch into master

README.md

@@ -2,7 +2,7 @@
 # oxkat
 
 <b>"Ceci n'est pas une pipe[line]"</b><br> 
-(_The Treachery of Images_, René Magritte, 1929)
+_The Treachery of Images_, René Magritte, 1929
 
 ---
 
@@ -19,7 +19,7 @@
 * A set of `Python` scripts with the aim of (semi-)automatically processing [MeerKAT](https://www.sarao.ac.za/science-engineering/meerkat/) data. 
 
 
-* At the core is a set of functions that generate calls to [various pieces](README.md#software-package-roll-call) of radio astronomy software, a semi-modular bunch of CASA scripts for performing reference calibration, and a fairly sizeable list of default parameters. The default at present cater for full-band Stokes-I continuum imaging at L-band or UHF.
+* At the core is a set of functions that generate calls to [various pieces](README.md#software-package-roll-call) of radio astronomy software, a semi-modular bunch of CASA scripts for performing reference calibration, and a [fairly sizeable list](oxkat/config.py) of default parameters. The defaults at present cater for full-band Stokes-I continuum imaging, including the correction of direction-dependent effects.
 
 
 * Job script generation and dependency chains are automatically handled when running on the [ilifu/IDIA](https://www.idia.ac.za/) cluster, UKZN's [hippo](https://astro.ukzn.ac.za/~hippo/) cluster, or the [CHPC](https://www.chpc.ac.za/)'s [Lengau](https://www.chpc.ac.za/index.php/resources/lengau-cluster) cluster.
@@ -28,7 +28,7 @@
 * Setup scripts glue the above components together into a processing recipe. The default procedure is broken down into stages, after each of which it is advisable to pause and examine the state of the process before continuing.  
 
 
-* The intention is that the bar to entry is low. If you have stock `Python` then nothing else needs installing apart from [`Singularity`](https://github.com/hpcng/singularity), which is available on both the ilifu/IDIA and CHPC clusters, in which case all the underlying radio astronomy packages are containerised. The `Singularity` layer can also be disabled for running installations on your own machine, either directly, or inside a Python virtual environment.
+* The intention is that the bar to entry is low. If you have stock `Python` then nothing else needs installing apart from [`Singularity`](https://github.com/hpcng/singularity), which is available on both the ilifu/IDIA and CHPC clusters. All the underlying radio astronomy packages are containerised. The `Singularity` layer can also be disabled for running installations on your own machine, either directly, or inside a Python virtual environment.
 
 
 * If you publish results that have made use of `oxkat` then [please cite the ACSL entry](https://ui.adsabs.harvard.edu/abs/2020ascl.soft09003H/abstract), and (more importantly) the [underlying packages](README.md#software-package-roll-call) used.
@@ -41,7 +41,7 @@
 
 ## Quick start
 
-1. If you have your [containers all set up](README.md#containers) then log into your machine or cluster, e.g.:
+1. Once you have the [container(s) in place](README.md#containers) then log into your machine or cluster, e.g.:
 
    ```
    $ ssh [email protected]
@@ -65,9 +65,14 @@
    $ ln -s /idia/projects/mightee/1538856059/1538856059_sdp_l0.full_1284.full_pol.ms .
    ```
 
-5. Ensure the setup is [configured](https://github.com/IanHeywood/oxkat/blob/dev/oxkat/config.py#L8) for either L-band (the default) or UHF (somewhat experimental).
+5. The first step is to run a script that gathers some required information about the observation:
 
-6. Generate and submit (or run) the jobs required for the reference calibration (1GC):
+   ```
+   $ python setups/0_GET_INFO.py idia
+   $ ./submit_info_job.sh
+   ```
+
+6. Once this is complete you can generate and submit the jobs required for the reference calibration (1GC):
 
    ```
    $ python setups/1GC.py idia
@@ -82,29 +87,15 @@
 
 8. Once all the jobs have completed then you can examine the products, and move on to the setup for the next steps in the same fashion. 
 
-Please see the [setups README](setups/README.md) for more details. Most of the settings can be tuned via the [`config.py`](oxkat/config.py) file. Note that for use on a cluster you might have to load a `Python 3` module. On the ilifu/IDIA cluster this is achieved with:
-
-   ```
-   $ module load anaconda3
-   ```
-
-and on the CHPC's Lengau cluster:
-
-   ```
-   $ module load chpc/python/3.7.0
-   ```
-
-These can also be added to your `~/.bashrc` file for simplicity.
+Please see the [setups README](setups/README.md) for more details about the general workflow. Most of the settings can be tuned via the [`config.py`](oxkat/config.py) file.
 
 ---
 
 ## Containers
 
-Singularity can be used to download and build containers from [Docker Hub](https://hub.docker.com/). There is a [script](https://github.com/IanHeywood/oxkat/blob/master/tools/pull_containers.sh) included to download them for you. [@SpheMakh](https://github.com/sphemakh)'s [stimela](https://hub.docker.com/u/stimela) project maintains containers for most radio astronomy applications, and repository of pre-built containers is now available at both IDIA and CHPC (in support of the [`CARACal`](https://github.com/caracal-pipeline) software).
-
-The default container paths are specified in the [`config.py`](oxkat/config.py) file. The scripts will select the required containers via pattern matching so if a container is replaced with a newer version it should be seamless.
+There is a dedicated `Singularity` container (`oxkat-0.41.sif`) available that contains all the necessary packages and dependencies. This is available in the general container repository on the ilifu/IDIA cluster, and the default settings should pick it up automatically when that cluster is being used. For other systems the container will have to be downloaded (or copied over). The container can be downloaded [here](https://entangled.physics.ox.ac.uk/index.php/s/jmHRBQyyB6Zm2fj). 
 
-The IDIA slurm head node does not have `Singularity` available, so if you are pulling your own containers that must be done either via a standalone node or a worker node, or otherwise copied over via the `transfer.ilifu.ac.za` node. You will not be able to use the `pull_containers.sh` script on the Lengau head node, and the worker nodes at CHPC do not have external connectivity, so you will have to build the containers elsewhere and then transfer them to CHPC via their `scp.chpc.ac.za` node.
+String patterns for package-specific containers are specified in the [`config.py`](oxkat/config.py) file. The scripts will search for containers that match these patterns in the container paths, so it's simple to swap a particular package out for a different version as long as you have it containerised.
 
 ---
 
@@ -113,15 +104,15 @@ The IDIA slurm head node does not have `Singularity` available, so if you are pu
 
 | Package | Stage | Purpose | Reference |
 | --- | --- | --- | --- | 
-| [`CASA`](https://casa.nrao.edu/) | 1GC, 2GC | Averaging, splitting, cross calibration, DI self-calibration, flagging | [McMullin et al., 2007](https://ui.adsabs.harvard.edu/abs/2007ASPC..376..127M/abstract)|
+| [`astropy`](https://www.astropy.org/) | 1GC, 3GC | Coordinates, time standards, FITS file manipulation | [Astropy Collaboration, 2013](https://ui.adsabs.harvard.edu/abs/2013A%26A...558A..33A/abstract), [Astropy Collaboration, 2018](https://ui.adsabs.harvard.edu/abs/2018AJ....156..123A/abstract)|
+| [`CASA`](https://casa.nrao.edu/) | 1GC | Averaging, splitting, cross calibration, DI self-calibration, flagging | [McMullin et al., 2007](https://ui.adsabs.harvard.edu/abs/2007ASPC..376..127M/abstract)|
 | [`CubiCal`](https://github.com/ratt-ru/CubiCal) | 2GC, 3GC | DI / DD self-calibration | [Kenyon et al., 2018](https://ui.adsabs.harvard.edu/abs/2018MNRAS.478.2399K/abstract)|
 | [`DDFacet`](https://github.com/saopicc/DDFacet) | 3GC | Imaging with direction-dependent corrections | [Tasse et al., 2018](https://ui.adsabs.harvard.edu/abs/2018A%26A...611A..87T/abstract) | 
-| [`killMS`](https://github.com/saopicc/killMS) | 3GC | DD self-calibration| - |
-| [`owlcat`](https://github.com/ska-sa/owlcat/) | 2GC, 3GC |FITS file manipulation | - |
+| [`killMS`](https://github.com/saopicc/killMS) | 3GC | DD self-calibration| [Tasse, 2014](https://ui.adsabs.harvard.edu/abs/2014A%26A...566A.127T/abstract); [Smirnov & Tasse, 2014](https://ui.adsabs.harvard.edu/abs/2015MNRAS.449.2668S/abstract) |
+| [`owlcat`](https://github.com/ska-sa/owlcat/) | 2GC, 3GC | FITS file manipulation | - |
 | [`ragavi`](https://github.com/ratt-ru/ragavi/) | 1GC, 2GC | Plotting gain solutions | - |
-| [`shadeMS`](https://github.com/ratt-ru/shadeMS/) | 1GC | Plotting visibilities | - |
+| [`shadeMS`](https://github.com/ratt-ru/shadeMS/) | 1GC | Plotting visibilities | [Smirnov et al., 2022](https://ui.adsabs.harvard.edu/abs/2022ASPC..532..385S/abstract) |
 | [`Singularity`](https://github.com/hpcng/singularity) | 1GC, FLAG, 2GC, 3GC | Containerisation | [Kurtzer, Sochat & Bauer, 2017](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459) |
-| [`Stimela`](https://github.com/ratt-ru/Stimela/) | 1GC, FLAG, 2GC, 3GC | Containers | [Makhathini, 2018](http://hdl.handle.net/10962/57348) |
 | [`tricolour`](https://github.com/ska-sa/tricolour) | FLAG | Flagging | [Hugo et al., 2022](https://ui.adsabs.harvard.edu/abs/2022arXiv220609179H/abstract) |
 | [`wsclean`](https://gitlab.com/aroffringa/wsclean) | FLAG, 2GC, 3GC | Imaging, model prediction | [Offringa et al., 2014](https://ui.adsabs.harvard.edu/abs/2014MNRAS.444..606O/abstract)|
 

data/cubical/3GC_peel_2dirs.parset

@@ -0,0 +1,260 @@
+[data]
+_Help = Visibility data options
+ms = 
+column = DATA
+time-chunk = 72
+freq-chunk = 256
+rebin-time = 1
+rebin-freq = 1
+chunk-by = SCAN_NUMBER
+chunk-by-jump = 1
+single-chunk = 
+single-tile = -1
+normalize = 0
+
+[sel]
+_Help = Data selection options
+field = 0
+ddid = None
+taql = 
+chan = 
+diag = False
+
+[out]
+_Help = Options for output products
+dir = peeling
+name = peeling.cc-out/peeling
+overwrite = False
+backup = 1
+mode = sr
+apply-solver-flags = True
+column = CORRECTED_DATA
+derotate = None
+model-column = 
+weight-column = 
+reinit-column = False
+subtract-model = 0
+subtract-dirs = 1
+correct-dir = 0
+plots = 1
+casa-gaintables = True
+
+[model]
+_Help = Calibration model options
+list = MODEL_DATA+-DIR1_DATA+-DIR2_DATA:DIR1_DATA:DIR2_DATA
+ddes = auto
+beam-pattern = None
+beam-l-axis = None
+beam-m-axis = None
+feed-rotate = auto
+pa-rotate = False
+
+[montblanc]
+_Help = Montblanc simulation options
+device-type = CPU
+dtype = double
+mem-budget = 1024
+verbosity = WARNING
+threads = 0
+pa-rotate = None
+
+[weight]
+_Help = Weighting options
+column = WEIGHT_SPECTRUM
+fill-offdiag = False
+legacy-v1-2 = False
+
+[flags]
+_Help = General flagging options
+apply = FLAG
+auto-init = None
+save = 0
+save-legacy = 1
+reinit-bitflags = False
+warn-thr = 0.3
+see-no-evil = 0
+
+[degridding]
+_Help = Options for the degridder. Only in use when predicting from DicoModels using DDFacet
+OverS = 11
+Support = 7
+Nw = 100
+wmax = 0.0
+Padding = 1.7
+NDegridBand = 16
+MaxFacetSize = 0.25
+MinNFacetPerAxis = 1
+NProcess = 8
+BeamModel = None
+NBand = 0
+FITSFile = beam_$(corr)_$(reim).fits
+FITSFeed = None
+FITSFeedSwap = False
+DtBeamMin = 5.0
+FITSParAngleIncDeg = 5.0
+FITSLAxis = -X
+FITSMAxis = Y
+FITSVerbosity = 0
+FeedAngle = 0.0
+FlipVisibilityHands = 0
+
+[postmortem]
+_Help = Options for "postmortem" flagging based on solution statistics
+enable = False
+tf-chisq-median = 1.2
+tf-np-median = 0.5
+time-density = 0.5
+chan-density = 0.5
+ddid-density = 0.5
+
+[madmax]
+_Help = Options for the "Mad Max" flagger
+enable = 0
+residuals = 0
+estimate = corr
+diag = True
+offdiag = True
+threshold = 10
+global-threshold = 12
+plot = 1
+plot-frac-above = 0.01
+plot-bl = 
+flag-ant = 0
+flag-ant-thr = 5
+
+[sol]
+_Help = Solution options which apply at the solver level
+jones = G,DE
+precision = 32
+delta-g = 1e-06
+delta-chi = 1e-06
+chi-int = 5
+last-rites = True
+stall-quorum = 0.99
+term-iters = 50,50
+flag-divergence = 0
+min-bl = 150.0
+max-bl = 0
+subset = 
+
+[bbc]
+_Help = Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
+load-from = 
+compute-2x2 = False
+apply-2x2 = False
+save-to = {out[name]}-BBC-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
+per-chan = True
+plot = True
+
+[dist]
+_Help = Parallelization and distribution options
+ncpu = 6
+nworker = 0
+nthread = 0
+max-chunks = 2
+min-chunks = 0
+pin = 0
+pin-io = False
+pin-main = io
+safe = True
+
+[log]
+_Help = Options related to logging
+memory = True
+stats = chi2:.3f
+stats-warn = chi2:10
+boring = True
+append = False
+verbose = 0
+file-verbose = None
+
+[debug]
+_Help = Debugging options for the discerning masochist
+pdb = False
+panic-amplitude = 0.0
+stop-before-solver = False
+escalate-warnings = 0
+
+[misc]
+_Help = Miscellaneous options
+random-seed = None
+parset-version = 0.1
+
+[g]
+_Help = Options for G-Jones term
+label = G
+solvable = 1
+type = complex-2x2
+delay-estimate-pad-factor = 8
+load-from = 
+xfer-from = 
+save-to = {out[name]}-{JONES}-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
+dd-term = 0
+fix-dirs = 
+update-type = full
+estimate-pzd = False
+time-int = 18
+freq-int = 256
+max-prior-error = 0.3
+max-post-error = 0.3
+low-snr-warn = 75
+high-gain-var-warn = 30
+clip-low = 0.1
+clip-high = 10
+clip-after = 5
+max-iter = 50
+pin-slope-iters = 0
+epsilon = 1e-06
+delta-chi = 1e-06
+conv-quorum = 0.99
+ref-ant = None
+prop-flags = default
+diag-only = 0
+offdiag-only = False
+robust-cov = compute
+robust-scale = 1
+robust-npol = 2
+robust-int = 1
+robust-save-weights = 0
+estimate-delays = False
+_Templated = True
+
+[de]
+_Help = Options for G-Jones term
+label = dE
+solvable = 1
+type = complex-2x2
+delay-estimate-pad-factor = 8
+load-from = 
+xfer-from = 
+save-to = {out[name]}-{JONES}-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
+dd-term = 1
+fix-dirs = 0
+update-type = full
+estimate-pzd = False
+time-int = 72
+freq-int = 64
+max-prior-error = 5.0
+max-post-error = 5.0
+low-snr-warn = 75
+high-gain-var-warn = 30
+clip-low = 0.0
+clip-high = 1000000
+clip-after = 5
+max-iter = 50
+pin-slope-iters = 0
+epsilon = 1e-06
+delta-chi = 1e-06
+conv-quorum = 0.99
+ref-ant = None
+prop-flags = default
+diag-only = 0
+offdiag-only = False
+robust-cov = compute
+robust-scale = 1
+robust-npol = 2
+robust-int = 1
+robust-save-weights = 0
+estimate-delays = False
+_Templated = True
+