fab.settings

  1'''
  2This module provides the configuration values that are used throughout
  3the rest of the `fab` package.
  4
  5A default configuration is loaded at startup, but you can provide an 
  6extra configuration file where you can override default values and configure
  7new DataSources and Instruments. 
  8
  9The configuration is given as a `toml` file, that can be specified by 
 10calling:
 11
 12```python
 13from fab.settings import update_config
 14update_config("example_config.toml")
 15```
 16You can call this function repeatedly to load configuration from multiple
 17files. (Eg, a common `beamtime` file plus a personal one). In case of the
 18same property being defined in multiple files, the last definition will
 19take precence.
 20
 21If your config file is in the current working directory, or in one of its
 22parents, you can also use the `fab.magic` module to automatically load it:
 23
 24```python
 25from fab.magic import config
 26```
 27
 28Please avoid mixeing the two methods, as it might lead to unexpected results.
 29
 30## Basic settings:
 31
 32You can specifiy a beamtime number to select which files you want to load, 
 33by setting the `beamtime` parameter:
 34
 35```toml
 36beamtime = 12345678
 37```
 38
 39If you are running from within a gpfs beamtime folder, using the 
 40`from fab.magic import beamtime` statement will automatically detect 
 41the beamtime number and set this parameter for you. Please avoid mixing
 42the two methods.
 43If you want more flexible control over which files to load, or if you are not
 44running on the maxwell cluster, you can specify a glob pattern for where fab 
 45should look for hdf5 files using the `hdf_path` argument. 
 46Be aware that only one of `beamtime` and `hdf_path` can be specified at a time.
 47
 48```toml
 49hdf_path = "/asap3/path/to/hdf/*.h5"
 50```
 51
 52The fab libray will produce a cache file where metadata about the HDF files
 53is stored. If you did not specify a beamtime number, by default a file 
 54`file_index.h5` will be creaded in your working directory. You can specify 
 55a path for this file by setting:
 56
 57```toml
 58idx_path = "path/to/index.h5"
 59```
 60
 61If your datasources are configured to preload the data (as opposed to use
 62lazy dask arrays) an extra file with the preloaded data is created. You can 
 63customize its location with:
 64
 65```toml
 66preload_path = "path/to/preload.h5"
 67```
 68
 69You can control the logging level with:
 70
 71```toml
 72logging_level = 'INFO'
 73```
 74
 75set this to one of `'DEBUG'`, `'INFO'`, `'WARNING'`, `'ERROR'`, `'CRITICAL'` to configure
 76how many messages you want to see printed on screen.
 77
 78### Note on telemetry
 79
 80The fab package automatically logs anonymized usage data to a central location. This is
 81used to improve the package and to understand how it is being used. You can disable this
 82behaviour by setting:
 83
 84```toml
 85telemetry = false
 86```
 87
 88Contact the package maintainers if you have any questions or concerns about this.
 89
 90## Creating custom DataSources and Instruments
 91
 92You can define a source by adding a table to the configuration file. The attributes of the
 93tables should be the argument you want to pass to the constructur of the source. 
 94
 95```toml
 96[sources.delay_set]
 97hdf_key = "/zraw/FLASH.SYNC/LASER.LOCK.EXP/F2.PPL.OSC/FMC0.MD22.0.POSITION_SET.WR/dGroup"
 98fillna_method = 'ffill'
 99```
100
101You can then istantiate the source as:
102```python
103from fab.datasources import DataSource
104source = DataSource.from_name('delay_set')
105```
106
107You can specify entire Instruments this way. In the example below we create an Instrument 
108named `ursa` composed of various differenct sources. The sources definition follows the 
109same pattern as before, except for the extra attribute '__type__', that is used to 
110specify the class of the source.
111
112```toml
113[instruments.ursa.delay_set]
114__type__ = 'fab.datasources.HDFSource'
115hdf_key = "/zraw/FLASH.SYNC/LASER.LOCK.EXP/F2.PPL.OSC/FMC0.MD22.0.POSITION_SET.WR/dGroup"
116fillna_method = 'ffill'
117
118[instruments.ursa.retarder]
119__type__ = 'fab.datasources.HDFSource'
120hdf_key = '/FL2/Experiment/URSA-PQ/TOF/HV retarder'
121fillna_method = 'ffill'
122
123[instruments.ursa.eTof]
124__type__ = 'fab.datasources.SlicedADC'
125hdf_key = '/FL2/Experiment/MTCA-EXP1/ADQ412 GHz ADC/CH00/TD'
126offset = 2246
127window = 3000
128period = 9969.23
129t0 = 306
130dt = 0.0005
131baseline = 200
132
133[instruments.ursa.GMD]
134__type__ = 'fab.datasources.GMD'
135hdf_key = "/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy hall"
136
137[instruments.ursa.BAM]
138__type__ = 'fab.datasources.BAM'
139hdf_key = "/zraw/FLASH.SDIAG/BAM.DAQ/FL2.SEED5.ARRIVAL_TIME.ABSOLUTE.SA2.COMP/dGroup"
140```
141
142You can then use this instrument in your code by simply importing it:
143
144```python
145from fab.magic import config, ursa
146data = ursa.load()
147```
148
149## Preprocessors
150
151You can configure some basic preprocessing of the data to happen automatically 
152after the data is loaded, by specifying the `__preprocess__` attribute in a DataSource.
153For example, to directly integrate over dimension `diode_trace` form the loaded 
154data, you can configure the `fab.preprocessing.integrate' preprocessor as follows:
155
156``` toml
157[instruments.ursa.uv_diode]
158__type__ = 'fab.datasources.SlicedADC'
159hdf_key = '/zraw/FLASH.LASER/FLASH2CPUULGAN1.ADCSCOPE/CH26.TD/dGroup'
160offset = 2268
161window = 6
162period = 45
163baseline = 5
164dim_names = ['shot_id', 'diode_trace']
165__preprocess__ = [ {__name__='fab.preprocessing.integrate', dims='diode_trace'} ]
166```
167
168You can specify a list of preprocessors, that will be executed *in order* on the loaded
169data before the DataArray is returned. Preprocessors can be applied to DataSources or
170to entire instruments. You can create your own custom preprocessors to expand this 
171functionality as needed. See the documentation in the `fab.preprocessing` section for more. 
172
173## Dask and Maxwell cluster configuration
174
175You can use the toml config file to specify how dask should start jobs on the maxwell
176cluster. Please refer to the maxwell module documentation for a detailed overview of 
177the options.
178'''
179
180from .setup import cfg, dask_client, dask_cluster, cfg_context
181from .setup import update_context, update_config, maxwell_start_if_not_already, get_config_object, fab_setup