POWGEN | BL-11A | SNS#
The autoreduction pipeline on POWGEN instrument is working robustly to automatically produce the Bragg diffraction data and the total scattering data once the raw measurement data is becoming available on the hard drive. To guarantee a successful running of the autoreduction pipeline, some necessary configurations need to be in place, including those calibration and characterization runs and a series of control parameters for the data reduction routines. Concerning the principles of the data reduction, one could refer to the page here in current book for more information. The calibration and characterization information is stored in the POWGEN shared location on the analysis cluster,
/SNS/PG3/shared/CALIBRATION
and each cycle has a series of calibration and characterization files corresponding to different sample environments or instrument setups. The configuration file for controlling the autoreduction routine is stored in json format and is stored in POWGEN shared location at,
/SNS/PG3/shared/autoreduce
The central configuration file auto_config.json
located there contains high level controls over the autoreduction routine and also it contains the pointer to sample environment or setup specific configuration files. Those specific configuration files are stored here,
/SNS/PG3/shared/autoreduce/auto_config_files
Based on the information provided in the central configuration file auto_config.json
, the autoreduction routine knows how to find those specific configuration files which contains the parameters to be used by the backend data reduction engine.
Usually, all the files mentioned above need to be populated or modified by first logging into the analysis cluster, followed by editing those relevant files one-by-one using plain text editor. First, this is a tedious process. Second, it is very easy to make mistakes, which could then kill the whole autoreduction pipeline without anyone having a quick idea about what is happening. To make the whole process more transparent, easy to configure and error-proofing, we implemented a web-based interface in the ADDIE web interface. The web interface intends to replace the tedious way of configuring the POWGEN autoreduction pipeline. Several key things to mention,
To get access to the web interface concerning the POWGEN autoreduction configuration, one needs to hold a valid UCAMS/XCAMS account. Further, for the specific interfaces that will write files to the central POWGEN shared locations, only POWGEN team members can have the access.
The configuration web interface is only for the configuration purpose but is
NOT
intended to interfere with the running of the autoreduction routine. To do the interference with the autoreduction running, one needs to go to the monitor web service here.
The autoreduction configuration is implemented in such a way that a specific IPTS has its dedicated version of the configuration. In practice, if the IPTS specific version of the configuration cannot be found, the central configuration (those located in the POWGEN shared location as mentioned above) will be copied over to the IPTS specific location. Otherwise, depending on the configuration in the central config file, either the central version or the IPTS specific version will win over. See the section below for more details.
A demonstration video has been recorded to go through the how-tos of using the web interface for characterization and autoreduction configuration, as can be found below,
from IPython.display import YouTubeVideo
YouTubeVideo('yNTG0qD4HMs', width=720, height=500)
AutoReduction Configuration#
Here follows is a template autoreduction configuration file in json format,
{
"Cycle": "2025-1_11A_CAL",
"SampleEnvironment": "PAC",
"OverwriteUserJsonFile": "Y",
"ConfigFile": {
"PAC": "PAC_default.json",
"HTJANIS": "HTJANIS_default.json",
"LTJANIS": "LTJANIS_default.json",
"OC": "OC_default.json",
"MICAS": "MICAS_default.json",
"MICAS-33420": "MICAS_33420.json",
"MICAS-32378": "MICAS_32378.json",
"AGES": "AGES_default.json",
"JANIS_GAS_noscale": "JANIS_GAS_noscale.json",
"JANIS_GAS": "JANIS_GAS.json",
"MAG": "MAG_default.json",
"MAG-He3": "MAG_He3_default.json"
},
"Notes": "",
"SNSPowderRedAbsCacheDir": "/SNS/PG3/shared/autoreduce/snspdr_abs_ms",
"Comment-1": "Entries below are for MantidTotalScattering",
"ClientID": ".....",
"Secret": ".....",
"TokenURL": "https://snsapp1.sns.ornl.gov/xprod_ro/proposal_production/oauth/token",
"IPTSURL": "https://snsapp1.sns.ornl.gov/xprod_ro/proposal_production/sec_smpl/getSampleDetailsByIPTS/",
"ONCatID": ".....",
"ONCatSecret": ".....",
"CacheDir": "/SNS/PG3/shared/autoreduce/mts_abs_ms",
"EnvironmentName": "InAir",
"InstrumentGeometryRun": "PG3_56131",
"GroupingAllFile": "/SNS/PG3/shared/autoreduce/configs/pg3_group_all.xml",
"SampleElementSize": 1.0,
"ContainerElementSize": 1.0,
"MergeRunsOptions": "2",
"ReGenerateGrouping": 0,
"QMaxByBank": "40.0",
"TMinBragg": "1.0",
"TMaxBragg": "266.0",
"TBinBragg": "-0.0008",
"ResampleX": -8000,
"QParamsProcessing": "0.5,0.0005,40.",
"OverrideUserConfig": true,
"MTSFourierTransform": {
"TMIN": 3000.0,
"TMAX": 270000.0,
"QMin": 0.5,
"QMax": 35.0,
"RMax": 50.0,
"RStep": 0.01,
"QOffset": 0.0,
"RCutoff": 1.0,
"Lorch": false
},
"ContainerPeaks": "None",
"BkgScale": 1.0,
"CyclePackingFrac": true,
"ResonanceFilter": {
"Units": "Wavelength",
"LowerLimits": [0.3],
"UpperLimits": [0.5]
},
"ReCaching": false,
"FinalDIFC": 22585.8,
"LastModifiedBy": "y8z",
"LastModifiedTime": "07-16-2024 17:38:01 EST"
}
Cycle
Mandatory
Specification of the run cycle. The value coming with the key should be a valid directory name under
/SNS/PG3/shared/CALIBRATION/
SampleEnvironment
Mandatory
Specification of the sample environment name. The value should be one of the keys in the following
ConfigFile
value. TheCycle
,SampleEnvironment
andConfigFile
keys together pin down the configuration file to use specific to the sample environment in a certain cycle. In the template presented above, theSampleEnvironment
value isPAC
and therefore one should be looking for the value inConfigFile
corresponding to thePAC
key (namely,PAC_default.json
in the example). So, the environment specific configuration to use in this case is located at,/SNS/PG3/shared/CALIBRATION/2025-1_11A_CAL/PAC_default.json
OverwriteUserJsonFile
Mandatory
Whether or not to overwrite the potentially existing IPTS specific configuration file. The mechanism of central and IPTS specific configuration working together is presented in the section above. With the current key, the instrument team has the power to control whether or not to let the central configuration win over.
Y
is forYes
andN
forNo
.ConfigFile
Mandatory
This specifies a dictionary pointing each sample environment to its dedicated configuration file name. As mentioned above, the value here will be used together with the
Cycle
andSampleEnvironment
keys to pin down the configuration file to load. The configuration to load applies to only specific sample enviroments and this is why we are separating out the sample environment specific configurations.Notes
Optional
This is expected to be a single string without any white spaces (tabs or spaces). If a non-empty string is given, it will be used as part of the name for the calibration file to be used. See the sample environment specific configurations documentation below for more details.
SNSPowderRedAbsCacheDir
Optional
With the
abs_pre_calc
routine, it is now possible to pre-calculate the absorption given a certain IPTS and the pre-calculated absorption will be saved into the specified location here. If the key is provided here, the autoreduction will try to find the pre-calculated absorption saved in the specified location and if existing, autoreduction will load it in for boosting the calculation. If the key is not provided, autoreduction will use an IPTS specific directory for absorption caching while running the reduction.Comment-1
Optional
This is just an annotation entry for the verbose purpose without any real functionality attached. It is for indicating that the configurations above the comment entry are for
SNSPowderReduction
and below the comemnt entry, the configurations are forMantidTotalScattering
.ClientID
Mandatory
The client ID for talking to the
ITEMS
database.Secret
Mandatory
The secret for talking to the
ITEMS
database.TokenURL
Mandatory
The token URL for talking to the
ITEMS
database.IPTSURL
Mandatory
The IPTS URL for talking to the
ITEMS
database.ONCatID
Mandatory
The ONCAT ID for talking to the
ONCAT
database.ONCatSecret
Mandatory
The ONCAT secret for talking to the
ONCAT
database.CacheDir
Mandatory
Specify the cache directory for the
MantidTotalScattering
reduction.EnvironmentName
Optional
Name of the sample environment, Usually it is just
InAir
and we barely need to touch it.GroupingAllFile
Optional
The XML file specifying the grouping of all detectors. As long as there is no detector upgrade on the instrument, there is no need to change this.
SampleElementSize
Optional
Cuboid size in
mm
for the absorption calculation for the sample.ContainerElementSize
Optional
Cuboid size in
mm
for the absorption calculation for the container.MergeRunsOptions
Optional
The criterion for merging different run numbers. Two options are available,
1
for using the run title with dynamic temperature information and2
for using the run title without the dynamic temperature. The default option is2
as usually we are not interested in the temperature fluctuation around the set point.ReGenerateGrouping
Optional
This is the key for detector grouping according to the similarity in absorption spectra as mentioned earlier. If no upgrade to the detectors, one can always stay with
0
as the input here, indicating that we don’t want to re-generate the group. For POWGEN, the grouping files and information are stored in the following directory,/SNS/PG3/shared/autoreduce/configs
.QMaxByBank
Mandatory
The
QMax
value to be used for each of the output bank of data. If one value is given, it will be used for banks. Multiple values can be provided, separated by comma. The value should be given in a string, as presented in the example above.TMinBragg
Mandatory
The minimal TOF value for the output Bragg diffraction data for each bank.
TMaxBragg
Mandatory
The maximal TOF value for the output Bragg diffraction data for each bank.
QParamsProcessing
Mandatory
This is the \(Q\)-space binning parameters used internally for MTS data processing. Since we are going to throw away the neutron events after the data processing, we have to use a fine \(Q\)-space binning to make sure that we can always go with a coarser bin for the binning down the road.
OverrideUserConfig
Optional
Any time the autoreduction is running, it will try to load or save the central configuration to the experiment-specific location so that later on, on the user side, they can change the parameter locally without influencing others. The default value for this flag is
false
, indicating that as long as the previously saved user configuration file can be found, the autoreduction routine will pick it up. If the flag is set totrue
, the central configuration will take the priority and after the reduction, the central configuration file will be copied into the experiment-specific location, with whatever existing configuration files there backed up to avoid losing history.CyclePackingFrac
Optional
For the autoreduction purpose, we could check the log file from
MTS
running, grab the suggested packing fraction and start a new reduction with the tweaked packing fraction until the convergence of the packing fraction. This flag controls whether we want to perform such a loop. The default option istrue
, but in some cases, due to the poor data quality, the loop may get stuck, in which case we want to turn off the cycling.MTSFourierTransform
Optional
A dictionary should be given as the value. Here are the entries in the dictionary,
TMIN
Mandatory
This value will go into the
TMin
key ofAlignAndFocusArgs
for the sample, specifying the minimum TOF for the align-and-focus operation in microsecond.TMAX
Mandatory
This value will go into the
TMax
key ofAlignAndFocusArgs
for the sample, specifying the maximum TOF for the align-and-focus operation in microsecond.QMin
Mandatory
This key controls the minimal \(Q\) used for Fourier transform, to obtain the PDF.
QMax
Mandatory
This key controls the maximal \(Q\) used for Fourier transform, to obtain the PDF.
RMax
Mandatory
This key controls the maximal \(r\) of the PDF data.
RStep
Mandatory
This key controls the interval of the PDF data.
QOffset
Mandatory
The offset in \(Q\)-space that we want to apply to the reduced total scattering data before the Fourier transform.
RCutoff
Mandatory
The cutoff in \(r\)-space regarding the Fourier filter, i.e., all signals below the cutoff will be Fourier filtered.
Lorch
Mandatory
Whether to apply the lorch smoothing algorithm for the PDF data.
FinalDIFC
Mandatory
After the total scattering data processing, the data would be in \(Q\)-space. To obtain the Bragg diffraction data in TOF-space which can be imported into Rietveld programs for data analysis, we have to convert the \(Q\)-space data with some arbitrary values of DIFC for each bank of data. Here we can give the list of the DIFC values to be used for such a purpose and the aim is to keep consistent whatever conventional values that other data reduction programs were using.
ContainerPeaks
Optional
With this flag, one can specify a list of peaks (in
d
-space) for the container in case there are Bragg peaks in the container signal that needs to be stripped off. The list should be given in a string and different peak positions should be separated withcomma
.BkgScale
Optional
Specify a scale factor to be multiplied onto the container background in case of over or under subtraction of the container signal.
ResonanceFilter
Optional
A dictionary is expected to be provided with the key. Here are the expected entries of the dictionary,
Units
Mandatory
Specify the space in which the resonance filter parameters will be applied. Acceptable values are those in Mantid unit factor – see here.
LowerLimits
Mandatory
Specify a list of lower limits of the window within which the neutron counts are to be excluded.
UpperLimits
Mandatory
Specify a list of upper limits of the window within which the neutron counts are to be excluded. The length of the
LowerLimits
andUpperLimits
lists should be equal to each other.