POWGEN | BL-11A | SNS

POWGEN | BL-11A | SNS#

The autoreduction pipeline on POWGEN instrument is working robustly to automatically produce the Bragg diffraction data and the total scattering data once the raw measurement data is becoming available on the hard drive. To guarantee a successful running of the autoreduction pipeline, some necessary configurations need to be in place, including those calibration and characterization runs and a series of control parameters for the data reduction routines. Concerning the principles of the data reduction, one could refer to the page here in current book for more information. The calibration and characterization information is stored in the POWGEN shared location on the analysis cluster,

/SNS/PG3/shared/CALIBRATION

and each cycle has a series of calibration and characterization files corresponding to different sample environments or instrument setups. The configuration file for controlling the autoreduction routine is stored in json format and is stored in POWGEN shared location at,

/SNS/PG3/shared/autoreduce

The central configuration file auto_config.json located there contains high level controls over the autoreduction routine and also it contains the pointer to sample environment or setup specific configuration files. Those specific configuration files are stored here,

/SNS/PG3/shared/autoreduce/auto_config_files

Based on the information provided in the central configuration file auto_config.json, the autoreduction routine knows how to find those specific configuration files which contains the parameters to be used by the backend data reduction engine.

Usually, all the files mentioned above need to be populated or modified by first logging into the analysis cluster, followed by editing those relevant files one-by-one using plain text editor. First, this is a tedious process. Second, it is very easy to make mistakes, which could then kill the whole autoreduction pipeline without anyone having a quick idea about what is happening. To make the whole process more transparent, easy to configure and error-proofing, we implemented a web-based interface in the ADDIE web interface. The web interface intends to replace the tedious way of configuring the POWGEN autoreduction pipeline. Several key things to mention,

To get access to the web interface concerning the POWGEN autoreduction configuration, one needs to hold a valid UCAMS/XCAMS account. Further, for the specific interfaces that will write files to the central POWGEN shared locations, only POWGEN team members can have the access.
The configuration web interface is only for the configuration purpose but is NOT intended to interfere with the running of the autoreduction routine. To do the interference with the autoreduction running, one needs to go to the monitor web service here.

The autoreduction configuration is implemented in such a way that a specific IPTS has its dedicated version of the configuration. In practice, if the IPTS specific version of the configuration cannot be found, the central configuration (those located in the POWGEN shared location as mentioned above) will be copied over to the IPTS specific location. Otherwise, depending on the configuration in the central config file, either the central version or the IPTS specific version will win over. See the section below for more details.

A demonstration video has been recorded to go through the how-tos of using the web interface for characterization and autoreduction configuration, as can be found below,

from IPython.display import YouTubeVideo
YouTubeVideo('yNTG0qD4HMs', width=720, height=500)

AutoReduction Configuration#

Here follows is a template autoreduction configuration file in json format,

{
    "Cycle": "2025-1_11A_CAL",
    "SampleEnvironment": "PAC",
    "OverwriteUserJsonFile": "Y",
    "ConfigFile": {
        "PAC": "PAC_default.json",
        "HTJANIS": "HTJANIS_default.json",
        "LTJANIS": "LTJANIS_default.json",
        "OC": "OC_default.json",
        "MICAS": "MICAS_default.json",
        "MICAS-33420": "MICAS_33420.json",
        "MICAS-32378": "MICAS_32378.json",
        "AGES": "AGES_default.json",
        "JANIS_GAS_noscale": "JANIS_GAS_noscale.json",
        "JANIS_GAS": "JANIS_GAS.json",
        "MAG": "MAG_default.json",
        "MAG-He3": "MAG_He3_default.json"
    },
    "Notes": "",
    "SNSPowderRedAbsCacheDir": "/SNS/PG3/shared/autoreduce/snspdr_abs_ms",
    "Comment-1": "Entries below are for MantidTotalScattering",
    "ClientID": ".....",
    "Secret": ".....",
    "TokenURL": "https://snsapp1.sns.ornl.gov/xprod_ro/proposal_production/oauth/token",
    "IPTSURL": "https://snsapp1.sns.ornl.gov/xprod_ro/proposal_production/sec_smpl/getSampleDetailsByIPTS/",
    "ONCatID": ".....",
    "ONCatSecret": ".....",
    "CacheDir": "/SNS/PG3/shared/autoreduce/mts_abs_ms",
    "EnvironmentName": "InAir",
    "InstrumentGeometryRun": "PG3_56131",
    "GroupingAllFile": "/SNS/PG3/shared/autoreduce/configs/pg3_group_all.xml",
    "SampleElementSize": 1.0,
    "ContainerElementSize": 1.0,
    "MergeRunsOptions": "2",
    "ReGenerateGrouping": 0,
    "QMaxByBank": "40.0",
    "TMinBragg": "1.0",
    "TMaxBragg": "266.0",
    "TBinBragg": "-0.0008",
    "ResampleX": -8000,
    "QParamsProcessing": "0.5,0.0005,40.",
    "OverrideUserConfig": true,
    "MTSFourierTransform": {
        "TMIN": 3000.0,
        "TMAX": 270000.0,
        "QMin": 0.5,
        "QMax": 35.0,
        "RMax": 50.0,
        "RStep": 0.01,
        "QOffset": 0.0,
        "RCutoff": 1.0,
        "Lorch": false
    },
    "ContainerPeaks": "None",
    "BkgScale": 1.0,
    "CyclePackingFrac": true,
    "ResonanceFilter": {
        "Units": "Wavelength",
        "LowerLimits": [0.3],
        "UpperLimits": [0.5]
    },
    "ReCaching": false,
    "FinalDIFC": 22585.8,
    "LastModifiedBy": "y8z",
    "LastModifiedTime": "07-16-2024 17:38:01 EST"
}

Cycle

Mandatory

Specification of the run cycle. The value coming with the key should be a valid directory name under /SNS/PG3/shared/CALIBRATION/
SampleEnvironment

Mandatory

Specification of the sample environment name. The value should be one of the keys in the following ConfigFile value. The Cycle, SampleEnvironment and ConfigFile keys together pin down the configuration file to use specific to the sample environment in a certain cycle. In the template presented above, the SampleEnvironment value is PAC and therefore one should be looking for the value in ConfigFile corresponding to the PAC key (namely, PAC_default.json in the example). So, the environment specific configuration to use in this case is located at,
```
/SNS/PG3/shared/CALIBRATION/2025-1_11A_CAL/PAC_default.json
```
OverwriteUserJsonFile

Mandatory

Whether or not to overwrite the potentially existing IPTS specific configuration file. The mechanism of central and IPTS specific configuration working together is presented in the section above. With the current key, the instrument team has the power to control whether or not to let the central configuration win over. Y is for Yes and N for No.
ConfigFile

Mandatory

This specifies a dictionary pointing each sample environment to its dedicated configuration file name. As mentioned above, the value here will be used together with the Cycle and SampleEnvironment keys to pin down the configuration file to load. The configuration to load applies to only specific sample enviroments and this is why we are separating out the sample environment specific configurations.
Notes

Optional

This is expected to be a single string without any white spaces (tabs or spaces). If a non-empty string is given, it will be used as part of the name for the calibration file to be used. See the sample environment specific configurations documentation below for more details.
SNSPowderRedAbsCacheDir

Optional

With the abs_pre_calc routine, it is now possible to pre-calculate the absorption given a certain IPTS and the pre-calculated absorption will be saved into the specified location here. If the key is provided here, the autoreduction will try to find the pre-calculated absorption saved in the specified location and if existing, autoreduction will load it in for boosting the calculation. If the key is not provided, autoreduction will use an IPTS specific directory for absorption caching while running the reduction.
Comment-1

Optional

This is just an annotation entry for the verbose purpose without any real functionality attached. It is for indicating that the configurations above the comment entry are for SNSPowderReduction and below the comemnt entry, the configurations are for MantidTotalScattering.
ClientID

Mandatory

The client ID for talking to the ITEMS database.
Secret

Mandatory

The secret for talking to the ITEMS database.
TokenURL

Mandatory

The token URL for talking to the ITEMS database.
IPTSURL

Mandatory

The IPTS URL for talking to the ITEMS database.
ONCatID

Mandatory

The ONCAT ID for talking to the ONCAT database.
ONCatSecret

Mandatory

The ONCAT secret for talking to the ONCAT database.
CacheDir

Mandatory

Specify the cache directory for the MantidTotalScattering reduction.
EnvironmentName

Optional

Name of the sample environment, Usually it is just InAir and we barely need to touch it.
GroupingAllFile

Optional

The XML file specifying the grouping of all detectors. As long as there is no detector upgrade on the instrument, there is no need to change this.
SampleElementSize

Optional

Cuboid size in mm for the absorption calculation for the sample.
ContainerElementSize

Optional

Cuboid size in mm for the absorption calculation for the container.
MergeRunsOptions

Optional

The criterion for merging different run numbers. Two options are available, 1 for using the run title with dynamic temperature information and 2 for using the run title without the dynamic temperature. The default option is 2 as usually we are not interested in the temperature fluctuation around the set point.
ReGenerateGrouping

Optional

This is the key for detector grouping according to the similarity in absorption spectra as mentioned earlier. If no upgrade to the detectors, one can always stay with 0 as the input here, indicating that we don’t want to re-generate the group. For POWGEN, the grouping files and information are stored in the following directory, /SNS/PG3/shared/autoreduce/configs.
QMaxByBank

Mandatory

The QMax value to be used for each of the output bank of data. If one value is given, it will be used for banks. Multiple values can be provided, separated by comma. The value should be given in a string, as presented in the example above.
TMinBragg

Mandatory

The minimal TOF value for the output Bragg diffraction data for each bank.
TMaxBragg

Mandatory

The maximal TOF value for the output Bragg diffraction data for each bank.
QParamsProcessing

Mandatory

This is the \(Q\)-space binning parameters used internally for MTS data processing. Since we are going to throw away the neutron events after the data processing, we have to use a fine \(Q\)-space binning to make sure that we can always go with a coarser bin for the binning down the road.
OverrideUserConfig

Optional

Any time the autoreduction is running, it will try to load or save the central configuration to the experiment-specific location so that later on, on the user side, they can change the parameter locally without influencing others. The default value for this flag is false, indicating that as long as the previously saved user configuration file can be found, the autoreduction routine will pick it up. If the flag is set to true, the central configuration will take the priority and after the reduction, the central configuration file will be copied into the experiment-specific location, with whatever existing configuration files there backed up to avoid losing history.
CyclePackingFrac

Optional

For the autoreduction purpose, we could check the log file from MTS running, grab the suggested packing fraction and start a new reduction with the tweaked packing fraction until the convergence of the packing fraction. This flag controls whether we want to perform such a loop. The default option is true, but in some cases, due to the poor data quality, the loop may get stuck, in which case we want to turn off the cycling.
MTSFourierTransform

Optional

A dictionary should be given as the value. Here are the entries in the dictionary,
- TMIN
  
  Mandatory
  
  This value will go into the TMin key of AlignAndFocusArgs for the sample, specifying the minimum TOF for the align-and-focus operation in microsecond.
- TMAX
  
  Mandatory
  
  This value will go into the TMax key of AlignAndFocusArgs for the sample, specifying the maximum TOF for the align-and-focus operation in microsecond.
- QMin
  
  Mandatory
  
  This key controls the minimal \(Q\) used for Fourier transform, to obtain the PDF.
- QMax
  
  Mandatory
  
  This key controls the maximal \(Q\) used for Fourier transform, to obtain the PDF.
- RMax
  
  Mandatory
  
  This key controls the maximal \(r\) of the PDF data.
- RStep
  
  Mandatory
  
  This key controls the interval of the PDF data.
- QOffset
  
  Mandatory
  
  The offset in \(Q\)-space that we want to apply to the reduced total scattering data before the Fourier transform.
- RCutoff
  
  Mandatory
  
  The cutoff in \(r\)-space regarding the Fourier filter, i.e., all signals below the cutoff will be Fourier filtered.
- Lorch
  
  Mandatory
  
  Whether to apply the lorch smoothing algorithm for the PDF data.
FinalDIFC

Mandatory

After the total scattering data processing, the data would be in \(Q\)-space. To obtain the Bragg diffraction data in TOF-space which can be imported into Rietveld programs for data analysis, we have to convert the \(Q\)-space data with some arbitrary values of DIFC for each bank of data. Here we can give the list of the DIFC values to be used for such a purpose and the aim is to keep consistent whatever conventional values that other data reduction programs were using.
ContainerPeaks

Optional

With this flag, one can specify a list of peaks (in d-space) for the container in case there are Bragg peaks in the container signal that needs to be stripped off. The list should be given in a string and different peak positions should be separated with comma.
BkgScale

Optional

Specify a scale factor to be multiplied onto the container background in case of over or under subtraction of the container signal.
ResonanceFilter

Optional

A dictionary is expected to be provided with the key. Here are the expected entries of the dictionary,
- Units
  
  Mandatory
  
  Specify the space in which the resonance filter parameters will be applied. Acceptable values are those in Mantid unit factor – see here.
- LowerLimits
  
  Mandatory
  
  Specify a list of lower limits of the window within which the neutron counts are to be excluded.
- UpperLimits
  
  Mandatory
  
  Specify a list of upper limits of the window within which the neutron counts are to be excluded. The length of the LowerLimits and UpperLimits lists should be equal to each other.

POWGEN | BL-11A | SNS

Contents

POWGEN | BL-11A | SNS#

AutoReduction Configuration#