Calibration

[perrymok]

This PR fixes #6.

TL;DR

In order to use auto calibration, do the following:

1. Call EDSespm::plot_table to see how shifted the dataset is.
2. Call EDSespm::auto_calibrate with window you obtained in step 1.
   This does the following:
   • Afflinely correct the energy axis of the dataset
   • Correct the line table with higher degree polynomial
3. Call EDSespm::build_G with use_calibration=True if you want to use the higher degree correction.

WRT

The EDSespm::auto_calibrate method does the following:

1. For each element, individually fit the lines with cross section $\geq$ filter_cs $\times$ [cross section of the largest line in that element].
2. ~~Fit a degree degree polynomial $p$ weighted by the amplitude of each peak. ~~
3. Apply this polynomial $p$ to each line in the theoretical table, and store the calibrated table at EDXS::calibrated_db_dict. $p$ will also be stored at EDXS::energy_calibration_poly.

Calling EDSespm::build_G with use_calibration=True will automatically trigger auto_calibration.

Calling EDSespm::auto_calibrate will

1. Fit each individual lines using EDSespm::fit_single_peak
2. Fit an affine (degree 1 polynomial) correction for the energy axis using these fitted lines, and update self.axes_manager[-1].{scale, offset}
3. Fit a degree degree polynomial correction for the table in this new coordinate system, and store it in self.model

Running EDSespm::auto_calibrate followed by EDSespm::build_G will build G using the correction done in step 2 but not the correction in step 3. In order to also benefit from the correction done in step 3, use_calibration=True has to be passed to build_G.

In a nutshell, step 2 is affine correction of the datacube assuming the theoretical lines are correct, and step 3 is the high degree correction of the theoretical lines themselves.

EDSespm::fit_table and EDSespm::fit_single_peak have no side effect (it does not mutate self or any of its properties). EDSespm::auto_calibrate, on the other hand, does have side effect (it sets self.model.{calibrated_db_dict, energy_calibration_poly})

For a quick tour, please see notebooks/calibration.ipynb.

[perrymok]

BTW, the auto calibration only supports lines_db-type table and not compact_db-type. And in my opinion support for the latter should be dropped entirely since only default_xrays.json uses that format.

[PauTorru]

The auto_calibration does no write the new energy_scale, energy offset etc anywhere, resulting in the EDXS model being initialized with not "auto_calibrated" energy axis. Then when calling build G, checks are performed against this stale energy axis causing problems: if you crop from 0.7 keV a spectrum image that contains oxygen, it gives an error.

[PauTorru]

Also out of range lines always give an error when using autocalibrate

[perrymok]

The auto_calibration does no write the new energy_scale, energy offset etc anywhere, resulting in the EDXS model being initialized with not "auto_calibrated" energy axis. Then when calling build G, checks are performed against this stale energy axis causing problems: if you crop from 0.7 keV a spectrum image that contains oxygen, it gives an error.

I have solved the problem in a dual form. It is a matter of frame of reference:

• I calibrate the model to the energy axis of the dataset
• You want the dataset to be calibrated to the energy axis of the model

I can convert the solution to your form easily.

Also out of range lines always give an error when using autocalibrate

That will be solved automatically after the convertion.

[adriente]

I will try to check the PRs soon but I'll be on vacations starting from the 31/07 and I'll be back on the 24/08. I am sorry I couldn't work on the PRs recently.

If you need to implement the PRs soon, I guess you can go ahead and I'll review the changes afterwards. It is not ideal but I don't want to slow down everything.

[perrymok]

The algorithm now behaves as follows:

Calling EDSespm::auto_calibrate will

1. Fit each individual lines using EDSespm::fit_single_peak
2. Fit an affine (degree 1 polynomial) correction for the energy axis using these fitted lines, and update self.axes_manager[-1].{scale, offset}
3. Fit a degree degree polynomial correction for the table in this new coordinate system, and store it in self.model

Running EDSespm::auto_calibrate followed by EDSespm::build_G will build G using the correction done in step 2 but not the correction in step 3. In order to also benefit from the correction done in step 3, use_calibration=True has to be passed to build_G.

In a nutshell, step 2 is affine correction of the datacube assuming the theoretical lines are correct, and step 3 is the correction of the theoretical lines themselves.

[perrymok]

I will try to check the PRs soon but I'll be on vacations starting from the 31/07 and I'll be back on the 24/08. I am sorry I couldn't work on the PRs recently.

If you need to implement the PRs soon, I guess you can go ahead and I'll review the changes afterwards. It is not ideal but I don't want to slow down everything.

@adriente no worries. i say good-practices can be merged directly, and optimisation can also be merged, except you need to pay attention to the commit i mentioned in the PR description. @PauTorru will review this PR.

[PauTorru]

@perrymok The auto calibration is doing a very poor job, it is impossible that the initial spectrum is this off. see image attached:

[perrymok]

you used a window that is too big. i will implement the plotting of table in espm to visualise the shift to the user so that they can choose the correct window.

[perrymok]

i have added a TL;DR in notebooks/calibration.ipynb:

TL;DR

In order to use auto calibration, do the following:

1. Call EDSespm::plot_table to see how shifted the dataset is.
2. Call EDSespm::auto_calibrate with window you obtained in step 1.
   This does the following:
   • Afflinely correct the energy axis of the dataset
   • Correct the line table with higher degree polynomial
3. Call EDSespm::build_G with use_calibration=True if you want to use the higher degree correction.