Getting Started

pyXsurf (formerly pyXTel)

Python library for X-Ray Optics, Metrology Data Analysis and Telescopes Design.

This library starts from code collected during many years of work on surface metrology and X-ray optics and wants to offer to the community an extensible tool to perfom common operations on surface and profile data, together with a set of tools to perform typical operations on data, in particular related to X-ray optics.

Changes

2023/07/15 Starting implementing major changes towards common standards and best-practices. 2022/09/06 Created new branch pyXsurf_nbdev for experimenting with 2021/07/21 Upgraded installation mechanism. 2020/11/09 Changed repository name from pyXTel to pyXSurf.

Installation

Before proceding, it can be advisable to update your environment, and especially pip, to the most recent version.

At the moment the best way to install the most updated version, which contains many bug fixes and updated functionalities, is to clone the source code from github and install from there.

You can use the Code button at top of page, or git clone from command-line git clone https://github.com/vincenzooo/pyXSurf.git.

At this point you should move to the folder with the code and call from command line pip install . (equivalend to python setup.py install). This will perform a “regular” installation (meaning that the code is copied to site-packages folders, which holds all installed Python libraries, and made accessible to your system).

If instead you plan to modify the code, it is suggest to install the library in “developer” mode, use pip install -e .``or ``pip install -e '.[dev]' (experimental). Doing this, the library will be installed from current position (no local copy), any change to the code in this folder will be immediately available in the imported library (the second command, with the [dev]option, will install also additional files with tests and tools).

Uninstalling

Code can be uninstalled calling pip uninstall pyXsurf from a command prompt, or from source code folder calling directly the setup file python setup.py develop  -u (if you installed as developer, deinstallation is expected to work only with most recent pip versions, at least >=19.1.1, otherwise it should be enough to delete the folder).

How to use

At this point you can open Python and try import pySurf, if this works without errors, you can now import and use the different modules (see description below) which are part of the library, e.g., with:

from pySurf.data2D import data2D

In addition to the official documentation , you can find examples and data in a test subfolder of each module and at the end of module files. Functions are usually documented with docstrings (but these might be in the wrong place, please be patient and look well in the code). There is a lot more, but this is a big work-in-progress, and they are not easily accessible yet, please read below for details, or write an email or an issue.

Modules

A basic description of the different modules is (N.B.: some of these functions have been temporarily moved out of the codebase to simplify the reimplementation, in case these are needed, please send an email):

• dataIO Generic routines for accessing and manipulating data and files.

• notebooks Jupyter notebooks, not necessarily related to the libraries, include test and experiments on python.

• plotting Plotting functions for pySurf data.

• pyGeo3D Functions for geometry in space (lines and planes).

• pyProfile Equivalent of pySurf acting on Profiles.

• pySurf Functions and classes acting on 3D points or surfaces.

• thermal Functions for modelling of thermal forming of glass.

Status of the library and additional resources

The main part of the library is well defined and it works well. I am constantly adding functions when I find they are needed during my daily work.

I have tried the installation according to the above instructions on a few computers and it worked smoothly. You are very welcome to help signaling any problem or missing information, please see Contributing below.

There are many examples scattered around in different folders Demo, Tutorial, Test, etc. , some of which are not precisely documented, or even finished. I am still trying to give an organic structure to documentation (it might takes time, as I am completely inesperienced). Until then, the best thing is to dig for notebooks .ipynb or .py files in these folders, or look in the single .py files from code library, which often contain small test functions or examples.

There is not a real API documentation, as I am still trying to understand the standards, but there are a good amount of docstrings (even if they might be in the wrong place or format, sorry for that). Please use Python introspection (?, ??, autocompletion, etc.), and some patience, to find more. If you are a documentation expert I could really use some advice on that.

There are developer branches on github, which have extra functionalities, at the moment under development (they are quite messy, and some of the features might have been forgotten, so feel free to ask if you are looking for something in particular). These can be installed from code in the same way as the main branch and should work equally (just have more unfinished stuff). For git beginners, the only thing you need to do differently, is to switch branch, e.g. git checkout documentation (where documentation is the name of the branch) after cloning the repository and before running the setup. Please check developers notes for a list of active branches and their features and for more details. Also, if you plan to make changes to the code and want to keep the changes automatically in synch, remember to install the code as “developer” (as explained above). Otherwise you can still make changes to the installed code (in site-packages), but you will need to reimport after every change.

See developer notes Contributing for a detailed status of developement, how to access more recent features and last status of documentation (on developer brach), especially if you think you can help. Expecially installation and release mechanism, are in phase of improvement, as well as documentation.

Contributing

Please report bugs or feature requests, missing documentation, or open a issue on github https://github.com/vincenzooo/pyXsurf/issues.

Expecially appreciated is if you can provide templates, examples or hints on how to handle, documentation (Sphinx), packaging, continuous integration (Github).

Please check :ref:README_developers for the status of the development, or if are willing to help in any way.

Citation

License

This project is Copyright (c) Vincenzo Cotroneo and licensed under the terms of the BSD 3-Clause license. Parts of this package are based upon the Astropy package template which is licensed under the BSD 3-clause license. See the licenses folder for more information.

Author

Vincenzo Cotroneo vincenzo.cotroneo@inaf.it

For developers

Contributing

If you find bugs, errors, omissions or other things that need improvement, please create an issue or a pull request at https://github.com/vincenzooo/pyXsurf. Contributions are always welcome!

Here is summarised the current status and the best practices for development for code, documentation, packaging, and testing. This is a work in progress, the project is going through a major restyling with the intent of standardizing the workflow for updates of software and documentation, but many aspects are still being defined.

If you are interested in contributing, please check often these pages. In the meanwhile you are very welcome to signal any problem or missing information, especially if anything is not working as documented or if you can point to templates or examples.

Developers Installation

To start, you need to make sure that the necessary prerequisites_ are installed. Then, instead of pip-installing the latest release from PyPI_, you should get the newest development version (a.k.a. “master”) with Git, then move to the project folder and install as developer:

git clone https://github.com/vincenzooo/pyXsurf.git
cd pyXsurf
pip install -e .

… where -e stands for --editable. This maintains the import synchronized with the software (in other words, you don’t need to reinstall the package every time you make changes). This will work also if you checkout another branch (you might need to reimport the module in that case).

In addition, if you are using IPython or derived tools (e.g. Jupyter notebooks) it is suggested to add the following magic commands at the beginning of your work:

%load_ext autoreload %autoreload 2

This will automatically make sure that the code is reimported at any change.

Working on documentation

If you want to work on the documentation only, or you want to test it, you can put yourself in the docs directory and call:

• make html will normally compile the “official” documentation for the pyXsurf library at the current status of development.

• make html .\source_test_doc will compile also the test documentation, creating a front page which links to the “official” documentation, but also at a test documentation, where the dubious concepts are tested in details.

In both cases the compiled HTML documentation will be visible in docs\build\html\index.html.

You can refer to the official pages of Sphinx and nbspinx tools for reference and syntax guide:

If you modify any file, and you suspect it could be an improvement, please write to vincenzo.cotroneo@inaf.it, or use regular git functions (pull requests or issue tracker, you probably know this stuff better than I do).

Note

Themes are defined in the conf.py file of each folder, so there might be a difference in the aspect of the “official” documentation, according to which compilation command is run. TODO: make this point uniform, by making the configuration for tests to recall the configuration for the “official page”.

Active Branches

At the moment, the updated project is on the master branch, which is the only used. In the past there were attempts to establish development branches, which are now not maintained and are:

• documentation The most recent branching, it is mostly devoted at developing better sphinx documentation

• readers_dev Development to Reader routines for I/O of file formats

Roadmap

The roadmap is detailed in the following section, main area of work are:

• documentation

• testing

• installation

• continuous Integration

Status of installer

The library is installable with common python practices,

I have tried the installation on a few computers and it worked smoothly by setup.py. The folder test_install contains tests that verify a correct installation.

I can upload the package on test-PyPI, but not on the main PyPI, as I get an error, related to the too large size of the package.

Plan is to achieve a full installable distribution on PyPI, and possibly on conda, and be able to maintain it through Continuous Integration.

Status of code

Code is quite consistent, and usually commented, but there are still a few modules in early stage of development or which might be removed or integrated somewhere else. Some are included for backwards compatibility, future needs or just to stay on the safe side of breaking dependencies. There might be additional features which I lost along the way while experimenting with git and might decide to reintegrate if I recover them for other needs.

Status of docs

Docstrings are scattered around the code and sometimes even in the right place for self-documentation to work, but this is quite non-uniform, as it mixes different conventions I tried over the time, and it will remain like this until I find one tool that I can use to maintain the documentation.

I am not very skilled in using sphinx features, but I am learning. I started a new documentation branch for the development of documentation, which includes also tests on the different sphinx features. Basic api documentation is very rough, but seems to contain a good amount of documentation, so the obvious step is to obtain a minimally effective, decently clean API documentation, as a starting point for a refinement of the standard in documentation format, which is often not homogeneous.

Formatting is indeed at the moment very poor, I have tried experimenting with templates (e.g. astropy), ipynb, rst and integration with github, things that I am really not sure how to handle. See more details in the homepage of the documentation branch.

Also, some experiment with jupyter book, that looks a very appealing option on a long term, but implies a shift in paradygm.

Changes

2022/03/31 started documentation branch. Upgrades to sphinx docs and github readme and readme_developers. did more attempts in separate folder based on astropy and astropy templates. See VS workspace.

2022/03/22 Started restructuring repository. Renamed “source” folder to source (from pyXsurf). Add installation instructions for developers to this file and readme.txt.

2021/07/21 Upgraded installation mechanism.

2020/11/09 Changed repository name from pyXTel to pyXSurf.

2021/07/21 Really trying to improve installation and release. Mostly following https://medium.com/free-code-camp/from-a-python-project-to-an-open-source-package-an-a-to-z-guide-c34cb7139a22 and based on astropy template.

References

For users new to Python, the library was developed on anaconda distribution. Any other distribution should be equivalent, however I never had any problem with Anaconda and find it very easy to work with, so I can only recommend it. Same way, I used for developement notepad++/IPython/jupyter on Windows; Spyder; VScode and feel to recommend any of these. Of course any other environment or editor works same way.

if you are not sure of what you are doing (as I am most of the time), feel free to keep it locally and use only git or to use any expertise you have (and I don’t necessarily do have).

This is a good page explaining the basic usage of git https://kbroman.org/github_tutorial/pages/routine.html

See https://stackoverflow.com/questions/19048732/python-setup-py-develop-vs-install for comments on installation modes. TODO: summarize here.