Intermediate Research Software Development: Glossary

This lesson focuses on core, intermediate skills covering the whole software development life-cycle that will be of most use to anyone working collaboratively on code.

For code development in teams - you need more than just the right tools and languages. You need a strategy (best practices) for how you’ll use these tools as a team.

The lesson follows on from the novice Software Carpentry lesson, but this is not a prerequisite for attending as long as you have some basic Python, command line and Git skills and you have been using them for a while to write code to help with your work.

In order to develop (write, test, debug, backup) code efficiently, you need to use a number of different tools.

When there is a choice of tools for a task you will have to decide which tool is right for you, which may be a matter of personal preference or what the team or community you belong to is using.

Programming interfaces define how individual modules within a software application interact among themselves or how the application itself interacts with its users.

MVC is a software design architecture which divides the application into three interconnected modules: Model (data), View (user interface), and Controller (input/output and data manipulation).

The software project we use throughout this course is an example of an MVC application that manipulates patients’ inflammation data and performs basic statistical analysis using Python.

Virtual environments keep Python versions and dependencies required by different projects separate.

A virtual environment is itself a directory structure.

Use venv to create and manage Python virtual environments.

Use pip to install and manage Python external (third-party) libraries.

pip allows you to declare all dependencies for a project in a separate file (by convention called requirements.txt) which can be shared with collaborators/users and used to replicate a virtual environment.

Use pip3 freeze > requirements.txt to take snapshot of your project’s dependencies.

Use pip3 install -r requirements.txt to replicate someone else’s virtual environment on your machine from the requirements.txt file.

An IDE is an application that provides a comprehensive set of facilities for software development, including syntax highlighting, code search and completion, version control, testing and debugging.

With the correct extensions installed VS Code recognises virtual environments configured from the command line using venv and pip.

A branch is one version of your project that can contain its own set of commits.

Feature branches enable us to develop / explore / test new code features without affecting the stable main code.

Working with Git within VS Code

Always assume that someone else will read your code at a later date, including yourself.

Community coding conventions help you create more readable software projects that are easier to contribute to.

Python Enhancement Proposals (or PEPs) describe a recommended convention or specification for how to do something in Python.

Style checking to ensure code conforms to coding conventions is often part of IDEs.

Consistency with the style guide is important - whichever style you choose.

Use linting tools on the command line (or via continuous integration) to automatically check your code style.

Using testing requires us to change our practice of code development, but saves time in the long run by allowing us to more comprehensively and rapidly find faults in code, as well as giving us greater confidence in the correctness of our code.

The use of test techniques and infrastructures such as parameterisation and Continuous Integration can help scale and further automate our testing process.

The three main types of automated tests are unit tests, functional tests and regression tests.

We can write unit tests to verify that functions generate expected output given a set of specific inputs.

It should be easy to add or change tests, understand and run them, and understand their results.

We can use a unit testing framework like pytest to structure and simplify the writing of tests.

We should test for expected errors in our code.

Testing program behaviour against both valid and invalid inputs is important and is known as data validation.

We can assign multiple inputs to tests using parametrisation.

It’s important to understand the coverage of our tests across our code.

Writing unit tests takes time, so apply them where it makes the most sense.

Continuous Integration can run tests automatically to verify changes as code develops in our repository.

CI builds are typically triggered by commits pushed to a repository.

We need to write a configuration file to inform a CI service what to do for a build.

Builds can be enabled and configured separately for each branch.

We can run - and get reports from - different CI infrastructure builds simultaneously.

Unit testing can show us what does not work, but does not help us locate problems in code.

Use a debugger to help you locate problems in code.

A debugger allows us to pause code execution and examine its state by adding breakpoints to lines in code.

Use preconditions to ensure correct behaviour of code.

Ensure that unit tests check for edge and corner cases too.

Using linting tools to automatically flag suspicious programming language constructs and stylistic errors can help improve code robustness.

Software engineering takes a wider view of software development beyond programming (or coding).

Ensuring requirements are sufficiently captured is critical to the success of any project.

Following a process makes development predictable, can save time, and helps ensure each stage of development is given sufficient consideration before proceeding to the next.

When writing software used for research, requirements will almost always change.

Consider non-functional as well as functional requirements.

Consider the intended longevity of any code before you write it.

The perspective and language of a particular requirement stakeholder group should be reflected in requirements for that group.

Planning software projects in advance can save a lot of effort and reduce ‘technical debt’ later - even a partial plan is better than no plan at all.

The environment in which users run our software has an effect on many design choices we might make.

By breaking down our software into components with a single responsibility, we avoid having to rewrite it all when requirements change. Such components can be as small as a single function, or be a software package in their own right.

When writing software used for research, requirements will almost always change.

‘Good code is written so that is readable, understandable, covered by automated tests, not over complicated and does well what is intended to do.’

A software paradigm describes a way of structuring or reasoning about code.

Different programming languages are suited to different paradigms.

Different paradigms are suited to solving different classes of problems.

A single piece of software will often contain instances of multiple paradigms.

Classes allow us to organise data into distinct concepts.

By breaking down our data into classes, we can reason about the behaviour of parts of our data.

Relationships between concepts can be described using inheritance (is a) and composition (has a).

By breaking down our software into components with a single responsibility, we avoid having to rewrite it all when requirements change. Such components can be as small as a single function, or be a software package in their own right.

Collaborative techniques and tools play an important part of research software development in teams.