Research software for hardware control and automation

Last Tuesday 17th of June, the Imperial College London Research Software Engineering community hosted a session of the “Research Software Conversation Series: Tools and techniques for modern research in different domains”. This event explored how research software is used for hardware control and automation, and specially the challenges and lessons learnt when doing so.

The event opened with a short talk by Dr Matthew Ward on Building a Solution Processing Robot: How Hard Can it Be? and was followed by a roundtable discussion with Prof. Jenny Nelson (Professor of Physics), Dr Jon Murray (Research Fellow), Dr Alex Dewar (Senior Research Software Engineer) and Dr Matthew Ward (Research Associate), and was facilitated by Dr Diego Alonso Álvarez (Head of Research Software Engineering).

Figure 1: The panel discussion. From left to right, Diego Alonso Alvarez, Matthew Ward, Alex Dewar, Jenny Nelson and Jon Murray

The event was very lively, with several of the attendees in person and online asking insightful questions and participating in the discussion, which made it more of an informal, open fireside chat than a formal academic event. Many topics were covered, and many other interesting ones were left in the drawer because of a simple lack of time. Here we summarise the key points, and it is expected that some of them will result in specific action points to make the development of research software for hardware control and automation easier, more impactful, and its value more recognised.

Pros and cons of writing software for automation

Research, by nature, is novel and unknown and requires equipment tailored to cover very specific, and often niche, needs. Out of the box solutions might facilitate some aspects of the experimental setups, but they are unlikely to cover the end-to-end problem at hand. Custom wiring of all of these components - literally, sometimes - is desirable for multiple reasons, from speeding up the discovery process, to make it more robust and reproducible, or to automate boring, repetitive tasks, freeing researcher's time to address other issues. Expensive solutions that claim to cover all the bases of the problem that needs solving are normally outside of reach within a research context unless significant funding is available, and even in those cases, they are sometimes not really worth the costs.

Despite the appeal of creating custom tools for automation, it comes with costs. To start with, the knowledge to write and build the required software or electronics might not be available, and researchers might need to self-teach a lot of new skills on the fly. Often, progress might be hampered by poor - or plainly wrong - documentation by the hardware manufacturers, lack of learning resources or the need to do trial and error when developing functionality, which could result in a risk for the equipment itself. Moreover, due to the limited experience or knowledge, researchers might choose the wrong tool for the task, for example a programming language that does not allow accessing all the required equipment, or a graphical user interface framework that limits the options when the software grows in size and complexity.

All of this requires a significant time investment and is not, in itself, a research output that can be used as a metric of a successful career as researcher. This is in contrast to papers, patents or even more traditional research software like the one created for modelling or simulation, which can be easily shared with collaborators and the community all around the world. However, despite these caveats, this effort is normally worthwhile, eventually paying off when the setup is up and running, and it is often the only path available to discovery.

The toolbox of a researcher

How the software development is done, the tools used and the time it takes to master them is obviously dependent on the researcher. Dr Ward explored some of these in his talk, and left the following take away messages:

• Don't reinvent the wheel - but don't buy a Ferrari. Reuse available tools and hardware that is fit for purpose whenever possible, but do not trust that an expensive piece of kit is worth its value, specially in a very customised use case.
• You need cheerleaders. Writing software for hardware and automation is often not as flashy as other software, and might be a solitary and ungrateful task until you get to the end. You would need other colleagues, lab users and, in general, interested parties involved along the way, supporting your work and providing constructive feedback.
• Use version control from the onset. It is not only a basic tool for any form of software development that should always be used, but also extremely useful when dealing with hardware interfaces, where a lot of trial and error might be necessary before landing on the solution that actually works.
• Artificial intelligence, used via ChatGPT or Copilot, for example, might be very useful to support the development process, specially to help with the first steps of interfacing with a piece of hardware out of the manufacturer specifications. However, it should be used within a very concise scope, and not as a black box. The solutions suggested by the AIs are not valid straight away, but will pave the way for the researcher to find the solution themselves.
• If a graphical user interface is to be implemented, and it might be desirable to avoid the misuse of the equipment and facilitating the on-boarding of users, an exploration of the possible frameworks should be done upfront. For example, tkinter, built-into Python, might be an easy to use solution, but down the line might curtail your freedom to scale up the complexity of the tool. Frameworks with a steeper learning curve, like PySide or PyQt , might be worth the time investment. Or even commercial solutions like LabView, which sometimes also facilitate interfacing with hardware.

Knowledge, skills and career paths

To support the development and appeal of custom software for hardware control and automation, a few aspects of the research ecosystem should be improved.

Having the support of a research software engineering team is invaluable to achieve good quality software, but in-house knowledge and skills are also paramount. There should be a baseline training and resources available, independent of the department, that researchers who need to can access to learn about software engineering, hardware interfaces and electronics. While software is ubiquitous in research, so is the need to effectively use, customise and improve experimental setups and lab equipment, where knowledge of electronics and interfaces is key. These resources are currently lacking except, maybe, for some specific departments (e.g. Electrical and Electronic Engineering), or they are hard to access, or their existence is simply not known.

In this same line, it is necessary to facilitate a direct knowledge exchange among researchers, fostering communities of practice within the departments themselves. This will reduce the duplication of efforts when interfacing with lab equipment and other hardware, which is likely to be common or very similar within the same domain, e.g. spectrometry.

Finally, there should be an established career path for these highly skilled PhD students and postdocs who have the technical know how and wish to continue their career development within the departments without competing for academic positions, which are very few and which have a totally different remit. Retaining this talent would facilitate the maintenance and long term sustainability of these resources, and potentially serving as a medium to train new people on the required skills.