Copenhagen Atomics’ Reactor Control System

In any kind of nuclear reactor or even any kind of industrial system you need some sort of control and monitoring. This article describes the reactor control system, which is used by Copenhagen Atomics.

This article is for technical nerds who want to know how it works and how it is different from other systems used in other reactors or other industrial systems.

This article does not cover the safety critical shutdown system of our reactors. These two systems are completely separated.

Molten Salt Reactors

Molten Salt Reactors are unique in that their power output (generated heat) automatically self-adjusts to generate the same amount of power as is removed from the system. This is called load following. Most solid fuel reactors do not have this feature, therefore they need a different kind of control and management. This of course results in a very different control system in Copenhagen Atomics reactors than all the classic solid fuel reactors.

When you monitor a CA reactor there are a number of things which you would want to be able to monitor in real time and also record, such as temperature, pressure, flow-rates, etc. We would also like to be able to monitor these values remotely. It is common sense to record the data continuously and upload them to a database.

There are already many products for industrial data monitoring in the market. But we looked at them and most of them had a single point of failure. This means they were designed from first principles such that they had one or more single points of failure. Many of them also had a lock-in philosophy, where using products from Siemens or Schneider would require you to only use their products and programming tools. There are of course varying degrees of lock-in but we learned that less lock-in typically also leads to more single points of failure in the system.

The next problem is that because of the lock-in there is not a lot of competition and this has hampered innovation in the industrial control sector for decades. The PLC and SCADA systems feel very much like an old IBM mainframe computer. But what we really wanted was something which is open source, with lots of innovation and flexibility and competition. The price was also an important parameter. A Siemens PLC is typically 10 - 100 times more expensive than a Raspberry Pi. And the raspberry pi computer has most of the things we want in terms of open source and flexibility where the Siemens, Honeywell and Schneider’s of the world did not.

There is a fun anecdote from the early days. The kind of people who tend to like a large all controlling government also like the model where once you select Siemens as your industrial controller, then you are forced to follow their rules and prices and recommendations. We did not like that, we wanted freedom and full flexibility and no single point of failure, low price and open source.

Architecture & RAFT

There exists several open source PLC systems like OpenPLC. But we decided to take the big step and completely abandon the IBM Mainframe era and jump to a modern architecture, which fully embraces open source, Raspberry Pi and modern development tools.

Figure 1: The system on the left is the 30-heater controller. The system on the right is a mid size (19 boxes) system with 100+ data points, 10 Hz. These are examples of CA control systems.

We chose the following technologies:

Over time people disliked Arduino and RS485, thus they are less dominant in our systems today.

Architecture wise we chose a system of IO boxes that are either collecting measurements (data) or controlling output (e.g. turning things on and off). These IO boxes always have a STM32 microcontroller in them, which communicates through USB-C with a computer node in an ethernet network. That computer node is always a Raspberry Pi. But of course the system is open source and you can choose any microcontroller or any PC CPU to run your network nodes. The system could also work with another cable and protocol than USB-C.

In this architecture all the network nodes collect data from all the IO boxes which are connected to them via USB-C continuously, which is then shared with the rest ten times per second (10 Hz). Thus ten times per second the system collects data from all measurements such as temperature and pressure and organizes all of these in a big vector. All the network nodes get all the data, we use an open source protocol called RAFT to do this.

The RAFT algorithm is robust and allows a computer in the network to reboot or get disconnected temporarily for various reasons. The system still works and all nodes get the vector, even if some of the measurement data are missing. Important measurements are always collected by multiple network nodes, by multiple boxes with multiple sensors. The system has a built-in algorithm for how to handle wrong or missing measurements and bit errors.

Ten times per second when all the network nodes (Raspberry Pi computers) also called PiHubs have received the big vector with data from all the other nodes, then they all run the “control program” to find out what the system should do next. Basically they calculate all the output settings of the system, such as turning motors and valves on or off and set the RPM of pumps or the target value of a PID controller. These output values are saved in the big vector and shared to all the network nodes, such that they can evaluate if all nodes agree on what to do next (called voting) and then execute the output in the big vector which all nodes agree on. If a valve or a heater is connected to a specific node, then this node is responsible for turning that valve or heater on or off.

In the case of a valve, you need to have several valves in parallel or in series to obtain redundancy. The choice of parallel or series depends on what is the safe state. If you choose a “normally open” valve, then you put your valves in series and allow 3 different IO boxes and network nodes to control each valve.

For pumps and motors with VFDs we have a VFD with two ethernet ports and built-in redundancy, such that it can always control the speed of the pump, even if one of the ethernet cables is broken or one of the microcontrollers dies. If you are used to building redundant systems, I am sure you get the picture.

Figure 2: Each network node is called PiHub in the diagram above. Yellow lines indicate ethernet cables

To upload data to the database we chose a QuestDB timeseries database, which is open source. But you can choose any other database if you like. QuestDB runs in a Kubernetes cluster on a server or in the cloud. These Kubernetes services also allow you to view the data in near real time (with 1 second delay) in a web plotter and you can control things like update software and configuration and name nodes and sensors via the same web user interface. You choose if you want to use your mobile phone or your laptop computer for this interaction.

For reactors we did not want users to be able to update the software or even send commands to the reactor when it is running. Therefore we have implemented a data diode, which is inserted just before the ethernet cable leaves the reactor container. This data diode has optocouplers inside, which ensure that data is only sent from the reactor up to the database and nothing can be sent in the other direction.

Configuration

In a system like the above with thousands of sensors and hundreds of IO boxes and twenty or more network nodes (PiHubs) manual configuration would take hours and is inherently error-prone. Therefore we have implemented autoconfig, which detects all sensors and boxes on the system and automatically generates an IO.config text file, which is the configuration of the system. Through a web UI you can change names, update software, turn valves on and off and much more. After you are done connecting the ethernet cables and the USB-C cables, you will run or re-run the autoconfig command and then everything starts uploading sensor data to the database.

Programming is done via graphical state machines. On state-machine transitions you select which value from the big vector to use as input to change state. Of course it can handle all kinds of mathematical functions and filters. Even people who have never programmed a single line of code ever, can understand this and use it after less than one hour introduction.

Open Source

Most of our code and electronics designs are open source and can be downloaded from GitHub. We also sell IO boxes and PiHub computers at close to cost price. People around the world use this system to control the beer brewing process in the garage and their scientific vacuum system in their university lab and pumps aboard an old guy’s boat in the harbor. Even kids can figure this out on their own. It is almost like the LEGO of control systems and then it is nuclear grade and open source.

Available IO boxes

Temperature, pressure, humidity, oxygen and other gasses, gas and liquid flow sensors, DC ports, AC ports, VFDs, current, voltage, oven controllers, light controllers, motors, audio, video, scales, position, liquid level, density, viscosity, integration with Festo and other pneumatics systems. And much more.

Connections to other systems via SignalR, LabView or IFTTT. The system can also send email or give alarms in other ways and it supports many users with different access levels. The system supports redundancy and floating leader network nodes, which means it is hard to bring down the system even if you cut cables or remove boxes. The system supports a large number of input / output modules via USB and RS485.