HMI For ABB Robots
Prepared by:
Ben Grumann, Joshua Parkinson, Madelyn Brown
Figure 1: Main Product Image
Product Description:
HMI for ABB Robots is a project that has created an open-source Human Machine Interface(HMI) to work with ABB robotic systems. This project is sponsored by Flex Automation and has been in progress since September 2023. The HMI uses a Raspberry Pi 4 and a 7 inch touchscreen display to interface with the robot and instruct it to do certain tasks, such as controlling I/O (Inputs/Outputs) or moving to different positions. HMI for ABB Robots was developed to be a low cost alternative to industrial programmable logic controllers (PLC) in a maintenance or manufacturing environment, where all aspects of a PLC are not required.
Methodology in Brief:
The first half of this project was spent gathering the tools and technology needed to complete this project. The team acquired a Raspberry Pi and touch screen due to its availability, customization options, and easy to understand interface. Python was also used to build the HMI screen for similar reasons and, as such, the display is completely customizable.
The second half was focused on more technical goals, such as connecting over Ethernet to a loaner robot provided to the team by Flex Automation so the Pi could send and receive signals to and from the machine. Unfortunately, the robot lacked the PC Interface option, which is the software needed to communicate with another device over Ethernet, so resorting to a simulated robot was necessary. Installing PC Interface onto the provided robot would be a near $2000 expense, and Flex Automation advised the team to turn to RobotStudio, ABB’s simulation software. The robot also had physical limitations that were investigated later on regarding Joint 1 which would have restricted what was possible with the HMI screen as well.
As soon as a simulation was deemed a better testing option, the team researched how to send signals back and forth between two separate programs using socketing. Setting up socketing through a Raspberry Pi is like setting up a small server that listens for devices that want to connect to it, and it was determined that this was the most straightforward way to send signals.
On the Python side of the device, the HMI Screen was designed to handle 6 I/O Controls, 6 Move Controls, and 6 Programmed Actions (Figure 2). This interface is entirely customizable to the customer’s needs, and can support more or less of each category, as well as new categories if desired. For each button, a command string was assigned when the window was clicked or pressed, and this string was sent through socketing to the RobotStudio code. This ensures that no unauthorized operators can simply plug into the robot with a computer and begin performing actions, as a majority of the movements must be implemented by a trusted engineer within the robots own code.
Figure 2: HMI Screen Design
On the RobotStudio side, each position’s coordinates are pre-determined and hard coded into the beginning of the program. These coordinates define the positions the robot will move to when a string from Python is received, and can be swapped around as desired. In the current code there are 6 movements the robot can go through, as well as 6 actions that have the robot build and move a simulated block tower. This proves that socketing is able to control I/O on the robot, as the gripper can open and close through button presses on the HMI. Once again, these actions and sequences can be programmed to whatever the customer desires, such as a home position, a maintenance position, turning end effectors on or off, or anything else a regular PLC is capable of.
Bill Of Materials
Tools Used
Assembly (On Single Test Device)
- Download both RobotStudio and Visual Studio Code. Create a Python file in VSCode and paste the Python Code [Appendix A] into the file. [Tutorial on creating a Python file in VSCode]
- Open File Explorer and navigate to the Stations Folder in RobotStudio (Usually under Documents > RobotStudio > Stations). Download the station file (HMI Station.rsstn) from the repository [Link] and place them in this folder.
- Open RobotStudio. Click Open > Open, until File Explorer opens. Navigate to the HMI Station file and double click to open the station [Figure 3].
Figure 3: Opening the HMI Station [‘Recent’ will only be available after opening for the first time]
- To navigate to the RobotStudio code, click RAPID [Top bar] > RAPID [Side bar] > T_ROB > Socket > Main, then double click to open [Figure 4]
Figure 4: Accessing Main Code
- Double check that all code is present. If not, insert RobotStudio Code [Appendix A] into the code window. Navigate to the Simulation tab on the top bar and click on it, then press Play. There should be a message that lets the user know the program has started [Figure 5].
Figure 5: Play Button Indication and Program Start Message
- Navigate back to the Python file and click the Play button in the top left corner. The Flex Automation HMI Screen should appear in full screen. The ‘X’ in the top left corner will exit full screen to allow the user to see the robot’s movements when on a single device.
- The HMI should be fully operational. Refer to the move guide below as a reference to what each button should do.
Move Guide
| I/O Controls | Move Controls | Programmed Actions |
| Open Gripper | Home (Pointed Forward) | Pick Top Block |
| Close Gripper | J2 Bent Back | Move Top Block |
| Unused | All Joints Straight Up | Pick Bottom Block |
| Unused | Twist J4 | Move Bottom Block |
| Unused | Point J5 Up | Move Bottom Block Back |
| Unused | Pick Bottom Block | Move Top Block Back |
- There is an ‘Unpack and Work’ option for the robot as well. In RobotStudio go to File > Share > Unpack and Work [Figure 6], then select the downloaded station file (HMI Station.rspag) from the OSF Repository [Link]. This is a secondary option if the primary setup does not work.
Figure 6: Unpack and Work Option
Programming a Custom Button (VS Code side)
- Within the python code there are three for loops going from 0 to 6 (for i in range(6)), each loop has a comment above for which column of buttons it is creating.
- If more or less buttons are needed, the value currently set to 6 can be changed to the number of buttons needed in that specific column.
- If the text displayed on the button is desired to be something different, the code would need to be modified slightly:
for i in range(6):
- If the text displayed on the button is desired to be something different, the code would need to be modified slightly:
button_number = button_number + 1
if button_number = x:
button = PushButton(move_controls_button_box, args=(“Move”,button_number), text=”desired button text”, command=move_button_click, align=”top”, width=”fill”, height=”fill”)
else:
button = PushButton(move_controls_button_box, args=(“Move”,button_number), text=”Move {button_number}”, command=move_button_click, align=”top”, width=”fill”, height=”fill”)
- In the case above, more if statements could be added for each button desired to have different text, x needs to be the number of the button, and the text= in the if statement can be changed to the desired text to display on button number x.
- At the beginning of the code there are three function definitions to handle the actions of each column of buttons, declared as def io_button_click, def move_button_click, or def programmed_button_click. Currently these all just have laddered if statements to ensure there is no unintended signal being sent. Each if button has the value of i set to its position counting up from one starting at the top (for example the top button has an i value of 1, and the current bottom button has an i value of 6).
- Any added buttons will need to have another if statement added for its position in the column (for example if one more button was added, its i value would be 7, an if statement, if i == 7:, will need to be added).
- The command string can be anything you wish, as long as the RobotStudio code expects the command string the Python code is sending.
Programming a Custom Button (RobotStudio side)
- Navigate to the series of If statements. These statements are where the button press commands are processed.
- Copy and paste the code from a previous button.
- In the quotation marks (“”) put in the command string that your Python program is sending.
- Now, in between the THEN and ENDIF, add the code you wish to be activated when the button is pressed. This should allow the program to be run.
- These instructions can be used to process any inputs from python and allow whatever is inside the IF statements to be run including: I/O, robot movements, and anything else that is able to be coded in RAPID.
Sponsors/Acknowledgements
Sponsored by Flex Automation
Contacts: Scott Pink and Keith Weller
Thank you for consistent guidance and support.
References
Appendix A – Code [Python, RobotStudio]
Python Code
from guizero import App, Box, PushButton, Text, Window, Picture
from PIL import Image, ImageTk
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((‘127.0.0.1’, 55555))
# Functions to handle button click
def io_button_click(value, i):
print(f”{value} Button {i} clicked”)
if i == 1:
command = “io 1”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 2:
command = “io 2”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 3:
command = “io 3”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 4:
command = “io 4”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 5:
command = “io 5”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 6:
command = “io 6”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
def move_button_click(value, i):
print(f”{value} Button {i} clicked”)
if i == 1:
command = “move 1”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 2:
command = “move 2”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 3:
command = “move 3”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 4:
command = “move 4”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 5:
command = “move 5”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 6:
command = “move 6”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
def programmed_button_click(value, i):
print(f”{value} Button {i} clicked”)
if i == 1:
command = “program 1”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 2:
command = “program 2”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 3:
command = “program 3”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 4:
command = “program 4”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 5:
command = “program 5”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
if i == 6:
command = “program 6”
print(socket.gethostbyname(socket.gethostname()))
sock.send(command.encode())
print(sock.recv(4096).decode(‘UTF-8’))
def open_io_window():
IO_window.set_full_screen()
IO_window.show(wait = True)
def exit_fullscreen():
main_window.exit_full_screen()
def close_window():
IO_window.hide()
def ESTOP():
ESTOP_window.set_full_screen()
ESTOP_window.show(wait = True)
# Window Definitions
main_window = App(title=”Flex Automation HMI”, bg=”#8a8c8c”)
IO_window = Window(main_window, title=”IO Window”)
IO_window.hide()
ESTOP_window = Window(main_window, title=”ESTOP Window”)
ESTOP_window.hide()
# Main Window Layout
main_window_top_bar = Box(main_window, width = “fill”, align = “top”)
image = Image.open(“OSHE_Gear_MTUColors_Square.png”)
resized_image = image.resize((40, 40))
scaled_image = ImageTk.PhotoImage(resized_image)
oshe = Picture(main_window_top_bar, align = “left”, image = scaled_image)
open_io_button = PushButton(main_window_top_bar, command = open_io_window, text = “Open IO Window”, align = “left”)
close_button = PushButton(main_window_top_bar, command = exit_fullscreen, text = “x”, align = “right”)
ESTOP_button_main = PushButton(main_window_top_bar, command = ESTOP, text = “ESTOP”, width = “fill”, align = “right”)
# Create a box for each column
io_controls_box = Box(main_window, align=”left”, width=”fill”, height=”fill”)
move_controls_box = Box(main_window, align=”left”, width=”fill”, height=”fill”)
programmed_actions_box = Box(main_window, align=”left”, width=”fill”, height=”fill”)
# Text labels for each column
io_controls_label = Text(io_controls_box, text=”I/O Controls”, align=”top”, height=1, size=20)
io_controls_button_box = Box(io_controls_box, align=”top”, width=”fill”, height=”fill”)
move_controls_label = Text(move_controls_box, text=”Move Controls”, align=”top”, height=1, size=20)
move_controls_button_box = Box(move_controls_box, align=”top”, width=”fill”, height=”fill”)
programmed_actions_label = Text(programmed_actions_box, text=”Programmed Actions”, align=”top”, height=1, size=20)
programmed_actions_button_box = Box(programmed_actions_box, align=”top”, width=”fill”, height=”fill”)
# Buttons in the first column
button_number = 0
for i in range(6):
button_number = button_number + 1
button = PushButton(io_controls_button_box, args=(“I/O”,button_number), text=f”IO Control {button_number}”, command=io_button_click, align=”top”, width=”fill”, height=”fill”)
# Buttons in the second column
button_number = 0
for i in range(6):
button_number = button_number + 1
if button_number = x:
button = PushButton(move_controls_button_box, args=(“Move”,button_number), text=f”Move Control {button_number}”, command=move_button_click, align=”top”, width=”fill”, height=”fill”)
# Buttons in the third column
button_number = 0
for i in range(6):
button_number = button_number + 1
button = PushButton(programmed_actions_button_box, args=(“Program”,button_number), text=f”Action {button_number}”, command=programmed_button_click, align=”top”, width=”fill”, height=”fill”)
# Display the app
main_window.set_full_screen()
main_window.display()
RobotStudio Code
MODULE Socket
VAR jointtarget home:=[[0,0,0,0,0,0],[0, 9E9, 9E9, 9E9, 9E9, 9E9]];
VAR jointtarget away:=[[0,12,12,0,12,0],[0, 9E9, 9E9, 9E9, 9E9, 9E9]];
VAR jointtarget pos1:=[[-91.7959,0,0,0,0,0],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget pos2:=[[-2.11694,-64.0494,0,0,0,0],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget pos3:=[[-0.0616317,-1.86471,-82.989,0,0,0],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget pos4:=[[-0.00181666,-0.0549641,-2.44619,93.6866,0,0],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget pos5:=[[-9.85037E-6,1.05668,1.04371,0.507992,-94.4349,0],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget pos6:=[[0.0143926,59.5328,-3.24122,-3.14195E-07,33.7084,0.340009],[9E+09,9E+09,9E+09,9E+09,9E+09,9E+09]];
VAR jointtarget pos7:=[[0.0144536,62.6616,-3.89281,7.60414E-07,31.2312,0.340069],[9E+09,9E+09,9E+09,9E+09,9E+09,9E+09]];
VAR jointtarget safepos:=[[0.0144539,54.6234,-1.23053,6.61144E-07,36.6071,0.340069],[9E+09,9E+09,9E+09,9E+09,9E+09,9E+09]];
VAR jointtarget safepos1:=[[0.0144539,55.3065,-1.76975,-2.02767E-06,36.4632,0.185804],[9E+09,9E+09,9E+09,9E+09,9E+09,9E+09]];
VAR jointtarget newloc1:=[[-48.1214,63.4881,-4.74863,-1.58072E-5,31.1176,0.807374],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget safenewloc1:=[[-48.1214,55.3114,-1.7799,-1.3745E-5,36.4684,0.807372],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget newloc2:=[[-48.1214,57.8405,-2.46682,-1.62857E-5,34.6262,0.807365],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
VAR jointtarget safenewloc2:=[[-48.1214,49.1011,-0.958213,-1.28171E-5,41.857,0.807371],[9E+9,9E+9,9E+9,9E+9,9E+9,9E+9]];
PROC main()
VAR socketdev server_socket;
VAR socketdev client_socket;
VAR string receive_string;
VAR string client_ip;
VAR bool safe;
VAR bool grabbed;
VAR bool safe2;
VAR bool safe3;
grabbed:=FALSE;
safe :=FALSE;
safe2:=FALSE;
safe3:=FALSE;
SocketCreate server_socket;
SocketBind server_socket, “127.0.0.1”, 55555;
SocketListen server_socket;
SocketAccept server_socket, client_socket\ClientAddress:=client_ip\Time:=WAIT_MAX;
SetDO Attach, 0;
SetDO Detach,1;
! Waiting for a connection request
WHILE TRUE DO
SocketReceive client_socket \Str := receive_string;
SocketSend client_socket \Str := receive_string;
IF receive_string = “home” THEN
MoveAbsJ home, v4000, z15, tool0;
ENDIF
IF receive_string = “go away” THEN
MoveAbsJ away, v4000, z15, tool0;
ENDIF
IF receive_string = “move 1” THEN
MoveAbsJ pos1, v4000, z15, tool0;
ENDIF
IF receive_string = “move 2” THEN
MoveAbsJ pos2, v4000, z15, tool0;
ENDIF
IF receive_string = “move 3” THEN
MoveAbsJ pos3, v4000, z15, tool0;
ENDIF
IF receive_string = “move 4” THEN
MoveAbsJ pos4, v4000, z15, tool0;
ENDIF
IF receive_string = “move 5” THEN
MoveAbsJ pos5, v4000, z15, tool0;
ENDIF
IF receive_string = “move 6” THEN
MoveAbsJ pos7, v4000, z15, tool0;
ENDIF
IF receive_string = “program 5” THEN
!Release original bottom block in original location 011
IF safe2=TRUE THEN
IF safe = FALSE THEN
IF grabbed=FALSE THEN
grabbed:=TRUE;
MoveAbsJ safenewloc2, v4000, z15, tool0;
MoveAbsJ newloc2, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 1;
SetDO Detach,0;
WaitTime 1;
MoveAbsJ safenewloc2, v4000, z15, tool0;
MoveAbsJ safepos, v4000, z15, tool0;
MoveAbsJ pos7, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 0;
SetDO Detach,1;
WaitTime 1;
MoveAbsJ safepos, v4000, z15, tool0;
ELSE
receive_string := “Top block not moved”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not moved”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “You have the top block”;
SocketSend client_socket \Str := receive_string;
ENDIF
ENDIF
IF receive_string = “program 3” THEN
! MoveAbsJ pos6, v4000, z15, tool0;
!Grab original bottom block from original location
IF safe2=FALSE THEN
IF safe = TRUE THEN
IF grabbed = FALSE THEN
MoveAbsJ safepos, v4000, z15, tool0;
MoveAbsJ pos7, v4000, z15, tool0;
WaitTime 1;
SetDO Detach,0;
SetDO Attach,1;
WaitTime 1;
grabbed:=TRUE;
MoveAbsJ safepos, v4000, z15, tool0;
ELSE
receive_string := “Top block not released”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not released”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not moved”;
SocketSend client_socket \Str := receive_string;
ENDIF
ENDIF
IF receive_string = “program 1” THEN
!Grab original top block from original location
IF safe2=FALSE THEN
IF safe=FALSE THEN
IF grabbed=FALSE THEN
grabbed:=TRUE;
MoveAbsJ safepos1, v4000, z15, tool0;
MoveAbsJ pos6, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 1;
SetDO Detach,0;
WaitTime 1;
MoveAbsJ pos6, v4000, z15, tool0;
MoveAbsJ safepos1, v4000, z15, tool0;
ELSE
receive_string := “Block in Gripper”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Block in Gripper”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Bottom block not placed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ENDIF
IF receive_string = “program 2” THEN
!Move original top block to new location
IF safe2=FALSE THEN
IF safe=FALSE THEN
IF grabbed = TRUE THEN
MoveAbsJ safenewloc1, v4000, z15, tool0;
MoveAbsJ newloc1, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 0;
SetDO Detach,1;
WaitTime 1;
grabbed:=FALSE;
safe:=TRUE;
MoveAbsJ safenewloc1, v4000, z15, tool0;
ELSE
receive_string := “Top block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not moved”;
SocketSend client_socket \Str := receive_string;
ENDIF
ENDIF
IF receive_string = “program 4” THEN
!Move original bottom block to new location
IF safe2=FALSE THEN
IF safe = TRUE THEN
IF grabbed = TRUE THEN
MoveAbsJ safenewloc2, v4000, z15, tool0;
MoveAbsJ newloc2, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 0;
SetDO Detach,1;
WaitTime 1;
grabbed:=FALSE;
safe:=FALSE;
safe2:=TRUE;
MoveAbsJ safenewloc2, v4000, z15, tool0;
ELSE
receive_string := “Bottom Block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Bottom Block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not moved”;
SocketSend client_socket \Str := receive_string;
ENDIF
ENDIF
IF receive_string = “program 6” THEN
!Release original top block in original location
IF safe2 =TRUE THEN
IF safe=FALSE THEN
IF grabbed = TRUE THEN
MoveAbsJ safenewloc1, v4000, z15, tool0;
MoveAbsJ newloc1, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 1;
SetDO Detach,0;
WaitTime 1;
MoveAbsJ safenewloc1, v4000, z15, tool0;
MoveAbsJ safepos1, v4000, z15, tool0;
WaitTime 1;
SetDO Attach, 0;
SetDO Detach,1;
WaitTime 1;
MoveAbsJ safepos1, v4000, z15, tool0;
safe2:=FALSE;
safe:=FALSE;
grabbed:=FALSE;
ELSE
receive_string := “Top block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ELSE
receive_string := “Top block not grabbed”;
SocketSend client_socket \Str := receive_string;
ENDIF
ENDIF
IF receive_string = “io 1” THEN
SetDO Attach,0;
SetDO Detach,1;
grabbed:=FALSE;
ENDIF
IF receive_string = “io 2” THEN
SetDO Attach,1;
SetDO Detach,0;
grabbed:=TRUE;
ENDIF
IF receive_string = “end” THEN
SocketClose client_socket;
ENDIF
ENDWHILE
ERROR
RETRY;
UNDO
SocketClose server_socket;
SocketClose client_socket;
ENDPROC
ENDMODULE