Introduction
| DISCLAIMER: Development started with the release of the Trio guest mode and we have not yet been able to test it. It may be the subject of a future article once we have studied it. |
|---|
Developing a Qt application around an asynchronous framework in Python can be tricky. In this article, we will try to describe the issues we encountered when creating Parsec, which uses the Trio library.
What is Parsec?
Parsec is an open-source zero-trust file-sharing program (ergonomically similar to Dropbox, but with client-side encryption). The source code is available on Github.
Parsec is centered around a metadata server, and is a client application, originally available only through a command-line interface (CLI). It makes extensive use of the asynchronous programming paradigm(https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)) with Trio. All this works well as long as we only use the CLI, but becomes complex when we want a desktop application with Qt.
What is the problem with asynchronous and Qt?
Qt is built around an event loop that usually runs in the main thread. As long as there is no interaction with the software, the loop waits for an event to occur. As soon as the event occurs (a user clicks on a button for example), a callback is called and allows us to react.
from PyQt5.QtWidgets import QApplication, QMainWindow, QPushButton, QMessageBox
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
# We add a button
button = QPushButton("Click me", parent=self)
# Bind the button's "clicked" signal to our method
button.clicked.connect(self._on_button_clicked)
def _on_button_clicked(self):
# We display a message that the button has been clicked
QMessageBox.information(self, "Information", "The button has been clicked!")
if __name__ == "__main__":
app = QApplication([])
# We create our window
win = MainWindow()
win.show()
# Start the event loop
app.exec_()
The problem arises when we try to start Trio at the same time. Since Qt monopolizes the main thread, we can't run Trio simultaneously (Qt will block Trio or vice versa). The obvious solution is to use multiple threads (one for Qt, one for Trio), but other problems arise as we're not only forced to navigate between threads, but also between synchronous and asynchronous contexts. Added to this is the impossibility of updating a Qt widget from a thread other than that of the Qt event loop.
In Parsec, we need to react in the GUI (Graphical User Interface) to events received from the metadata server, and send information to the server based on user input.
[Serveur de métadonnées] <—-> [Core – Trio – Async] <—-> [GUI – Qt – Sync]
How did we "solve" the problem?
We use threads (surprise!). We let Qt run in the main thread (by constraint because only the main thread can make changes to the GUI), we start another thread for Trio, and we use a specific context to interact between the two. Communications from the GUI to the core Trio are done by starting jobsIn addition, communications from the Trio core to the GUI use Qt signals. These can be thread-safe with the argument QueuedConnection). By default, Qt uses AutoConnection which automatically uses DirectionConnection (the callback is called immediately, i.e. in the same thread as the signal sender) or QueueConnection (the callback is called in the receiver thread) depending on the context.
# Imports are omitted
@contextmanager
def run_trio_thread():
# QtToTrioJobScheduler is defined [here](https://github.com/Scille/parsec-cloud/blob/master/parsec/core/gui/trio_thread.py#L153)
job_scheduler = QtToTrioJobScheduler()
thread = threading.Thread(target=trio_run, args=[job_scheduler._start])
thread.setName("TrioLoop")
thread.start()
job_scheduler.started.wait()
try:
yield job_scheduler
finally:
job_scheduler.stop()
thread.join()
class MainWindow(QMainWindow):
def __init__(self, jobs_ctx):
super().__init__()
self.jobs_ctx = jobs_ctx
if __name__ == "__main__":
app = QApplication([])
with run_trio_thread() as jobs_ctx:
# We create our window by providing the jobs_ctx
win = MainWindow(jobs_ctx)
win.show()
app.exec_()
One of the weak points is the obligation to walk jobs_ctx between the classes that need it.
Communication from Qt to the Trio core
We start a new job by passing Qt signals as parameters. For example, if we ask the user for his name and we want to provide it to the core :
class MainWindow(QMainWindow):
# Both signals are used to determine the status of the asynchronous function call
set_name_success = pyqtSignal()
set_name_failure = pyqtSignal()
def __init__(self, jobs_ctx):
super().__init__()
self.jobs_ctx = jobs_ctx
button = QPushButton("Click me", parent=self)
button.clicked.connect(self._on_button_clicked)
# We link both signals to our methods
self.set_name_success.connect(self._on_set_name_success)
self.set_name_failure.connect(self._on_set_name_failure)
def _on_button_clicked(self):
# The user is asked for his name
name = QInputDialog.getText(self, "GiveName", "Please enter your name:")
# Start the job
self.jobs_ctx.submit_job(
# ThreadSafeQtSignal is a wrapper that makes sure the signal is sent with the Qt.QueuedConnection parameter
ThreadSafeQtSignal(self, "set_name_success"),
ThreadSafeQtSignal(self, "set_name_failure"),
# core_set_name is our asynchronous function that runs in a Trio context
core_set_name,
name=name,
)
# If core_set_name is successful, the "set_name_success" signal is invoked, which will call this callback
def _on_set_name_success(self):
QMessageBox.information(self, "Success!", "Your name has been taken into account.")
# If core_set_name fails, the "set_name_failure" signal is invoked, which will call this callback
def _on_set_name_failure(self):
QMessageBox.warning(self, "Error!", "Your name was not taken into account.")
submit_job also returns the job if we need it (to cancel the job for example) but for the sake of patenting, we don't detail it here. The full implementation is available here.
Communication from core Trio to Qt
The core also needs to communicate with the GUI. For example, a user shares a file with us. It would be nice to be able to react to this event and inform us.
This case is simpler because we can use the Qt signals directly.
class MainWindow(QMainWindow):
# Ce signal sera émit quand un fichier est partagé avec nous. Les deux paramètres sont l'utilisateur qui a créé le partage et le nom du fichier.
file_shared = pyqtSignal(str, str)
def __init__(self, jobs_ctx):
super().__init__()
self.jobs_ctx = jobs_ctx
# On se connecte à l'événement FILE_SHARED du core et appelons _on_core_file_shared quand il arrive.
# L'event_bus est une simple clé associée à une liste de callback. L'implémentation complète est disponible [ici](https://github.com/Scille/parsec-cloud/blob/master/parsec/event_bus.py).
self.jobs_ctx.event_bus.connect(FILE_SHARED, self._on_core_file_shared)
# On connect notre signal file_shared à _on_file_shared.
# QueuedConnection est utilisé par défaut par Qt mais on le rend explicite pour l'exemple.
self.file_shared.connect(self._on_file_shared, Qt.QueuedConnection)
def _on_core_file_shared(self, user_name, file_name):
# On passe les arguments au signal
# On ne fait rien dans cette méthode, nous ne sommes pas dans le thread Qt.
self.file_shared.emit(user_name, file_name)
def _on_file_shared(self, user_name, file_name):
QMessageBox.information(self, "Fichier partagé !", f"Le fichier '{file_name}' a été partagé par l'utilisateur '{user_name}'")
Conclusion
This solution works, we are well able to juggle between threads and contexts, but it adds a lot of complexity and a lot of boilerplate. In our case, the interactions between Trio and Qt are not too numerous but in more complex cases, we can easily end up with spaghetti code.
To address this issue, Trio has released its guest mode, and some libraries have been built around it, such as QTrio, but these are very recent and we have not yet had the opportunity to test them.
