🔧 ETL pipeline

As my main project at the LNMC group of the Blue Brain Project, I worked in developing an ETL pipeline for electrophysiological time series data. It monitors incoming raw data from several robots which conduct automated patch-clamp experiments on single cells to determine the effects of drugs on ion channels.

The pipeline uses Dagster and is deployed through Docker on an internal server. The raw measurements and metadata of each experiment are validated and converted to an internal HDF5 specification. The new data is analysed together with the previous experiments by the scientists on Python notebooks through a specialized statistical analysis library, including ML classifiers on engineered features to automatically filter succesful experiments.

For more information on the project, see Channelpedia, where the drug screening results are made available, among others.