Keyword: GPU
Paper Title Other Keywords Page
THBPA04 Orchestrating MeerKAT's Distributed Science Data Processing Pipelines ion, controls, framework, network 1152
  • A.F. Joubert, B. Merry
    SKA South Africa, National Research Foundation of South Africa, Cape Town, South Africa
  The 64-antenna MeerKAT radio telescope is a precursor to the Square Kilometre Array. The telescope's correlator beamformer streams data at 600 Gb/s to the science data processing pipeline that must consume it in real time. This requires significant compute resources, which are provided by a cluster of heterogeneous hardware nodes. Effective utilisation of the available resources is a critical design goal, made more challenging by requiring multiple, highly configurable pipelines. We initially used a static allocation of processes to hardware nodes, but this approach is insufficient as the project scales up. We describe recent improvements to our distributed container deployment, using Apache Mesos for orchestration. We also discuss how issues like non-uniform memory access (NUMA), network partitions, and fractional allocation of graphical processing units (GPUs) are addressed using a custom scheduler for Mesos.  
slides icon Slides THBPA04 [8.485 MB]  
DOI • reference for this paper ※  
Export • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)  
THPHA042 ASCI: A Compute Platform for Researchers at the Australian Synchrotron ion, interface, hardware, synchrotron 1455
  • J. Marcou, R.R.I. Bosworth
    ASCo, Clayton, Victoria, Australia
  • R. Clarken
    SLSA-ANSTO, Clayton, Australia
  • P. Martin, A. Moll
    SLSA, Clayton, Australia
  The volume and quality of scientific data produced at the Australian Synchrotron continues to grow rapidly due to advancements in detectors, motion control and automation. This makes it critical that researchers have access to computing infrastructure that enables them to efficiently process and extract insight from their data. To facilitate this, we have developed a compute platform to enable researchers to analyse their data in real time while at the beamline as well as post-experiment by logging in remotely. This system, named ASCI, provides a convenient web-based interface to launch Linux desktops running inside Docker containers on high-performance compute hardware. Each session has the user's data mounted and is preconfigured with the software required for their experiment. This poster will present the architecture of the system and explain the design decisions in building this platform.  
poster icon Poster THPHA042 [1.402 MB]  
DOI • reference for this paper ※  
Export • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)  
THPHA186 Parallel Execution of Sequential Data Analysis ion, GUI, data-analysis, controls 1877
  • J.F.J. Murari, K. Klementiev
    MAX IV Laboratory, Lund University, Lund, Sweden
  The Parallel Execution of Sequential Data Analysis (ParSeq) software has been developed to work on large data sets of thousands spectra of a thousand points each. The main goal of this tool is to perform spectroscopy analysis without delays on the large amount of data that will be generated on Balder beamline at Max IV *. ParSeq was developed using Python and PyQt and can be operated via scripts or graphical user interface (GUI). The pipeline is consisted of nodes and transforms. Each node generally has a common group of components: data manager (also serves as legend), data combiner, metadata viewer, transform dialog, help panel and a plot window (from silx library **) as main element. The transforms connect nodes, applying the respective parameters in the active data. It is also possible to create cross-data linear combinations (e.g. averaging, RMS or PCA) and propagate them downstream. Calculations will be done with parallel execution on GPU. The GUI is very flexible and user-friendly, containing splitters, dock widgets, colormaps and undo/redo options. The features mentioned are missing in other analysis platforms what justifies the creation of ParSeq.
* Klementiev, K., et al. "The BALDER Beamline at the MAX IV Laboratory" Journal of Physics: Conference Series. IOP Publishing, 2016
** Scientific Library for eXperimentalists -
poster icon Poster THPHA186 [0.407 MB]  
DOI • reference for this paper ※  
Export • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)