Blue-Cloud D2.8 - Blue-Cloud Architecture (Release 3)

Schaap D. M. A., Thijsse P., Pagano P., Assante M., Candela L., Boldrini E., Buurman M., D'Antonio M., Ariyo C., Maudire G., Nys C., Chiavarini B., Lettere M.

System architecture  Virtual research environment  D4Science  Data Discovery  Data Analytics 

This deliverable D2.8 describes third and final release of the Blue-Cloud architecture as it is at Month 33 (June 2022) and it is and an update of the earlier 2nd release of the Blue-Cloud architecture document D2.7. Given the agreed extension allowed to the Blue-Cloud project until March 2023, there might be some further refinements to the architecture in the upcoming 9 months to allow to optimise some of its services to better respond to user needs. In order to make it easier for readers and reviewers, a table is included as part of Chapter 1, which indicates the elements and sections of this Deliverable 2.8, which have been updated or added in comparison with the earlier Deliverable 2.7. In this report, the current architecture and functionalities of each of the following components, part of the Blue-Cloud technical framework, are described in detail as well as the roles of partners that are developing and hosting modules: 1) the Blue-Cloud Data Discovery and Access service to serve federated discovery and access to blue data infrastructures; 2) the Blue-Cloud Virtual Research Environment (VRE) to provide a Blue-Cloud VRE as a federation of computing platforms and analytical services. The Blue-Cloud Data Discovery and Access service architecture is based upon a combination of the DAB metadata broker service of CNR-IIA, and the SeaDataNet CDI service modules as developed by MARIS, IFREMER, and EUDAT in the framework of the EU SeaDataCloud project. For the Blue-Cloud Data Discovery and Access service and its modules, additional developments were implemented in the period May 2021 - June 2022 such as adapting and upgrading existing services, adding new services, testing modules, integrating modules, and testing the integrated service, in order to achieve the planned functionality. The Blue-Cloud VRE is largely based upon the D4Science e-infrastructure as earlier developed and managed by CNR-ISTI [1]. This e-infrastructure hosted, already from the start, multiple Virtual Labs and offered a variety of services. These services have been adopted and adapted for Blue-Cloud, new services have been added and several original services have been upgraded. Moreover, new Virtual Labs have been constructed and deployed as part of the Blue-Cloud Demonstrators. The D4Science e- infrastructure also already included proven solutions for connecting to external computing platforms and means for orchestrating distributed services, which are instrumental for smart connections to the other e-infrastructures in the Blue-Cloud system, while further evolutions have taken place as part of the Blue-Cloud project, in response to the needs of the Virtual Labs and their users. With respect to the Virtual Labs, they are developed as real-life demonstrators embedded in the D4Science VRE and supported by data input from the Blue-Cloud Data Discovery and Access service, other data resources and additional computing services. They have been worked out in cooperation between WP3 and WP4 which have analysed their scientific workflows and identified the best technical set-up considering the D4Science VRE infrastructure and services. As part of their development, the demonstrators required upgrading of existing functionality and development of additional services. This is described, where relevant, in this document. In addition, consideration is given to integration aspects, such as two-way linking between the Blue- Cloud components, expanding the VRE with additional platforms for computing and algorithms, and direct access to data infrastructures where needed for specific Virtual Labs. Moreover, aspects of authentication and monitoring are considered on a full Blue-Cloud scale. The Blue-Cloud architecture as described in this report, is designed to be scalable and sustainable for near-future expansions, such as connecting additional blue data infrastructures, implementing more and advanced blue analytical services, configuring more dedicated Virtual Labs, and targeting more (groups of) users.

Source: ISTI Project report, Blue-Cloud, D2.8, 2022