Architecture

In the Big Data Europe framework, the Big Data Integrator is an application that can be thought as a “starter kit” to start working and implementing big data pipelines in your process. It is the minimal standalone system so you can create a project with multiple docker containers, upload it & make it run using a nice GUI.

At it’s core it is a simple web application that will render each different service’s frontends inside it, so it is easy to navigate between each system providing a sense of continuity in your workflow.

The basic application to start from is constituted of several components:

That service will be the “referee” that imposes the steps defined in the workflow builder. For more information check it’s repository.

Systems are organized following a microservices architecture and run together using a docker-compose script, some of them sharing microserviecs common to all architectures, like the identifier, dispatcher, or resource. This is a more visual representation of the basic architecture:

Have fun with it!

Related posts:

Sometimes writing is the only that keeps my days straight. From thoughts to therapy we stay busy. I hear the connections Owen is making and I see the progress, he is starting to use a coded language…

What happened to all those great social networks that came before. Were they relegated to the dustbin of history, never to be seen or heard from again, or did they morph into something else? Their…