When we talk about Big Data applications, the ‘Big’ is not only related to quantities, but also to the quality of data. In EW-Shopp project, we are dealing with multi-form, multi-source data types, proprietary enterprise and customer data, market data and intelligence, open data and external data as well as multilingual content. For this reason, we are developing (and testing out) a data integration methodology which will help us incorporating both technical and business-analytical aspects in an effective workflow.
In EW-Shopp we are developing a cross-domain data integration platform which aims to support businesses and business ecosystems to leverage on the huge amounts of data available today in order to build relevant custom insights and business knowledge around weather- and event-based data.
Today, we know that in Big Data applications the ‘Big’ is just partially related to quantities (i.e. volumes of data): what’s much more relevant is quality. In EW-Shopp we are dealing with multi-form, multi-source data types, proprietary enterprise and customer data, market data and intelligence, open data and external data as well as multilingual content. Because of the variety and richness of this business-technical ecosystem, we also need to manage diverse data paradigms such as traditional tabular documents, Relational and NoSQL databases as well as various data formats.
Integrating multiple data sources and formats, and therefore harmonising and reconciling these huge amounts of data from a technical point of view, is an important result which we strive to obtain, and we are integrating the best tools and techniques in the domain. But that is it is only one side of the full data integration coin. In fact, in EW-Shopp we really want to enable businesses to gather meaningful weather- and events-based insights about their customers, markets and operations providing useful and usable services.
To this end, we are developing, and testing out, a data integration methodology which helps us incorporate both technical and business-analytical aspects in an effective workflow. We are applying this methodology to each of our Pilots which are testing EW-Shopp with real-word business cases and data.
The methodology foresees a workflow with four well-identifiable building blocks, or stages, each of which can also be managed by four different actors (people, companies, etc.), allowing for a degree of flexibility and modularity.
The stages are:
· Documentation
· Ingestion
· Enrichment
· Analytics
The figure below summarises a typical EW-Shopp integration scenario which comprises the four stages.
