As more organizations move their business intelligence (BI) environments to the cloud, loading and accessing enterprise data will become as important as the front-end visualizations. Oracle’s BI Cloud Service (BICS) offers two options for these data requirements that go beyond simple data uploads: Data Sync and Remote Data Connector (RDC).
Other options range from simple manual data loads of spreadsheets using the BICS front end to advanced programmatic options based on REST APIs. Each has a specific purpose, features, benefits, and limitations. As the migration to the cloud and the tools used to support that transition are still in their early stages, this blog post discusses the current state of Data Sync and RDC with the expectation that Oracle will continue to enhance the capabilities of each over time.
The full list of available BICS data load options includes:
RDC is different from all of the options above in that data is not moved to the cloud; it remains on-premise and is available both to cloud applications like BICS, as well as existing reporting tools in the on-premise environment.
At the most basic level, BI Data Sync and RDC represent two ends of the spectrum in providing access to data in the cloud. Data Sync is used to push data from on-premise sources to a cloud database, while RDC is used to pull data from an on-premise source database into BICS visualizations.
Data Sync provides a full-featured data transfer tool with a client interface that allows for scheduling load jobs that efficiently move data from flat files, database tables, and other cloud data sources into the BICS Database Schema Service or Oracle Database Cloud Service. It can also directly load data as a data set source for the Visual Analyzer projects that are available in BICS. It includes many of the features found in other data loading tools: logging of load job execution steps, restarting after failures, incremental loading of new or modified data, and configuring the sequence of load operations.
Rather than moving data to the cloud, RDC enables secure connection to on-premise data sources for analysis and visualization. BICS RDC utilizes the BI Server Data Gateway running in the BI Cloud Service environment to provide secure access to on-premises data using private/public key pairs and SSL communication. The primary benefit of RDC is that it preserves the investment in the technology used to house and load on-premise data warehouses. It offers a hybrid approach to transitioning to a cloud-based analytics environment without having to also migrate the entire data environment as well.
The decision of whether to use Data Sync versus RDC would be based on a number of factors:
- Concerns over data security in the cloud
- Data volumes in the local data warehouse tables that might be difficult to transfer to the cloud in a timely manner
- Synchronization of data transmissions to the cloud with current load processes
- Ongoing investment in ETL tools, processes, and employees that would not be ready to transition completely to the cloud
A Data Sync solution would more likely lend itself either to new development of data load processes or to a more agile analytics environment that allowed for changing processes and data models more rapidly than would be possible with an on-premise database.
Regardless of where a BI environment is located, it truly is all about the data. And with the push to migrate more of the analytics functions into the cloud, it is necessary to determine the optimal strategy for using data visualization tools to access that data. Oracle provides many options to do this, whether it is the relatively simple process of configuring access to existing on-premise databases by using RDC or implementing a fully formed data loading strategy into the cloud using BI Data Sync. The capabilities and tradeoffs for each method should be reviewed thoroughly before proceeding with a cloud-based BI solution.