Data Source
In the extract phase of ETL, data is collected from various sources, which could include databases, APIs, web services, cloud storage, streaming platforms, and other data repositories.
Last updated
In the extract phase of ETL, data is collected from various sources, which could include databases, APIs, web services, cloud storage, streaming platforms, and other data repositories.
Last updated
The Extract of ETL starts with selecting the required Data Sources.
There are various Data Sources from which we can extract.
Object
On selecting an Object as Data source, it enables us to select the SObject from the list of SObjects (Standard / Custom) from our Org. It also asks for the fields from the selected SObject from which the data needs to be extracted. Note: Data Source Name can be customized.
Query
The Data Source as SOQL Enables us to write our custom SOQL Query which will query all the data from the current org.
This also has the capability to query the Standard and Custom Objects.
Rest Service
This Rest Service as a Data Source helps us to get the required data using the Rest Service Api which can be used further. Note : To use this functionality the user needs to Set up the current Org Named Credential following this the user needs to create the Custom Setting Record for App Settings and set the Api Name of the created Named Credential in the Current Org Named Credential option.
Apex Class
Data Source as Apex Class enables the feature to write custom logic in Apex Classes and get the return from the apex class.
The apex class drop down lists all those classes which implement the IICDataSourceGenerator class.In the class there must be a method named getData having two parameters (String recordId, String input) with a return type as String. Values to the parameters can be passed using the Parameters input.
For successful usage the class the above method should return a serialized Map.
Custom JSON
This feature helps to add Custom Json will be one of the Data Source in the Extract Part.
External Service
The External Data Source feature helps the ETL to extract the response data from an existing integration channel after making a callout. To make this possible the Integration Channel must be active. The Channel Name section lists out all the Integration channels.
Data Sources
This section of the Extract helps to select the priory created Data Sources.
As we wrap up our exploration of the Extract phase in this section, the spotlight now shifts to the transformative realm. Join us in the next section as we delve into the intricacies of the Transform stage, unraveling the power of data metamorphosis and its pivotal role in shaping a comprehensive ETL process.
