Verteego Doc
  • Getting started
    • About Verteego
    • Sample Use Cases
    • Concepts
  • Data
    • Introduction
    • Datasources
      • URL connector specification
    • Datasets
  • Pipelines
    • Forecasting Pipelines
      • Getting started
      • Configuration
        • Identifying and preparing data
          • calculated_cols
          • cols_type
          • date_col
          • normalize
          • preprocessing
          • prediction_resolution
        • Configuring the Forecasting Algorithm
          • algo_name
          • algo_type
          • algorithm_parameters
          • fit_parameters
          • conditional_algorithm_parameters
        • Building the Training and Prediction Set
          • column_to_predict
          • features
          • input_prediction_columns
        • Using Hyperparameter Tuning for the Model
          • tuning_search_params
          • hyperparameter_tuning_parameters
        • Evaluating the Results of the Forecast
          • scores
        • Modifying the results of the forecast
          • postprocessing
      • Calculators
        • External source
          • get_from_dataset
          • weather
        • Mathematic
          • aggregate_val_group_by_key
          • binary_operation
          • count_rows_by_keys
          • hierarchical_aggregate
          • mathematical_expression
          • unary_operation
          • Moving Average (EWM)
        • Machine Learning
          • pca
          • clustering
          • glmm_encoder
          • one_hot_encode
          • words_similarity
        • Transformation
          • fillna
          • fill_series
          • case_na
          • interval_index
          • constant
          • cyclic
          • replace
        • Temporal
          • bank_holidays_countdown
          • bankholidays
          • date_attributes
          • date_weight
          • day_count
          • duration
          • events_countdown
          • seasonality
          • tsfresh
    • Optimization Pipelines
      • Getting started
      • Configuration
      • Constraints
        • Unary Constraint
        • Binary Constraint
        • Aggregation Constraint
        • Order Constraint
        • Multiplicative Equality Constraint
        • Generate constraints from a Dataset
  • Apps
    • About Apps
    • Recipes
      • Pipelines
      • Datasets
  • Users
    • User roles
  • Best practices
    • Performance analysis and ML model improvement
  • Developers
    • API
    • Change logs
Powered by GitBook
On this page
  1. Data

Datasources

Data Sources and Connectors Overview

In Verteego, Data Sources and Connectors facilitate the seamless import and export of data to and from the platform. By establishing these connections, you can integrate data from various external systems and databases, allowing Verteego to enhance decision-making with real-time insights and accurate data analysis.

Supported Data Sources

The following data sources are supported, each serving different types of data connections and integration methods:

  1. Google Storage (GCS) (Default)

    • This connector enables the integration of data stored in Google Cloud Storage. It is the default method for managing large datasets in the cloud.

    • You must ensure that your project service account can access the bucket.

  2. Google BigQuery (BQ)

    • A connector for Google BigQuery, a fully-managed, serverless data warehouse that enables real-time analytics. BigQuery allows for efficient querying of vast datasets directly from Verteego.

    • You must ensure that your project service account can access the table.

  3. Google Drive (GD)

    • Google Drive integration allows the platform to access and sync data from files stored in Google Drive, supporting common file formats like CSV, XLSX, and more.

    • You must ensure that your project service account can access the table.

  4. PostgreSQL (PG)

    • A relational database connector that facilitates the exchange of data between PostgreSQL databases and Verteego, ideal for transactional systems and structured data.

    • Password and login information are required, as well as IP whitelisting.

  5. Snowflake (SF)

    • Snowflake’s connector integrates with Verteego’s platform to provide seamless access to cloud-based data warehouses. It supports the efficient handling of semi-structured and structured data for analysis.

    • Password and login information are required, as well as IP whitelisting.

  6. URL / API (URL)

    • This connector is designed for integrating data from external web services or APIs. It supports RESTful APIs, allowing easy import of data through custom integrations.

    • The API must respond with either a status, or a parquet file.

    • More details on payloads and responses available here.

  7. Adoria API (AD)

    • Adoria API integration allows you to pull data from Adoria's suite of services, typically used for data related to restaurants.

  8. FTP Connector (FTP)

    • This connector facilitates the transfer of files from an FTP server to Verteego. It is useful for accessing historical data or data provided by external systems in a file-based format.

    • SSH and SFTP connection are supported.

  9. MySQL / MariaDB (MySQL)

    • These relational database connectors support integration with MySQL and MariaDB, allowing the platform to interact with these databases for both import and export of data.

    • Password and login information are required, as well as IP whitelisting.

Setup and Configuration of Data Sources

To set up a connector, follow these general steps:

  1. Create a Data Source:

    • Each connector is set up by creating a corresponding Data Source within the Verteego platform.

    • During setup, you will specify any parameter required to retrieve the data (table names, path, queries, ...).

  2. Establish the Connection:

    • For each data source, provide the necessary credentials, API keys, or connection strings to authenticate and allow data access.

  3. Use the Data Source:

    • After the Data Source is configured, it can be used in various ways:

      • Create Datasets: Once the connection is established, you can create Datasets from the imported data, which can then be analyzed, visualized, or used to generate insights.

      • Run Exports: Data can also be exported back to the connected systems or external platforms as needed.

Key Benefits of Data Sources and Connectors

  • Flexibility: Verteego supports a variety of data sources, catering to different technologies, databases, and cloud storage solutions.

  • Real-Time Access: With direct integrations, you can ensure that the data flowing into Verteego is always up-to-date.

  • Automation: Once set up, these connections can run automatically on schedules or trigger events for consistent data management.

  • Data Transparency: By using the correct Data Source for your dataset, you ensure that the integrity and structure of your data are preserved throughout the integration process.

  • Versioning: Extracted data are stored, timestamps and versioned.

PreviousIntroductionNextURL connector specification

Last updated 1 month ago