Sources (Community)

Last Updated: 2021-02-11

Now that you've added an App (see here if you haven't), your next step is to create a Source. A Source is a definition for how you pull data from an App into Grouparoo. See this doc to learn more about the core concepts in Grouparoo.

Generating a New Source

To generate a new Source for your Grouparoo application, run the generate command.

For example, let's say you have configured a Postgres App with the ID data_warehouse. You can generate a new Postgres Source (from a table in the database) like this:

grouparoo generate postgres:table:source users --parent postgres_app

Note here that users is the ID for the Source. An ID is always required when generating a config object.

This command will generate a file in your application directory at config/sources/users.js that looks something like this:

exports.default = async function buildConfig() {
  return [
    {
      class: "source",
      id: "users",
      name: "users",
      type: "postgres-table-import",
      appId: "...",
      options: {
        table: "...",
      },
      mapping: {
        id: "user_id",
      },
    },
  ];
};

The shape of the object(s) returned by the function will look different depending on the type of Source generated. More on this below.

List of Available Source Types

Postgres is just one type of Source you can create. To see a full list of available types, use the --list option, filtered by "source":

grouparoo generate source --list

This will give you something like the following:

bigquery:query:source (id, parent) - Config for a bigquery query Source. Work with multiple tables and build custom queries for its properties.
bigquery:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a bigquery table Source. Construct Sources and Properties without writing SQL.
csv:source (id, parent) - Config for a CSV Source
events:source (id, parent) - Config for a Grouparoo Source based on an Events App
google-sheets:source (id, parent) - Config for a Google Sheets Source
mailchimp:source (id, parent) - Config for a Mailchimp Source
manual:source (id, parent) - Config for a Grouparoo Source based on a Manual App
mysql:query:source (id, parent) - Config for a mysql query Source. Work with multiple tables and build custom queries for its properties.
mysql:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a mysql table Source. Construct Sources and Properties without writing SQL.
postgres:query:source (id, parent) - Config for a postgres query Source. Work with multiple tables and build custom queries for its properties.
postgres:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a postgres table Source. Construct Sources and Properties without writing SQL.
redshift:query:source (id, parent) - Config for a redshift query Source. Work with multiple tables and build custom queries for its properties.
redshift:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a redshift table Source. Construct Sources and Properties without writing SQL.
snowflake:query:source (id, parent) - Config for a snowflake query Source. Work with multiple tables and build custom queries for its properties.
snowflake:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a snowflake table Source. Construct Sources and Properties without writing SQL.

Configuring Your Source

The generate command attempts to make a reasonable guess at the values in the config file it generated (in the config/sources directory). There are ellipses ("...") in the places in which it couldn't make a reasonable guess. In the example above, after generating a Postgres Source, all but appId and options.table were pre-populated.

Other sections may be commented out but require some action by you. The best practice when configuring an Source is to read the comments, keys, and values within the generated file and fill in the appropriate values for your Source.

Batch Source Generation

Certain plugins may provide batch source generation options. This means that is you have a valid App already configured and applied, Grouparoo can inspect that app and generate a Source and Properties automatically for you. Learn more about this from the Generate CLI Command documentation.

Validating & Applying Your Config

You can validate your config at any time using the validate command:

grouparoo validate

And you can apply that config (save it to your Grouparoo application's database) using the apply command:

grouparoo apply

Note that apply will run validate, but it's recommended to run validate on its own first, just to be safe.