Sources (Code Config)

Last Updated: 2021-10-12

Now that you've added an App (see here if you haven't), your next step is to create a Source. A Source is a definition for how you pull data from an App into Grouparoo. See this doc to learn more about the core concepts in Grouparoo.

Generating a New Source

To generate a new Source for your Grouparoo application, run the generate command.

For example, let's say you have configured a Postgres App with the ID data_warehouse. You can generate a new Postgres Source (from a table in the database) like this:

grouparoo generate postgres:table:source users --parent postgres_app

Note here that users is the ID for the Source. An ID is always required when generating a config object.

This command will generate a file in your application directory at config/sources/users.js that looks something like this:

exports.default = async function buildConfig() {
  return [
    {
      class: "source",
      id: "users",
      name: "users",
      type: "postgres-table-import",
      appId: "...",
      options: {
        table: "...",
      },
      mapping: {
        id: "user_id",
      },
    },
  ];
};

The shape of the object(s) returned by the function will look different depending on the type of Source generated. More on this below.

List of Available Source Types

Postgres is just one type of Source you can create. To see a full list of available types, use the --list option, filtered by "source":

grouparoo generate source --list

This will give you something like the following:

bigquery:query:source (id, parent) - Config for a bigquery query Source. Work with multiple tables and build custom queries for its properties.
bigquery:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a bigquery table Source. Construct Sources and Properties without writing SQL.
csv:source (id, parent) - Config for a CSV Source
events:source (id, parent) - Config for a Grouparoo Source based on an Events App
google-sheets:source (id, parent) - Config for a Google Sheets Source
mailchimp:source (id, parent) - Config for a Mailchimp Source
manual:source (id, parent) - Config for a Grouparoo Source based on a Manual App
mysql:query:source (id, parent) - Config for a mysql query Source. Work with multiple tables and build custom queries for its properties.
mysql:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a mysql table Source. Construct Sources and Properties without writing SQL.
postgres:query:source (id, parent) - Config for a postgres query Source. Work with multiple tables and build custom queries for its properties.
postgres:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a postgres table Source. Construct Sources and Properties without writing SQL.
redshift:query:source (id, parent) - Config for a redshift query Source. Work with multiple tables and build custom queries for its properties.
redshift:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a redshift table Source. Construct Sources and Properties without writing SQL.
snowflake:query:source (id, parent) - Config for a snowflake query Source. Work with multiple tables and build custom queries for its properties.
snowflake:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a snowflake table Source. Construct Sources and Properties without writing SQL.

This list is determined by the plugins that you have installed. Learn more about Plugins.

Configuring Your Source

Each Source has a common set of options, while other options are specific to the type of Source. Here are the common options:

  • class: (required) Should be set to "source".
  • id: (required) The unique value that defines this Source.
  • name: (required) A display name for the Source. Will be set to the id by default.
  • type: (required) The type of Source will vary based on the Generator you're using. You likely don't want to change this value.
  • appId: (required) The id of the App (connection) that the Source uses.

The Generator makes a reasonable guess at the values in the config file it generated (in the config/sources directory). There are ellipses ("...") in the places in which it couldn't make a reasonable guess. In the example above, after generating a Postgres Source, all but appId and options.table were pre-populated.

Other sections may be commented out but require some action by you. The best practice when configuring a Source is to read the comments, keys, and values within the generated file and fill in the appropriate values for your Source.

Configuring Your First Source

The very first Source you create in Grouparoo will need a Property to use as a primary key for each Record. This must be unique to each of your Records, and is commonly a user ID or email address. Grouparoo will generate one Record per unique value for this Property.

Batch Source Generation

Certain plugins may provide batch source generation options. This means that is you have a valid App already configured and applied, Grouparoo can inspect that App and generate a Source and Properties automatically for you. Learn more about this from the Generate CLI Command documentation.

Adding a Schedule

A Schedule tells Grouparoo how frequently to check the Source for updated data and import it into the Application Database. This is the most common way to keep data fresh in Grouparoo.

Each Source config file has an example Schedule included with it as a separate object, commented out. Here are the options for a Schedule:

  • id: (required) Like the Source, a Schedule must have a unique id to identify it. This will be generated by default from the Source's id.
  • name: (required) A display name. It matches the id by default.
  • class: (required) This must be set to "schedule".
  • sourceId: (required) This must match the id of the Source. It will be set automatically.
  • recurring: (required, default: true) Should this Schedule run regularly?
  • recurringFrequency: (required) The length of the Schedule's interval. This is measured in ms. For example, 15 minutes would be 1000 * 60 * 15 or 900000.
  • options: These are the custom options that the specific Source type will bring.

Validating & Applying Your Config

You can validate your config at any time using the validate command:

$ grouparoo validate

And you can apply that config (save it to your Grouparoo application's database) using the apply command:

$ grouparoo apply