CLI: Configuration

grouparoo generate

$ grouparoo generate [template] [id]

Generate new code config files from templates. 

Options:
 -l, --list Display the available config templates to use with config-generate.  You can filter the list of templates by providing a string to match against, ie: `grouparoo generate app --list` to see templates which match "app"
 -d, --describe Display the options for the template in detail
 -p, --parent The id of the object that is the direct parent of this new object, e.g. the appId if you are creating a new Source.
 -f, --from For batch generators, where should we read the objects from?
 -w, --with For batch generators, what additional objects should we create? Use commas to separate names (--with "id,first_name,last_name") or "*" for all (`--with "*"`). Default is ``
 -m, --mapping For batch generators, how should we map this object? The remote key precedes the Grouparoo Property name. Use = to set the pair (--mapping "id=user_id").
 -H, --high-water-mark For batch generators, what should we use for the high-water-mark?
 -o, --overwrite Overwrite existing files?
 -h, --help display help for command

Example:
 App generation:
    grouparoo generate postgres:app data_warehouse

  Simple Source Generation (needs parent app):
    grouparoo generate postgres:table:source users_table \
      --parent data_warehouse

  Batch Source Generation (needs parent app to be applied first):
    grouparoo generate postgres:table:source users_table \
      --parent data_warehouse \
      --from users \
      --with id,first_name,email,last_name \
      --mapping 'id=user_id' \
      --high-water-mark updated_at
    

The generate command will create a template configuration file for you that you will then need to fill out.

grouparoo generate --list

There are many templates which you can use to generate from, depending on the plugins you have installed. You can use the --list flag to see available templates and filter the results. For example:

grouparoo generate --list postgres

Available Templates: matching "postgres"

  postgres:app (id) - Config for a Grouparoo postgres App
  postgres:destination (id, parent) - Config for a postgres Destination
  postgres:query:property (id, parent) - Config for a postgres Query Property
  postgres:query:source (id, parent) - Config for a postgres query Source. Work with multiple tables and build custom queries for its properties.
  postgres:table:property (id, parent) - Config for a postgres Table Property
  postgres:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a postgres table Source. Construct Sources and Properties without writing SQL.

You are required to provide and id to the generate command. This id will be used in the file, to set the filename, and may be used by you later. For example if you generated an app called data_warehouse, in your sources you would set appId: "data_warehouse" to indicate those sources rely on the data_warehouse app.

You can learn more about the configuration files here.

grouparoo generate --describe

You can use the --describe flag to see more information about the possible options you can use to generate a new object.

grouparoo generate postgres:table:source --describe

🦘 Grouparoo: generate postgres:table:source

Config for a postgres table Source. Construct Sources and Properties without writing SQL.

Required Inputs:
  * id (required) - The id of this new Source
  * parent (required) - The id of the postgres App to use for this Source, e.g: `--parent data_warehouse`

Optional Inputs:
  * from - The table to use for this source, e.g: `--from users`
  * high-water-mark - The name of the column to uses for this Source's Schedule, e.g: `--high-water-mark updated_at`.
  * mapping - The mapping between a column in this table and a Grouparoo Property, e.g: `--mapping "id=user_id"`
  * with - The names of the columns to create properties from, e.g: `--with users,id,first_name,last_name`. Defaults to '*'

Batch Generation

Certain plugins may provide additional generation options to help you generate many objects at once. For example, if you have a valid App already configured and applied, Grouparoo can inspect that App and generate a Source and Properties automatically for you. Batch generation is signified by using the --from flag, and potentially the --with, --mapping and --high-water-mark flags as well.

For example, the @grouparoo/postgres, @grouparoo/mysql,@grouparoo/redshift, @grouparoo/bigquery and @grouparoo/snowflake sources all support batch source generation:

# first, make an App
grouparoo generate postgres:app data_warehouse
# (fill out the App config)
grouparoo apply

# Then generate the Source, Schedule, and Properties all at once
grouparoo generate postgres:table:source users_table \
  --parent data_warehouse \
  --from users \ # The table to use
  --with id,first_name,email,last_name \  # The columns to make into properties (default *)
  --mapping 'id=user_id' \ # How the mapping should shake out + Bootstrapping if the column is in the `--with` list
  --high-water-mark updated_at

The above commands would create a Source, Schedule, and 4 Properties.

grouparoo validate

$ grouparoo validate

Validate your code config

Options:
 -l, --local Disable external validation
 -t, --timestamps Should timestamps be prepended to each log line?
 -h, --help display help for command

grouparoo apply

$ grouparoo apply

Apply changes from code config

Options:
 -l, --local Disable external validation
 -t, --timestamps Should timestamps be prepended to each log line?
 -h, --help display help for command