CLI: Configuration
Last Updated: 2021-02-08grouparoo generate
$ grouparoo generate [template] [id]
Generate new code config files from templates.
Options:
-l, --list Display the available config templates to use with config-generate. You can filter the list of templates by providing a string to match against, ie: `grouparoo generate app --list` to see templates which match "app"
-d, --describe Display the options for the template in detail
-p, --parent The id of the object that is the direct parent of this new object, e.g. the appId if you are creating a new Source.
-f, --from For batch generators, where should we read the objects from?
-w, --with For batch generators, what additional objects should we create? Use commas to separate names (--with "id,first_name,last_name") or "*" for all (`--with "*"`). Default is ``
-m, --mapping For batch generators, how should we map this object? The remote key precedes the Grouparoo Property name. Use = to set the pair (--mapping "id=user_id").
-H, --high-water-mark For batch generators, what should we use for the high-water-mark?
-o, --overwrite Overwrite existing files?
-h, --help display help for command
Example:
App generation:
grouparoo generate postgres:app data_warehouse
Simple Source Generation (needs parent app):
grouparoo generate postgres:table:source users_table \
--parent data_warehouse
Batch Source Generation (needs parent app to be applied first):
grouparoo generate postgres:table:source users_table \
--parent data_warehouse \
--from users \
--with id,first_name,email,last_name \
--mapping 'id=user_id' \
--high-water-mark updated_at
The generate command will create a template configuration file for you that you will then need to fill out.
grouparoo generate --list
There are many templates which you can use to generate from, depending on the plugins you have installed. You can use the --list
flag to see available templates and filter the results. For example:
grouparoo generate --list postgres
Available Templates: matching "postgres"
postgres:app (id) - Config for a Grouparoo postgres App
postgres:destination (id, parent) - Config for a postgres Destination
postgres:query:property (id, parent) - Config for a postgres Query Property
postgres:query:source (id, parent) - Config for a postgres query Source. Work with multiple tables and build custom queries for its properties.
postgres:table:property (id, parent) - Config for a postgres Table Property
postgres:table:source (id, parent, from, mapping, with, high-water-mark) - Config for a postgres table Source. Construct Sources and Properties without writing SQL.
You are required to provide and id
to the generate command. This id
will be used in the file, to set the filename, and may be used by you later. For example if you generated an app called data_warehouse
, in your sources you would set appId: "data_warehouse"
to indicate those sources rely on the data_warehouse
app.
You can learn more about the configuration files here.
grouparoo generate --describe
You can use the --describe flag
to see more information about the possible options you can use to generate a new object.
grouparoo generate postgres:table:source --describe
🦘 Grouparoo: generate postgres:table:source
Config for a postgres table Source. Construct Sources and Properties without writing SQL.
Required Inputs:
* id (required) - The id of this new Source
* parent (required) - The id of the postgres App to use for this Source, e.g: `--parent data_warehouse`
Optional Inputs:
* from - The table to use for this source, e.g: `--from users`
* high-water-mark - The name of the column to uses for this Source's Schedule, e.g: `--high-water-mark updated_at`.
* mapping - The mapping between a column in this table and a Grouparoo Property, e.g: `--mapping "id=user_id"`
* with - The names of the columns to create properties from, e.g: `--with users,id,first_name,last_name`. Defaults to '*'
Batch Generation
Certain plugins may provide additional generation options to help you generate many objects at once. For example, if you have a valid App
already configured and applied, Grouparoo can inspect that App and generate a Source and Properties automatically for you. Batch generation is signified by using the --from
flag, and potentially the --with
, --mapping
and --high-water-mark
flags as well.
For example, the @grouparoo/postgres
, @grouparoo/mysql
,@grouparoo/redshift
, @grouparoo/bigquery
and @grouparoo/snowflake
sources all support batch source generation:
# first, make an App
grouparoo generate postgres:app data_warehouse
# (fill out the App config)
grouparoo apply
# Then generate the Source, Schedule, and Properties all at once
grouparoo generate postgres:table:source users_table \
--parent data_warehouse \
--from users \ # The table to use
--with id,first_name,email,last_name \ # The columns to make into properties (default *)
--mapping 'id=user_id' \ # How the mapping should shake out + Bootstrapping if the column is in the `--with` list
--high-water-mark updated_at
The above commands would create a Source, Schedule, and 4 Properties.
grouparoo validate
$ grouparoo validate
Validate your code config
Options:
-l, --local Disable external validation
-t, --timestamps Should timestamps be prepended to each log line?
-h, --help display help for command
grouparoo apply
$ grouparoo apply
Apply changes from code config
Options:
-l, --local Disable external validation
-t, --timestamps Should timestamps be prepended to each log line?
-h, --help display help for command
Having Problems?
If you are having trouble, visit the list of common issues or open a Github issue to get support.