The YugabyteDB configuration parameters (GFLAGs and GUCs)

A database is a complex software with many layers especially when distributed, multiple instances. As with any software, there's a lot of configuration knobs, at many levels. People used to them, like database engineers, DBA, SRE may just mention their name thinking everybody knows what they are talking about. But for a developer, who doesn't spend her day tuning the knobs, it must be clear on how and where to do it.

In short, I'll describe the settings that concern:

• the whole YugabyteDB universe (a cluster of shared-nothing nodes connected through a public or private network). They are defined when starting the YugabyteDB processes (yb-master and yb-tserver) with some possibilities to update them later. Geeks reference them as 'G-Flags'.
• the PostgreSQL backend (which provides the YSQL API). They are defined, like in Postgres, with the SET command in your session, with many ways to define the defaults at higher level. Geeks reference them as 'GUCs'.

Here, I'll reference them as YugabyteDB setting (or GFLAG) and PostgreSQL setting (or GUC) and I may update this post with more questions coming.

PostgreSQL setting (or GUC)

In the PostgreSQL community, the PostgreSQL setting parameters are often called GUC, the name of the 'Grand Unified Configuration scheme' project that introduced them. They can be displayed with SHOW or by calling current_setting(), are visible from pg_settings. They can be set in a session with SET. The default value can also be set for a user with ALTER USER ... SETor for a database with ALTER DATABASE ... SET. The default values for a universe can also be set globally, as a comma separated list with the ysql_pg_conf_csv YugabyteDB setting (or GFLAG) but we will see that after explaining those.

As YugabyteDB reuses PostgreSQL you will find all PostgreSQL settings (or GUC) in YSQL but they are not all relevant to set. For example you will not set shared_buffers or WAL settings because YugabyteDB uses a distributed storage which has its own caching and logging.

There are also some additional settings specific to YugabyteDB. They often start with yb_ but, as both YugabyteDB and PostgreSQL are open source, you can also compare the guc.c definition. This is a quick awk script I use to compare PostgreSQL 11.2 with the master branch of YugabyteDB:

awk -F '"' '
/^static struct config_bool/ {
 struct=$0 
}
/^};/ {
 struct = "" ; guc="" 
}
/{"/ && struct=="static struct config_bool ConfigureNamesBool[] =" {
 guc=$2
}
/},/{
 guc=""
}
/gettext_noop/ && guc!="" {
 # record when first encountered
 if ( gucs[guc]=="" ) gucs[guc]=FILENAME
 desc[guc]=$2
}
END{
 for (i in gucs) if (length(i)>maxlen) maxlen=length(i)
 # list only first encountered in the last file
 for (i in gucs) if ( gucs[i]==FILENAME) printf "%-"maxlen"s  %-s\n", i ,desc[i]
}
' <(
# PostgreSQL
curl -s https://raw.githubusercontent.com/postgres/postgres/REL_11_STABLE/src/backend/utils/misc/guc.c
) <(
# YugabyteDB
curl -s https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/src/postgres/src/backend/utils/misc/guc.c
) | sort

This shows the settings that are specific to YugabyteDB:

force_global_transaction                      Forces use of global transaction table.
suppress_nonpg_logs                           Suppresses non-Postgres logs from appearing in the Postgres log file.
yb_binary_restore                             Enter a special mode designed specifically for YSQL binary restore.
yb_debug_log_catcache_events                  Log details for every catalog cache event such as a cache miss or cache invalidation/refresh.
yb_debug_log_docdb_requests                   Log the contents of all internal (protobuf) requests to DocDB.
yb_debug_log_internal_restarts                Log details for internal restarts such as read-restarts, cache-invalidation restarts, or txn restarts.
yb_debug_report_error_stacktrace              Append stacktrace information for error messages.
yb_disable_transactional_writes               Sets the boolean flag to disable transaction writes.
yb_enable_create_with_table_oid               Enables the ability to set table oids when creating tables or indexes.
yb_enable_expression_pushdown                 Push supported expressions down to DocDB for evaluation.
yb_enable_geolocation_costing                 Allow the optimizer to cost and choose between duplicate indexes based on locality
yb_enable_optimizer_statistics                Enables use postgres selectivity model.
yb_enable_upsert_mode                         Sets the boolean flag to enable or disable upsert mode for writes.
yb_format_funcs_include_yb_metadata           Include DocDB metadata (such as tablet splits) in formatting functions exporting system catalog information.
yb_non_ddl_txn_for_sys_tables_allowed         Enables the use of regular transactions for operating on system catalog tables in case a DDL transaction has not been started.
yb_planner_custom_plan_for_partition_pruning  If enabled, choose custom plan over generic plan
yb_read_from_followers                        Allow any statement that generates a read request to go to any node.
yb_test_fail_next_ddl                         When set, the next DDL (only CREATE TABLE for now)
yb_test_system_catalogs_creation              Relaxes some internal sanity checks for system catalogs to
ysql_upgrade_mode                             Enter a special mode designed specifically for YSQL cluster upgrades.

Always check the documentation or Yugabyte Community channels (https://www.yugabyte.com/community) as some settings are for specific situations only.

YugabyteDB setting (or GFLAG)

The above was limited to the YSQL layer. The settings that are global to all Yugabyte layers are set differently. You will find them referenced as "flags" like the yugabyted --master_flags and --tserver_flags and sometimes G-Flags:

You also find them in the /varz endpoint on the master and tserver.

The YugabyteDB settings (or GFLAGs) are set when starting the yb-master and yb-tserver. Some parameters are common to the master and tserver. Some make sense only for one of them. The reference documentation is the place to read about them. Some parameters can be changed dynamically with yb-ts-cli but the right way for persistent change is to set them at startup. And, this can be done while the application is running thanks to rolling restarts. You probably want to set the same value on all nodes, except the placement info ones, and the IP addresses.

ysql_pg_conf_csv

This is a YugabyteDB settings (or GFLAGs) that can set the default PostgreSQL setting (or GUC) by adding parameters to the ysql_pg.conf, the YugabyteDB equivalent of postgresql.conf. The value in ysql_pg_conf_csv declares them as Comma-Separated values. Then each name=value can be enclosed in double-quotes if, for example, there's a comma in the value. If they contain a double-quote, it must be doubled. And, as you put that in a bash command line, you may enclose all that with single quotes. Then, if there are single quotes in the value, I close the outer bash single quotes, and open double quotes to put the single quote not to be interpreted by bash...

Ok, here is an example that will add search_path = '"$user", public' in /yb/pg_data/ysql_pg.conf:

  yb-tserver --fs_data_dirs /yb --ysql_pg_conf_csv \
  '"search_path='"'"'""$user"",public'"'"'",yb_enable_expression_pushdown=on' 
# '             '" "'                '" "'   quotes for bash                '
#                 '                    '     single quotes (protected by double quotes)
#  "                                      "  doule quote protection for CSV value that contains commas
#                    ""     ""               doubled double-quotes (doubled as their are inside the CSV ones)

I have put, in comments, some explanations. But if you are not as fan as I am with escaping quotes in the command line, you can also put flags in file referenced by --flagfile. For example:

cat > flagfile.txt <<'CAT'
--ysql_pg_conf_csv="search_path='""$user"",public'",yb_enable_expression_pushdown=on
CAT

yb-tserver --fs_data_dirs /yb --flagfile flagfile.txt

From yugabyted, this might require additional escapes (the CSV list being enclosed between { and }), either in --tserver_flags or in the JSON configuration file. Be careful that { and } are interpreted by bash to expand a comma separated list and may have to be escaped.

To make it simpler, I use yugabyted to quick-start my labs. When I need more control on flags, I use a single --tserver_flags=flagfile=./tserver.flags set with flags in this file:

cat > ./tserver.flags <<'GFLAGS'
--ysql_enable_packed_row=true
--ysql_beta_features=true
--yb_enable_read_committed_isolation=true
--enable_deadlock_detection=true
--enable_wait_queues=true
--ysql_beta_feature_tablespace_alteration=true
--ysql_pg_conf_csv=log_statement='all',shared_preload_libraries="'auto_explain','pg_stat_monitor'"
GFLAGS

But if you want to put all in a single line, this should also work: --tserver_flags="ysql_enable_packed_row=true,ysql_beta_features=true,yb_enable_read_committed_isolation=true,enable_deadlock_detection=true,enable_wait_queues=true,ysql_beta_feature_tablespace_alteration=true,ysql_pg_conf_csv=\{log_statement='all',shared_preload_libraries="'auto_explain','pg_stat_monitor'"}\"

If you find this too complex, yes, it is. Escaping special characters, including those used to escape, has always been a mess—except with PostgreSQL strings. I would like to see any software use a PostgreSQL dollar-quoted string.

yugabyted vs. yb-master and yb-tserver flags

yugabyted is a wrapper on top of yb-master and yb-tserver and both have flags, and you can set both from yugabyted

• --config is where you can put yugabyted flags instead of as command line arguments https://docs.yugabyte.com/preview/reference/configuration/yugabyted/#advanced-flags
• two of those command line arguments are --master_flags and --tserver_flags which are used to pass yb-master and yb-tserver additional flags (only for those that are not set by yugabyted itself). They can be listed there in comma-separated list, and using {} for those which have their own comma-separated list. Or they can be a single flagfile= that reference a file with the yb-master or yb-tserver flags https://docs.yugabyte.com/preview/reference/configuration/yb-tserver/#flagfile