Configuring Apache Kudu

Configure Kudu

Configuration Basics

To configure the behavior of each Kudu process, you can pass command-line flags when you start it, or read those options from configuration files by passing them using one or more --flagfile=<file> options. You can even include the --flagfile option within your configuration file to include other files. Learn more about gflags by reading its documentation.

You can place options for masters and tablet servers into the same configuration file, and each will ignore options that do not apply.

Flags can be prefixed with either one or two - characters. This documentation standardizes on two: --example_flag.

Discovering Configuration Options

Only the most common configuration options are documented here. For a more exhaustive list of configuration options, see the Configuration Reference.

To see all configuration flags for a given executable, run it with the --help option. Take care when configuring undocumented flags, as not every possible configuration has been tested, and undocumented options are not guaranteed to be maintained in future releases.

Directory Configurations

Every Kudu node requires the specification of directory flags. The --fs_wal_dir configuration indicates where Kudu will place its write-ahead logs. The --fs_metadata_dir configuration indicates where Kudu will place metadata for each tablet. It is recommended, although not necessary, that these directories be placed on a high-performance drives with high bandwidth and low latency, e.g. solid-state drives. If --fs_metadata_dir is not specified, metadata will be placed in the directory specified by --fs_wal_dir. Since a Kudu node cannot tolerate the loss of its WAL or metadata directories, it may be wise to mirror the drives containing these directories in order to make recovering from a drive failure easier; however, mirroring may increase the latency of Kudu writes.

The --fs_data_dirs configuration indicates where Kudu will write its data blocks. This is a comma-separated list of directories; if multiple values are specified, data will be striped across the directories. If not specified, data blocks will be placed in the directory specified by --fs_wal_dir. Note that while a single data directory backed by a RAID-0 array will outperform a single data directory backed by a single storage device, it is better to let Kudu manage its own striping over multiple devices rather than delegating the striping to a RAID-0 array.

Additionally, --fs_wal_dir and --fs_metadata_dir may be the same as one of the directories listed in --fs_data_dirs, but must not be sub-directories of any of them.

Each directory specified by a configuration flag on a given machine should be used by at most one Kudu process. If multiple Kudu processes on the same machine are configured to use the same directory, Kudu may refuse to start up.
Once --fs_data_dirs is set, extra tooling is required to change it. For more details, see the Kudu Administration docs.
The --fs_wal_dir and --fs_metadata_dir configurations can be changed, provided the contents of the directories are also moved to match the flags.

Configuring the Kudu Master

To see all available configuration options for the kudu-master executable, run it with the --help option:

$ kudu-master --help
Table 1. Supported Configuration Flags for Kudu Masters
Flag Valid Options Default Description

--master_addresses

string

localhost

Comma-separated list of all the RPC addresses for Master consensus-configuration. If not specified, assumes a standalone Master.

--fs_data_dirs

string

List of directories where the Master will place its data blocks.

--fs_metadata_dir

string

The directory where the Master will place its tablet metadata.

--fs_wal_dir

string

The directory where the Master will place its write-ahead logs.

--log_dir

string

/tmp

The directory to store Master log files.

For the full list of flags for masters, see the Kudu Master Configuration Reference.

Configuring Tablet Servers

To see all available configuration options for the kudu-tserver executable, run it with the --help option:

$ kudu-tserver --help
Table 2. Supported Configuration Flags for Kudu Tablet Servers
Flag Valid Options Default Description

--fs_data_dirs

string

List of directories where the Tablet Server will place its data blocks.

--fs_metadata_dir

string

The directory where the Tablet Server will place its tablet metadata.

--fs_wal_dir

string

The directory where the Tablet Server will place its write-ahead logs.

--log_dir

string

/tmp

The directory to store Tablet Server log files

--tserver_master_addrs

string

127.0.0.1:7051

Comma separated addresses of the masters which the tablet server should connect to. The masters do not read this flag.

--block_cache_capacity_mb

integer

512

Maximum amount of memory allocated to the Kudu Tablet Server’s block cache.

--memory_limit_hard_bytes

integer

4294967296

Maximum amount of memory a Tablet Server can consume before it starts rejecting all incoming writes.

For the full list of flags for tablet servers, see the Kudu Tablet Server Configuration Reference.

Configure Kudu Tables

Kudu allows certain configurations to be set per table. To configure the behavior of a Kudu table, you can set these configurations at table creation, or alter them via the Kudu API or Kudu command line tool.

Table 3. Supported Configurable Properties for Kudu Tables
Configuration Valid Options Default Description

kudu.table.history_max_age_sec

integer

Number of seconds to retain history for tablets in this table.