Apache Kudu Release Notes

Introduction

If you are new to Kudu, check out its list of features and benefits.

Release notes specific to 0.10.0

Kudu 0.10.0 delivers a number of new features, bug fixes, and optimizations, detailed below.

Kudu 0.10.0 maintains wire-compatibility with previous releases, meaning that applications using the Kudu client libraries may be upgraded either before, at the same time, or after the Kudu servers. However, if you begin using new features of Kudu 0.10.0 such as manually range-partitioned tables, you must first upgrade all clients to this release.

This release does not maintain full Java API or ABI compatibility with Kudu 0.9.x due to a package rename and some other small changes. See below for details.

To upgrade to Kudu 0.10.0, see Upgrading from 0.9.x to 0.10.0.

Incompatible changes and deprecated APIs in 0.10.0

  • Gerrit #3737 The Java client has been repackaged under org.apache.kudu instead of org.kududb. Import statements for Kudu classes must be modified in order to compile against 0.10.0. Wire compatibility is maintained.

  • Gerrit #3055 The Java client’s synchronous API methods now throw KuduException instead of Exception. Existing code that catches Exception should still compile, but introspection of an exception’s message may be impacted. This change was made to allow thrown exceptions to be queried more easily using KuduException.getStatus and calling one of Status’s methods. For example, an operation that tries to delete a table that doesn’t exist would return a `Status that returns true when queried on isNotFound().

  • The Java client’s KuduTable.getTabletsLocations set of methods is now deprecated. Additionally, they now take an exclusive end partition key instead of an inclusive key. Applications are encouraged to use the scan tokens API instead of these methods in the future.

  • The C++ API for specifying split points on range-partitioned tables has been improved to make it easier for callers to properly manage the ownership of the provided rows.

    The TableCreator::split_rows API took a vector<const KuduPartialRow*>, which made it very difficult for the calling application to do proper error handling with cleanup when setting the fields of the KuduPartialRow. This API has been now been deprecated and replaced by a new method TableCreator::add_range_split which allows easier use of smart pointers for safe memory management.

  • The Java client’s internal buffering has been reworked. Previously, the number of buffered write operations was constrained on a per-tablet-server basis. Now, the configured maximum buffer size constrains the total number of buffered operations across all tablet servers in the cluster. This provides a more consistent bound on the memory usage of the client regardless of the size of the cluster to which it is writing.

    This change can negatively affect the write performance of Java clients which rely on buffered writes. Consider using the setMutationBufferSpace API to increase a session’s maximum buffer size if write performance seems to be degraded after upgrading to Kudu 0.10.0.

  • The "remote bootstrap" process used to copy a tablet replica from one host to another has been renamed to "Tablet Copy". This resulted in the renaming of several RPC metrics. Any users previously explicitly fetching or monitoring metrics related to Remote Bootstrap should update their scripts to reflect the new names.

  • The SparkSQL datasource for Kudu no longer supports mode Overwrite. Users should use the new KuduContext.upsertRows method instead. Additionally, inserts using the datasource are now upserts by default. The older behavior can be restored by setting the operation parameter to insert.

New features

  • Users may now manually manage the partitioning of a range-partitioned table. When a table is created, the user may specify a set of range partitions that do not cover the entire available key space. A user may add or drop range partitions to existing tables.

    This feature can be particularly helpful with time series workloads in which new partitions can be created on an hourly or daily basis. Old partitions may be efficiently dropped if the application does not need to retain historical data past a certain point.

    This feature is considered experimental for the 0.10 release. More details of the new feature can be found in the accompanying blog post.

  • Support for running Kudu clusters with multiple masters has been stabilized. Users may start a cluster with three or five masters to provide fault tolerance despite a failure of one or two masters, respectively.

    Note that certain tools (e.g. ksck) are still lacking complete support for multiple masters. These deficiencies will be addressed in a following release.

  • Kudu now supports the ability to reserve a certain amount of free disk space in each of its configured data directories. If a directory’s free disk space drops to less than the configured minimum, Kudu will stop writing to that directory until space becomes available. If no space is available in any configured directory, Kudu will abort.

    This feature may be configured using the fs_data_dirs_reserved_bytes and fs_wal_dir_reserved_bytes flags.

  • The Spark integration’s KuduContext now supports four new methods for writing to Kudu tables: insertRows, upsertRows, updateRows, and deleteRows. These are now the preferred way to write to Kudu tables from Spark.

Improvements and optimizations

  • KUDU-1516 The kudu-ksck tool has been improved and now detects problems such as when a tablet does not have a majority of replicas on live tablet servers, or if those replicas aren’t in a good state. Users who currently depend on the tool to detect inconsistencies may now see failures when before they wouldn’t see any.

  • Gerrit #3477 The way operations are buffered in the Java client has been reworked. Previously, the session’s buffer size was set per tablet, meaning that a buffer size of 1,000 for 10 tablets being written to allowed for 10,000 operations to be buffered at the same time. With this change, all the tablets share one buffer, so users might need to set a bigger buffer size in order to reach the same level of performance as before.

  • Gerrit #3674 Added LESS and GREATER options for column predicates.

  • KUDU-1444 added support for passing back basic per-scan metrics (e.g cache hit rate) from the server to the C++ client. See the KuduScanner::GetResourceMetrics() API for detailed usage. This feature will be supported in the Java client API in a future release.

  • KUDU-1446 improved the order in which the tablet server evaluates predicates, so that predicates on smaller columns are evaluated first. This may improve performance on queries which apply predicates on multiple columns of different sizes.

  • KUDU-1398 improved the storage efficiency of Kudu’s internal primary key indexes. This optimization should decrease space usage and improve random access performance, particularly for workloads with lengthy primary keys.

Fixed Issues

  • Gerrit #3541 Fixed a problem in the Java client whereby an RPC could be dropped when a connection to a tablet server or master was forcefully closed on the server-side while RPCs to that server were in the process of being encoded. The effect was that the RPC would not be sent, and users of the synchronous API would receive a TimeoutException. Several other Java client bugs which could cause similar spurious timeouts were also fixed in this release.

  • Gerrit #3724 Fixed a problem in the Java client whereby an RPC could be dropped when a socket timeout was fired while that RPC was being sent to a tablet server or master. This would manifest itself in the same way Gerrit #3541.

  • KUDU-1538 fixed a bug in which recycled block identifiers could cause the tablet server to lose data. Following this bug fix, block identifiers will no longer be reused.

Other noteworthy changes

  • This is the first release of Apache Kudu as a top-level (non-incubating) project!

  • The default false positive rate for Bloom filters has been changed from 1% to 0.01%. This will increase the space consumption of Bloom filters by a factor of two (from approximately 10 bits per row to approximately 20 bits per row). This is expected to substantially improve the performance of random-write workloads at the cost of an incremental increase in disk space usage.

  • The Kudu C++ client library now has Doxygen-based API documentation available online.

  • Kudu now uses the Raft consensus algorithm even for unreplicated tables. This change simplifies code and will also allow administrators to enable replication on a previously-unreplicated table. This change is internal and should not be visible to users.

Upgrading from 0.9.x to 0.10.0

To upgrade from Kudu 0.9.x to Kudu 0.10.0, perform the following high-level steps, which are detailed in the installation guide under Upgrade Procedure:

  1. Shut down all Kudu services.

  2. Install the new Kudu packages or parcels, or install Kudu 0.10.0 from source.

  3. Restart all Kudu services.

Rolling upgrades are not supported when upgrading from Kudu 0.9.x to 0.10.0 and they are known to cause errors in this release. If you run into a problem after an accidental rolling upgrade, shut down all services and then restart all services and the system should come up properly.
For the duration of the Kudu Beta, upgrade instructions are generally only given for going from the previous latest version to the newly released version.

Downgrading from 0.10.0 to 0.9.x

After upgrading to Kudu 0.10.0, it is possible to downgrade to 0.9.x with the following exceptions:

  1. Tables created in 0.10.0 will not be accessible after a downgrade to 0.9.x

  2. A multi-master setup formatted in 0.10.0 may not be downgraded to 0.9.x

Release notes specific to 0.9.1

Kudu 0.9.1 delivers incremental bug fixes over Kudu 0.9.0. It is fully compatible with Kudu 0.9.0.

Upgrading from 0.9.0 to 0.9.1

Before upgrading to Kudu 0.9.1 from Kudu 0.8.0, please read the Release notes specific to 0.9.0.

Upgrading from 0.8.0 or 0.9.0 to 0.9.1 is supported. To upgrade from Kudu 0.8.0 or Kudu 0.9.0 to Kudu 0.9.1, use the procedure documented in Upgrading from 0.8.0 to 0.9.x.

For the duration of the Kudu Beta, upgrade instructions are generally only given for going from the previous latest version to the newly released version.

Fixed Issues

  • KUDU-1469 fixed a bug in our Raft consensus implementation that could cause a tablet to stop making progress after a leader election.

  • Gerrit #3456 fixed a bug in which servers under high load could store metric information in incorrect memory locations, causing crashes or data corruption.

  • Gerrit #3457 fixed a bug in which errors from the Java client would carry an incorrect error message.

  • Several other small bug fixes were backported to improve stability.

Release notes specific to 0.9.0

Kudu 0.9.0 delivers incremental features, improvements, and bug fixes over the previous versions.

To upgrade to Kudu 0.10.0, see Upgrading from 0.8.0 to 0.9.x.

Incompatible changes

  • The KuduTableInputFormat command has changed the way in which it handles scan predicates, including how it serializes predicates to the job configuration object. The new configuration key is kudu.mapreduce.encoded.predicate. Clients using the TableInputFormatConfigurator are not affected.

  • The kudu-spark sub-project has been renamed to follow naming conventions for Scala. The new name is kudu-spark_2.10.

  • Default table partitioning has been removed. All tables must now be created with explicit partitioning. Existing tables are unaffected. See the schema design guide for more details.

New features

  • KUDU-1002 Added support for UPSERT operations, whereby a row is inserted if it does not already exist, but updated if it does. Support for UPSERT is included in Java, C++, and Python APIs, but not in Impala.

  • KUDU-1306 Scan token API for creating partition-aware scan descriptors. This API simplifies executing parallel scans for clients and query engines.

  • Gerrit 2848 Added a kudu datasource for Spark. This datasource uses the Kudu client directly instead of using the MapReduce API. Predicate pushdowns for spark-sql and Spark filters are included, as well as parallel retrieval for multiple tablets and column projections. See an example of Kudu integration with Spark.

  • Gerrit 2992 Added the ability to update and insert from Spark using a Kudu datasource.

Improvements

  • KUDU-1415 Added statistics in the Java client such as the number of bytes written and the number of operations applied.

  • KUDU-1451 Improved tablet server restart time when the tablet server needs to clean up of a lot previously deleted tablets. Tablets are now cleaned up after they are deleted.

Fixed Issues

  • KUDU-678 Fixed a leak that happened during DiskRowSet compactions where tiny blocks were still written to disk even if there were no REDO records. With the default block manager, it usually resulted in block containers with thousands of tiny blocks.

  • KUDU-1437 Fixed a data corruption issue that occured after compacting sequences of negative INT32 values in a column that was configured with RLE encoding.

Other noteworthy changes

All Kudu clients have longer default timeout values, as listed below.

Java
  • The default operation timeout and the default admin operation timeout are now set to 30 seconds instead of 10.

  • The default socket read timeout is now 10 seconds instead of 5.

C++
  • The default admin timeout is now 30 seconds instead of 10.

  • The default RPC timeout is now 10 seconds instead of 5.

  • The default scan timeout is now 30 seconds instead of 15.

  • Some default settings related to I/O behavior during flushes and compactions have been changed: The default for flush_threshold_mb has been increased from 64MB to 1000MB. The default cfile_do_on_finish has been changed from close to flush. Experiments using YCSB indicate that these values will provide better throughput for write-heavy applications on typical server hardware.

Upgrading from 0.8.0 to 0.9.x

Before upgrading, see Incompatible changes and Client compatibility. To upgrade from Kudu 0.8.0 to 0.9.0, perform the following high-level steps, which are detailed in the installation guide under Upgrade Procedure:

  1. Shut down all Kudu services.

  2. Install the new Kudu packages or parcels, or install Kudu 0.9.1 from source.

  3. Restart all Kudu services.

It is technically possible to upgrade Kudu using rolling restarts, but it has not been tested and is not recommended.

For the duration of the Kudu Beta, upgrade instructions are only given for going from the previous latest version to the newest.

Client compatibility

Masters and tablet servers should be upgraded before clients are upgraded. For specific information about client compatibility, see the Incompatible changes section.

Release notes specific to 0.8.0

Kudu 0.8.0 delivers incremental features, improvements, and bug fixes over the previous versions.

To upgrade to Kudu 0.8.0, see Upgrade from 0.7.1 to 0.8.0.

Incompatible changes

  • 0.8.0 clients are not fully compatible with servers running Kudu 0.7.1 or lower. In particular, scans that specify column predicates will fail. To work around this issue, upgrade all Kudu servers before upgrading clients.

New features

  • KUDU-431 A simple Flume sink has been implemented.

Improvements

  • KUDU-839 Java RowError now uses an enum error code.

  • Gerrit 2138 The handling of column predicates has been re-implemented in the server and clients.

  • KUDU-1379 Partition pruning has been implemented for C++ clients (but not yet for the Java client). This feature allows you to avoid reading a tablet if you know it does not serve the row keys you are querying.

  • Gerrit 2641 Kudu now uses earliest-deadline-first RPC scheduling and rejection. This changes the behavior of the RPC service queue to prevent unfairness when processing a backlog of RPC threads and to increase the likelihood that an RPC will be processed before it can time out.

Fixed Issues

  • KUDU-1337 Tablets from tables that were deleted might be unnecessarily re-bootstrapped when the leader gets the notification to delete itself after the replicas do.

  • KUDU-969 If a tablet server shuts down while compacting a rowset and receiving updates for it, it might immediately crash upon restart while bootstrapping that rowset’s tablet.

  • KUDU-1354 Due to a bug in Kudu’s MVCC implementation where row locks were released before the MVCC commit happened, flushed data would include out-of-order transactions, triggering a crash on the next compaction.

  • KUDU-1322 The C++ client now retries write operations if the tablet it is trying to reach has already been deleted.

  • Gerrit 2571 Due to a bug in the Java client, users were unable to close the kudu-spark shell because of lingering non-daemon threads.

Other noteworthy changes

  • Gerrit 2239 The concept of "feature flags" was introduced in order to manage compatibility between different Kudu versions. One case where this is helpful is if a newer client attempts to use a feature unsupported by the currently-running tablet server. Rather than receiving a cryptic error, the user gets an error message that is easier to interpret. This is an internal change for Kudu system developers and requires no action by users of the clients or API.

Release notes specific to 0.7.1

Kudu 0.7.1 is a bug fix release for 0.7.0.

Fixed Issues

  • KUDU-1325 fixes a tablet server crash that could occur during table deletion. In some cases, while a table was being deleted, other replicas would attempt to re-replicate tablets to servers that had already processed the deletion. This could trigger a race condition that caused a crash.

  • KUDU-1341 fixes a potential data corruption and crash that could happen shortly after tablet server restarts in workloads that repeatedly delete and re-insert rows with the same primary key. In most cases, this corruption affected only a single replica and could be repaired by re-replicating from another.

  • KUDU-1343 fixes a bug in the Java client that occurs when a scanner has to scan multiple batches from one tablet and then start scanning from another. In particular, this would affect any scans using the Java client that read large numbers of rows from multi-tablet tables.

  • KUDU-1345 fixes a bug where in some cases the hybrid clock could jump backwards, resulting in a crash followed by an inability to restart the affected tablet server.

  • KUDU-1360 fixes a bug in the kudu-spark module which prevented reading rows with NULL values.

Release notes specific to 0.7.0

Kudu 0.7.0 is the first release done as part of the Apache Incubator and includes a number of changes, new features, improvements, and fixes.

The upgrade instructions can be found at Upgrade from 0.6.0 to 0.7.0.

Incompatible changes

  • The C++ client includes a new API, KuduScanBatch, which performs better when a large number of small rows are returned in a batch. The old API of vector<KuduRowResult> is deprecated.

    This change is API-compatible but not ABI-compatible.
  • The default replication factor has been changed from 1 to 3. Existing tables will continue to use the replication factor they were created with. Applications that create tables may not work properly if they assume a replication factor of 1 and fewer than 3 replicas are available. To use the previous default replication factor, start the master with the configuration flag --default_num_replicas=1.

  • The Python client has been completely rewritten, with a focus on improving code quality and testing. The read path (scanners) has been improved by adding many of the features already supported by the C++ and Java clients. The Python client is no longer considered experimental.

New features

  • With the goal of Spark integration in mind, a new kuduRDD API has been added, which wraps newAPIHadoopRDD and includes a default source for Spark SQL.

Improvements

  • The Java client includes new methods countPendingErrors() and getPendingErrors() on KuduSession. These methods allow you to count and retrieve outstanding row errors when configuring sessions with AUTO_FLUSH_BACKGROUND.

  • New server-level metrics allow you to monitor CPU usage and context switching.

  • Kudu now builds on RHEL 7, CentOS 7, and SLES 12. Extra instructions are included for SLES 12.

Fixed Issues

  • KUDU-1288 fixes a severe file descriptor leak, which could previously only be resolved by restarting the tablet server.

  • KUDU-1250 fixes a hang in the Java client when processing an in-flight batch and the previous batch encountered an error.

Other noteworthy changes

  • The file block manager’s performance was improved, but it is still not recommended for real-world use.

  • The master now attempts to spread tablets more evenly across the cluster during table creation. This has no impact on existing tables, but will improve the speed at which under-replicated tabletsare re-replicated after a tablet server failure.

  • All licensing documents have been modified to adhere to ASF guidelines.

  • Kudu now requires an out-of-tree build directory. Review the build instructions for additional information.

  • The C` client library is now explicitly built against the link:https://gcc.gnu.org/onlinedocs/libstdc/manual/using_dual_abi.html[old gcc5 ABI]. If you use gcc5 to build a Kudu application, your application must use the old ABI as well. This is typically achieved by defining the `_GLIBCXX_USE_CXX11_ABI macro at compile-time when building your application. For more information, see the previous link and link:http://developerblog.redhat.com/2015/02/05/gcc5-and-the-c11-abi/.

  • The Python client is no longer considered experimental.

Limitations

See also Limitations of the Kudu Public Beta. Where applicable, this list adds to or overrides that list.

Operating System Limitations

  • Kudu 0.7 is known to work on RHEL 7 or 6.4 or newer, CentOS 7 or 6.4 or newer, Ubuntu Trusty, and SLES 12. Other operating systems may work but have not been tested.

Release notes specific to 0.6.0

The 0.6.0 release contains incremental improvements and bug fixes. The most notable changes are:

  • The Java client’s CreateTableBuilder and AlterTableBuilder classes have been renamed to CreateTableOptions and AlterTableOptions. Their methods now also return this objects, allowing them to be used as builders.

  • The Java client’s AbstractKuduScannerBuilder#maxNumBytes() setter is now called batchSizeBytes as is the corresponding property in AsyncKuduScanner. This makes it consistent with the C++ client.

  • The "kudu-admin" tool can now list and delete tables via its new subcommands "list_tables" and "delete_table <table_name>".

  • OSX is now supported for single-host development. Please consult its specific installation instructions in OS X.

Limitations

See also Limitations of the Kudu Public Beta. Where applicable, this list adds to or overrides that list.

Operating System Limitations

  • Kudu 0.6 is known to work on RHEL 6.4 or newer, CentOS 6.4 or newer, and Ubuntu Trusty. Other operating systems may work but have not been tested.

API Limitations

  • The Python client is still considered experimental.

Release Notes Specific to 0.5.0

Limitations

See also Limitations of the Kudu Public Beta. Where applicable, this list adds to or overrides that list.

Operating System Limitations

  • Kudu 0.5 is known to work on RHEL 7 or 6.4 or newer, CentOS 7 or 6.4 or newer, Ubuntu Trusty, and SLES 12. Other operating systems may work but have not been tested.

API Limitations

  • The Python client is considered experimental.

About the Kudu Public Beta

This release of Kudu is a public beta. Do not run this beta release on production clusters. During the public beta period, Kudu will be supported via a public JIRA and a public mailing list, which will be monitored by the Kudu development team and community members. Commercial support is not available at this time.

  • You can submit any issues or feedback related to your Kudu experience via either the JIRA system or the mailing list. The Kudu development team and community members will respond and assist as quickly as possible.

  • The Kudu team will work with early adopters to fix bugs and release new binary drops when fixes or features are ready. However, we cannot commit to issue resolution or bug fix delivery times during the public beta period, and it is possible that some fixes or enhancements will not be selected for a release.

  • We can’t guarantee time frames or contents for future beta code drops. However, they will be announced to the user group when they occur.

  • No guarantees are made regarding upgrades from this release to follow-on releases. While multiple drops of beta code are planned, we can’t guarantee their schedules or contents.

Limitations of the Kudu Public Beta

Items in this list may be amended or superseded by limitations listed in the release notes for specific Kudu releases above.

Schema Limitations

  • Kudu is primarily designed for analytic use cases and, in the beta release, you are likely to encounter issues if a single row contains multiple kilobytes of data.

  • The columns which make up the primary key must be listed first in the schema.

  • Key columns cannot be altered. You must drop and recreate a table to change its keys.

  • Key columns must not be null.

  • Columns with DOUBLE, FLOAT, or BOOL types are not allowed as part of a primary key definition.

  • Type and nullability of existing columns cannot be changed by altering the table.

  • A table’s primary key cannot be changed.

  • Dropping a column does not immediately reclaim space. Compaction must run first. There is no way to run compaction manually, but dropping the table will reclaim the space immediately.

Ingest Limitations

  • Ingest via Sqoop or Flume is not supported in the public beta. The recommended approach for bulk ingest is to use Impala’s CREATE TABLE AS SELECT functionality or use the Kudu Java or C++ API.

  • Tables must be manually pre-split into tablets using simple or compound primary keys. Automatic splitting is not yet possible. See Schema Design.

  • Tablets cannot currently be merged. Instead, create a new table with the contents of the old tables to be merged.

Replication and Backup Limitations

  • Replication and failover of Kudu masters is considered experimental. It is recommended to run a single master and periodically perform a manual backup of its data directories.

Impala Limitations

  • To use Kudu with Impala, you must install a special release of Impala called Impala_Kudu. Obtaining and installing a compatible Impala release is detailed in Kudu’s Impala Integration documentation.

  • To use Impala_Kudu alongside an existing Impala instance, you must install using parcels.

  • Updates, inserts, and deletes via Impala are non-transactional. If a query fails part of the way through, its partial effects will not be rolled back.

  • All queries will be distributed across all Impala hosts which host a replica of the target table(s), even if a predicate on a primary key could correctly restrict the query to a single tablet. This limits the maximum concurrency of short queries made via Impala.

  • No timestamp and decimal type support.

  • The maximum parallelism of a single query is limited to the number of tablets in a table. For good analytic performance, aim for 10 or more tablets per host or use large tables.

  • Impala is only able to push down predicates involving =, , >=, or BETWEEN comparisons between any column and a literal value, and < and > for integer columns only. For example, for a table with an integer key ts, and a string key name, the predicate WHERE ts >= 12345 will convert into an efficient range scan, whereas where name > 'lipcon' will currently fetch all data from the table and evaluate the predicate within Impala.

Security Limitations

  • Authentication and authorization are not included in the public beta.

  • Data encryption is not included in the public beta.

Client and API Limitations

  • Potentially-incompatible C++, Java and Python API changes may be required during the public beta.

  • ALTER TABLE is not yet fully supported via the client APIs. More ALTER TABLE operations will become available in future betas.

Application Integration Limitations

  • The Spark DataFrame implementation is not yet complete.

Other Known Issues

The following are known bugs and issues with the current beta release. They will be addressed in later beta releases.

  • Building Kudu from source using gcc 4.6 or 4.7 causes runtime and test failures. Be sure you are using a different version of gcc if you build Kudu from source.

  • If the Kudu master is configured with the -log_fsync_all option, tablet servers and clients will experience frequent timeouts, and the cluster may become unusable.

  • If a tablet server has a very large number of tablets, it may take several minutes to start up. It is recommended to limit the number of tablets per server to 100 or fewer. Consider this limitation when pre-splitting your tables. If you notice slow start-up times, you can monitor the number of tablets per server in the web UI.

Installation Options

  • A Quickstart VM is provided to get you up and running quickly.

  • You can install Kudu using provided deb/yum packages.

  • You can install Kudu, in clusters managed by Cloudera Manager, using parcels or deb/yum packages.

  • You can build Kudu from source.

For full installation details, see Kudu Installation.