org.apache.kudu.mapreduce

Class KuduTableMapReduceUtil

java.lang.Object

org.apache.kudu.mapreduce.KuduTableMapReduceUtil

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class KuduTableMapReduceUtil
extends Object

Utility class to setup MR jobs that use Kudu as an input and/or output.

Method Summary

Modifier and Type	Method and Description
static void	addCredentialsToJob(KuduClient client, org.apache.hadoop.mapreduce.Job job) Export the credentials from a KuduClient and store them in the given MapReduce Job so that KuduClients created from within tasks of that job can authenticate to Kudu.
static void	addDependencyJars(org.apache.hadoop.conf.Configuration conf, Class<?>... classes) Add the jars containing the given classes to the job's configuration such that JobClient will ship them to the cluster and add them to the DistributedCache.
static void	addDependencyJars(org.apache.hadoop.mapreduce.Job job) Add the Kudu dependency jars as well as jars for any of the configured job classes to the job configuration, so that JobClient will ship them to the cluster and add them to the DistributedCache.
static void	addKuduDependencyJars(org.apache.hadoop.conf.Configuration conf) Add Kudu and its dependencies (only) to the job configuration.
static KuduTable	getTableFromContext(org.apache.hadoop.mapreduce.TaskInputOutputContext context) Use this method when setting up a task to get access to the KuduTable in order to create Inserts, Updates, and Deletes.
static void	importCredentialsFromCurrentSubject(KuduClient client) Import credentials from the current thread's JAAS Subject into the provided KuduClient.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getTableFromContext

public static KuduTable getTableFromContext(org.apache.hadoop.mapreduce.TaskInputOutputContext context)

Use this method when setting up a task to get access to the KuduTable in order to create Inserts, Updates, and Deletes.

Parameters:

context - Map context

Returns:

The kudu table object as setup by the output format

addCredentialsToJob

public static void addCredentialsToJob(KuduClient client,
                                       org.apache.hadoop.mapreduce.Job job)
                                throws KuduException

Export the credentials from a KuduClient and store them in the given MapReduce Job so that KuduClients created from within tasks of that job can authenticate to Kudu. This must be used before submitting a job when running against a Kudu cluster configured to require authentication. If using KuduTableMapReduceUtil.TableInputFormatConfigurator, KuduTableMapReduceUtil.TableOutputFormatConfigurator or another such utility class, this is called automatically and does not need to be called.

Parameters:

client - the client whose credentials to export

job - the job to configure

Throws:

KuduException - if credentials cannot be exported

importCredentialsFromCurrentSubject

public static void importCredentialsFromCurrentSubject(KuduClient client)

Import credentials from the current thread's JAAS Subject into the provided KuduClient. This must be called for any clients created within a MapReduce job in order to adopt the credentials added by addCredentialsToJob(KuduClient, Job). When using KuduTableInputFormat or KuduTableOutputFormat, the implementation automatically handles creating the client and importing necessary credentials. As such, this is only necessary in jobs that explicitly create a KuduClient. If no appropriate credentials are found, does nothing.

addDependencyJars

public static void addDependencyJars(org.apache.hadoop.mapreduce.Job job)
                              throws IOException

Add the Kudu dependency jars as well as jars for any of the configured job classes to the job configuration, so that JobClient will ship them to the cluster and add them to the DistributedCache.

Throws:

IOException

addDependencyJars

public static void addDependencyJars(org.apache.hadoop.conf.Configuration conf,
                                     Class<?>... classes)
                              throws IOException

Add the jars containing the given classes to the job's configuration such that JobClient will ship them to the cluster and add them to the DistributedCache.

Throws:

IOException

addKuduDependencyJars

public static void addKuduDependencyJars(org.apache.hadoop.conf.Configuration conf)
                                  throws IOException

Add Kudu and its dependencies (only) to the job configuration.

This is intended as a low-level API, facilitating code reuse between this class and its mapred counterpart. It also of use to external tools that need to build a MapReduce job that interacts with Kudu but want fine-grained control over the jars shipped to the cluster.

Parameters:

conf - The Configuration object to extend with dependencies.

Throws:

IOException

See Also:

KuduTableMapReduceUtil, PIG-3285